JP7651833B2 - IMAGE EDITING DEVICE, IMAGE EDITING METHOD, AND IMAGE EDITING PROGRAM - Google Patents

Description

The present invention relates to an image editing device, an image editing method, and an image editing program.

In communication situations, verbal communication alone may not be sufficient to transmit information efficiently and accurately. For example, in meetings held to think up new plans or ideas, it is effective to use visual information to communicate. In communication situations, accurate means of visual information are desirable.

Patent Document 1 describes how an illustration displayed in an illustration search result display area is selected on the minutes editing screen of a graphic recording system, and the selected illustration is pasted into the minutes to create minutes that include illustrations. It describes how looking at the minutes that include illustrations allows for an efficient review of the meeting.

The technology described in Patent Document 1 only allows the user to select an illustration set in the graphic recording system, and depending on the illustration set, it may not be possible to fully convey the user's intentions.

The present invention has been made in consideration of the above, and aims to provide an image editing device, an image editing method, and an image editing program that can present visual information that can be edited by a user.

In order to solve the above-mentioned problems and achieve the object, an image editing device according to one aspect of the present invention has a display means for displaying image data, a first editing means for performing a first editing process on first image data having a first data format among the image data displayed on the display means, a second editing means for performing a second editing process on second image data having a second data format different from the first data format among the image data displayed on the display means, and a conversion means for converting the first data format into the second data format, and when the first editing process is performed by the first editing means, the display of the first image data is changed to the image data after the first editing process, and when the second editing process is performed on the second image data converted by the conversion means, the display of the second image data is changed to the display after the second editing process.

The present invention has the advantage of being able to present visual information that can be edited by the user.

FIG. 1 is a diagram showing a configuration of a visual communication system according to an embodiment. FIG. 2 is a diagram showing a hardware configuration of a computer applied to the visual communication system according to the embodiment. FIG. 3 is a diagram showing a configuration of a language information input unit in the embodiment. FIG. 4 is a diagram showing the configuration of an illustration display unit in the embodiment. FIG. 5 is a data flow diagram illustrating the conversion of image data in an embodiment. FIG. 6 is a diagram showing a data structure of illustration information stored in the illustration storage unit in the embodiment. FIG. 7 is a diagram showing the data structure of thumbnail information stored in the thumbnail storage unit in the embodiment. FIG. 8 is a diagram showing the configuration of a drawing operation unit in the embodiment. FIG. 9 is a flowchart showing the operation of the visual communication system according to the embodiment. FIG. 10 is a flowchart showing the flow of the registration process in the embodiment. FIG. 11 is a flowchart showing the flow of the communication support process in the embodiment. FIG. 12 is a flowchart showing the flow of the communication support process in the embodiment. FIG. 13 is a diagram showing the configuration of a display screen of the visual communication system according to the embodiment. FIG. 14 is a diagram showing an outline of the operation of a display screen of the visual communication system according to the embodiment. FIG. 15 is a diagram showing the flow of a two-stage editing process in this embodiment. FIG. 16 is a diagram showing a detailed operation of a display screen of the visual communication system according to the embodiment. FIG. 17 is a diagram showing a detailed operation of a display screen of the visual communication system according to the embodiment. FIG. 18 is a diagram showing a detailed operation of a display screen by the visual communication system in the embodiment. FIG. 19 is a diagram showing a detailed operation of a display screen of the visual communication system according to the embodiment. FIG. 20 is a diagram showing a detailed operation of a display screen of the visual communication system in the embodiment. FIG. 21 is a diagram showing a detailed operation of a display screen of the visual communication system according to the embodiment. FIG. 22 is a diagram showing a detailed operation of a display screen by the visual communication system in the embodiment. FIG. 23 is a diagram showing a detailed operation of a display screen by the visual communication system in the embodiment. FIG. 24 is a diagram illustrating a configuration of an online conference system according to a modified example of the embodiment. FIG. 25 is a diagram showing a camera image captured by a camera of a client terminal in a modified example of the embodiment. FIG. 26 is a diagram showing an image superimposed on the camera image of FIG. 25 by a drawing display unit in a modification of the embodiment. FIG. 27 is a diagram showing an image displayed on another client terminal in the modified example of the embodiment.

(Embodiment)
The visual communication system according to the embodiment has a function of supporting communication. In communication situations, communication by language information alone may not be sufficient to efficiently and accurately transmit information. For example, in a meeting held when thinking of new plans or ideas, it is effective to use visual information for communication. One possible way to visualize information is to sketch a picture of an idea by hand, but it may not be easy to visualize the scene in one's head from scratch. For this reason, a first technology is known that converts information into visual information based on language information that is easy for anyone to use.

However, the first technology, which converts language information into visual information, merely searches for illustrations prepared in advance. More specifically, there are many cases where the information cannot be properly transmitted because there are no illustrations with orientations or combinations that match the scene in one's head. Increasing the number of prepared illustrations increases the probability that an illustration matching the scene in one's head is present, but as the number of illustrations increases, the time required to select increases. Furthermore, even if one tries to edit the illustrations themselves in order to accurately reproduce the scene in one's head, if there is only a simple editing function, the edited illustration is unlikely to be close to the scene in one's head. For this reason, the first technology may not be able to accurately transmit information in communication situations such as meetings.

In this embodiment, the visual communication system searches for and displays illustrations that correspond to the language information received from the user, and enables a two-stage editing process in which the display form of the displayed illustration is changed, thereby supporting accurate information transmission in communication situations.

Specifically, the visual communication system converts language information into visual information that can be edited as desired, enabling accurate communication of the scene in one's head in a short time in a communication situation such as a conference. The visual communication system analyzes language information such as speech and character input in a communication situation in real time, and at the same time displays one or more thumbnail images in a specific area as candidates for illustrations related to the language information. The visual communication system has first correspondence information in which language information and thumbnail images are associated, and when language information is received, one or more thumbnail images corresponding to the language information in the first correspondence information are specified and displayed. In addition, the visual communication system has second correspondence information in which thumbnail images are associated with illustrations. When the illustration is a three-dimensional image, the visual communication system can display the thumbnail image at a higher speed than when displaying an illustration by making the thumbnail image a two-dimensional image. Each time language information from a user is received, the thumbnail image on the display screen can be switched to a new one and updated and displayed at a high speed. This allows the user to select one or more thumbnail images that are close to the scene in his or her head when the speech or character input is received from among them without stopping the conversation. According to the second correspondence information, three-dimensional illustration data associated with the selected thumbnail image is searched for, and the identified three-dimensional illustration data is displayed. This allows three-dimensional editing (three-dimensional movement, enlargement, reduction, and rotation) of the three-dimensional illustration data. The visual communication system then converts (fixes) the three-dimensional illustration data that can be three-dimensionally edited into two-dimensional illustration data that can be two-dimensionally edited. In response to this, two-dimensional editing (freely erasing parts, adding on top, or painting) of the two-dimensional illustration data is possible. This two-stage editing process triggered by automatic recognition of language information allows the scene in one's head to be accurately expressed in a short time in a real-time communication situation, enabling rapid and accurate communication.

More specifically, the visual communication system 4 may be configured as shown in FIG. 1. FIG. 1 is a diagram showing the configuration of the visual communication system 4.

The visual communication system 4 has a client terminal 1, a server 2, and a connection unit 3. The connection unit 3 connects the client terminal 1 and the server 2 so that they can communicate with each other.

The client terminal 1 has a language information input unit 100, an illustration display unit 200, and a drawing operation unit 300. The server 2 has a drawing display unit 400 and a memory unit 500. The memory unit 500 stores a program 500a.

When the visual communication system 4 receives a startup request from a user at the client terminal 1, the startup request is transmitted from the client terminal 1 to the server 2, and the server 2 reads the program 500a from the storage unit 500 in response to the startup request. In accordance with the program 500a, the visual communication system 4 functionally configures a language information input unit 100, an illustration display unit 200, and a drawing operation unit 300 in the client terminal 1, and functionally configures a drawing display unit 400 in the server 2, as shown in FIG. 1, for example.

In addition, the visual communication system 4 may configure the illustration display unit 200 and the drawing operation unit 300 in the client terminal 1 in the server 2 or another server. Alternatively, the visual communication system 4 may be configured so that all functional configurations including the language information input unit 100, the illustration display unit 200, the drawing operation unit 300, and the drawing display unit 400 are completed in the client terminal 1. Alternatively, the visual communication system 4 may configure a part of the multiple elements included in the illustration display unit 200 (e.g., the user interface and a part similar thereto) in the client terminal 1, and the remaining part in the server 2 or another server. Similarly, the visual communication system 4 may configure a part of the multiple elements included in the drawing operation unit 300 (e.g., the user interface and a part similar thereto) in the client terminal 1, and the remaining part in the server 2 or another server.

The connection unit 3 may be a wired communication line and/or a wireless communication line, may be a so-called communication network, may be a communication cable, etc. The connection unit 3 may be constructed using one or more of the Internet, a mobile communication network, a LAN (Local Area Network), etc. The connection unit 3 may include not only wired communication but also a wireless communication network such as 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), Wi-Fi (Wireless Fidelity) (registered trademark), WiMAX (Worldwide Interoperability for Microwave Access), or LTE (Long Term Evolution). If the program 500a is stored in the client terminal 1 and each functional configuration is configured to be completed within the client terminal 1, the connection unit 3 may be omitted.

In the client terminal 1, the language information input unit 100, which serves as a language information input means, accepts input of language information by the user. The illustration display unit 200 displays images such as characters and illustrations on the display. The drawing operation unit 300 accepts drawing operations by the user. Here, drawing includes not only handwriting but also selection operations on the display for the purpose of drawing on the display. The drawing operation unit 300 of the client terminal 1 transmits a drawing operation request to the drawing display unit 400 of the server 2. In response to the drawing operation request, the drawing display unit 400 of the server 2 updates the display information so as to change the display form of the displayed image, and transmits the updated display information to the client terminal 1. The client terminal 1 receives the updated display information and displays an image corresponding to the display information on the display. As a result, the result of the drawing operation by the user is displayed on the display of the client terminal 1.

The client terminal 1 and the server 2 may each be configured in hardware as a computer 5 as shown in FIG. 2. FIG. 2 is a diagram showing the hardware configuration of the computer 5 applied to the visual communication system 4.

As shown in FIG. 2, the computer 5 includes a CPU 501, a ROM 502, a RAM 503, a HD 504, a HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I/F (Interface) 508, a network I/F 509, a data bus 510, a keyboard 511, a pointing device 512, a DVD-RW (Digital Versatile Disk Rewritable) drive 514, a media I/F 516, a motion acquisition device 517, a microphone 518, a speaker 519, and a camera 520.

Of these, the CPU 501 controls the operation of the entire computer 5. The ROM 502 stores programs used to drive the CPU 501, such as IPL. The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data such as the program 500a. The HDD controller 505 controls the reading or writing of various data from the HD 504 according to the control of the CPU 501. The display 506 as a display means displays various information such as a cursor, menu, window, character, or image. The external device connection I/F 508 is an interface for connecting various external devices. In this case, the external device is, for example, a USB (Universal Serial Bus) memory or a printer. The network I/F 509 is an interface for data communication using the connection unit 3. The bus line 510 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501 shown in FIG. 2.

The keyboard 511 is a type of input means equipped with multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving the cursor, etc. The DVD-RW drive 514 controls the reading and writing of various data from the DVD-RW 513, which is an example of a removable recording medium. Note that this is not limited to a DVD-RW, and may be a DVD-R, etc. The media I/F 516 controls the reading and writing (storing) of data from the recording medium 515, such as a flash memory.

The motion acquisition device 517 is a circuit that detects the user's motion and converts it into an electrical signal, and is a type of input means. The user's motion may be detected by any of a number of methods, including an optical method that detects the reflection of light, a magnetic sensor, a mechanical sensor, or a magnetic sensor attached to the user, or a method that analyzes an image of the user, or a combination of these.

The microphone 518 is a built-in circuit that converts sound into an electrical signal. The speaker 519 is a built-in circuit that converts the electrical signal into physical vibrations to produce sounds such as music or voice.

Speaker 519 is a built-in circuit that converts electrical signals into physical vibrations to produce sounds such as music and voice.

Camera 520 is a type of built-in imaging means that captures an image of a subject and obtains image data.

In the computer 5 applied to the server 2, the HD 504 corresponds to the storage unit 500 and stores the program 500a. In the computer 5 applied to the client terminal 1, the program 500a is downloaded from the server 2 via the connection unit 3 and stored in the HD 504, and the program 500a is read from the HD 504 and executed by the CPU 501, so that the functional configuration as shown in FIG. 1 can be expanded in the RAM 503 either collectively at the time of compilation or sequentially as the processing progresses. In the computer 5 applied to the server 2, the program 500a is executed by the CPU 501, so that the functional configuration as shown in FIG. 1 can be expanded in the RAM 503 either collectively at the time of compilation or sequentially as the processing progresses.

In addition, the computer 5 applied to the server 2 may omit configuration related to the user interface, and at least one of the display 506, keyboard 511, pointing device 512, and action acquisition device 517 may be omitted.

In addition to the computer 5, the client terminal 1 and the server 2 may be an output device such as an IWB (Interactive White Board: an electronic whiteboard capable of intercommunication), digital signage, a HUD (Head Up Display) device, industrial machinery, medical equipment, network home appliances, automobiles (Connected Cars), mobile phones, smartphones, tablet terminals, game consoles, PDAs (Personal Digital Assistants), etc.

The language information input unit 100 shown in FIG. 1 can be functionally configured as shown in FIG. 3. FIG. 3 is a diagram showing the configuration of the language information input unit 100.

The language information input unit 100 includes a voice input unit 101, a character input unit 102, a recognition unit 103, and a transmission unit 104. Each of these units is a function or means realized by operating according to an instruction from the CPU 501 in accordance with a program loaded from the HD 504 onto the RAM 503.

The voice input unit 101 accepts, as language information, a voice signal converted from the user's voice by the microphone 318, or a voice signal received via the network I/F 509. The voice input unit 101 supplies the voice signal to the recognition unit 103.

The character input unit 102 accepts, as language information, character signals input by the user via the keyboard 541, the pointing device 512, or the motion acquisition device 517, or character signals received via the network I/F. The character input unit 102 supplies the character signals to the recognition unit 103. Characters are input to the character input unit 102 by typing on the keyboard 541, or by handwriting input using the pointing device 512 or the motion acquisition device 517.

When the recognition unit 103 receives language information from the voice input unit 101 or the character input unit 102, it recognizes character information based on the language information.

When the recognition unit 103 receives a voice signal from the voice input unit 101, it performs voice recognition processing on the voice signal and converts it into character information. The recognition unit 103 has a template voice signal for each character. The recognition unit 103 performs matching processing on the voice signal using the template voice signal, and can recognize characters for each voice based on the matching score. In this way, the recognition unit 103 generates character information as the recognition result for the voice signal.

When the recognition unit 103 receives a character signal from the character input unit 102, it generates character information as the recognition result for the character signal. When the recognition unit 103 receives character information typed on the keyboard 541, it takes the character information as the recognition result. When the recognition unit 103 receives a handwritten character image from the pointing device 512 or the motion acquisition device 517, it performs text recognition processing on the handwritten character image and converts it into character information. The recognition unit 103 has a template character image for each character. The recognition unit 103 performs matching processing on the handwritten character image using the template character image, and can recognize characters for each handwritten character image based on the matching score. In this way, the recognition unit 103 generates character information as the recognition result for the handwritten character image.

The recognition unit 103 supplies the recognition result to the transmission unit 104. The transmission unit 104 transmits the recognition result to the illustration display unit 200.

The illustration display unit 200 shown in FIG. 1 can be functionally configured as shown in FIG. 4. FIG. 4 is a diagram showing the configuration of the illustration display unit 200.

The illustration display unit 200 includes a receiving unit 201, a thumbnail search unit 202, a thumbnail display unit 203, a thumbnail selection unit 204, an illustration search unit 205, an illustration display unit 206, a context analysis unit 207, a priority determination unit 208, a thumbnail conversion unit 211, an illustration conversion unit 213, a three-dimensional data input unit 214, a keyword conversion unit 215, and a registered character input unit 216.

Each of these units is a function or means that is realized by operating according to instructions from the CPU 501 in accordance with the program 500a loaded onto the RAM 503 from the HD 504.

The illustration display unit 200 also includes a selection tendency storage unit 209, a thumbnail storage unit 210, and an illustration storage unit 212. Each of these units is constructed using a ROM 502, a RAM 503, or a HD 504.

The storage of data in the thumbnail storage unit 210 and the illustration storage unit 212 will be explained using Figures 4 and 5. Figure 5 is a data flow diagram showing the conversion of image data.

The three-dimensional data input unit 214 receives three-dimensional data. The three-dimensional data is, for example, polygon data as shown in FIG. 5(a) and includes multiple spatial coordinates. The three-dimensional data corresponds to a three-dimensional image data format (for example, OBJ format). The three-dimensional data input unit 214 supplies the three-dimensional data to the illustration unit 213.

The illustration conversion unit 213 converts the three-dimensional data into three-dimensional illustration data. The three-dimensional illustration data is, for example, three-dimensional line drawing data as shown in FIG. 5(b) and includes multiple spatial coordinates. The three-dimensional illustration data corresponds to a three-dimensional image data format (for example, the OBJ format). The illustration conversion unit 213 extracts line drawing information from the three-dimensional data, for example by identifying the three-dimensional positions of the edges of the three-dimensional shape represented by the three-dimensional data and connecting them with line drawing, thereby generating three-dimensional illustration data. The illustration conversion unit 213 additionally stores the three-dimensional illustration data in the illustration storage unit 212.

As a result, illustration information 212a as shown in FIG. 6 is stored in the illustration storage unit 212. FIG. 6 is a diagram showing the data structure of illustration information 212a stored in illustration storage unit 212. In illustration information 212a, three-dimensional illustration data and its identification information are associated with one or more pieces of three-dimensional illustration data. For example, illustration information 212a has an identification information column 212a1 and an access information column 212a2. In identification information column 212a1, information for identifying the three-dimensional illustration data is recorded, for example, an ID number of the three-dimensional illustration data is recorded. In access information column 212a2, information for accessing the three-dimensional illustration data is recorded, for example, a file name of the three-dimensional illustration data is recorded.

The thumbnail generating unit 211 acquires the three-dimensional illustration data from the illustration storage unit 212 when the three-dimensional illustration data is added to the illustration storage unit 212 or at predetermined intervals, and converts the three-dimensional illustration data into two-dimensional thumbnail data. The two-dimensional thumbnail data is, for example, two-dimensional line drawing data as shown in FIG. 5(c), and is line drawing data equivalent to the three-dimensional illustration data being reduced and two-dimensionalized. The two-dimensional thumbnail data corresponds to a two-dimensional image data format (for example, BMP format). The two-dimensional thumbnail data is associated with the identification information of the three-dimensional illustration data. The thumbnail generating unit 211 obtains a plurality of plane coordinates in which a plurality of spatial coordinates included in the three-dimensional illustration data are projected onto a predetermined plane, and reduces the two-dimensional line drawing data corresponding to the obtained plurality of plane coordinates, thereby generating two-dimensional thumbnail data that has been reduced and two-dimensionalized from the three-dimensional illustration data. The thumbnail generating unit 211 additionally stores the two-dimensional thumbnail data in the thumbnail storage unit 210 in a form associated with the identification information of the three-dimensional illustration data.

As a result, thumbnail information 210a as shown in FIG. 7 is stored in the thumbnail storage unit 210. FIG. 7 is a diagram showing the data structure of thumbnail information 210a stored in the thumbnail storage unit 210. Thumbnail information 210a is information in which a keyword, three-dimensional illustration data, and two-dimensional thumbnail data are associated with one or more pieces of two-dimensional thumbnail data. Thumbnail information 210a includes information in which a keyword and two-dimensional thumbnail data are associated with each other as first association information. As shown in FIG. 7, multiple keywords may be set for one piece of two-dimensional thumbnail data. Furthermore, thumbnail information 210a includes information in which two-dimensional thumbnail data and three-dimensional illustration data are associated with each other as second association information. For example, thumbnail information 210a has a keyword column 210a1, an identification information column 210a2, and an access information column 210a3. A keyword for calling up two-dimensional thumbnail data is recorded in the keyword column 210a1, but it is blank when no keyword is registered. The identification information field 210a2 records information for identifying the 3D illustration data linked to the 2D thumbnail data, for example, an ID number for the 3D illustration data. The access information field 210a3 records information for accessing the 2D thumbnail data, for example, a file name for the 2D thumbnail data.

The registered character input unit 216 receives character information to be linked to the three-dimensional illustration data when three-dimensional illustration data is added to the illustration storage unit 212, or at predetermined intervals. The registered character input unit 216 accepts the character information in a form associated with identification information of the three-dimensional illustration data. At this time, as indicated by the dotted arrow, the registered character input unit 216 may obtain identification information of multiple three-dimensional illustration data that are candidates to be linked to the character information from the illustration storage unit 212, and select and input identification information from the identification information of the multiple three-dimensional illustration data. The registered character input unit 216 supplies the character information to the keyword unit 215 in a form associated with the identification information of the three-dimensional illustration data.

When the keyword generation unit 215 receives the text information associated with the identification information of the three-dimensional illustration data, it accesses the thumbnail storage unit 210 and writes the text information into the keyword field 210a1 that corresponds to the identification information of the three-dimensional illustration data.

As a result, in the thumbnail information 210a stored in the thumbnail storage unit 210, one or more keywords are additionally recorded in the keyword column 210a1, as shown in FIG. 7.

The data size of the 2D thumbnail data is significantly smaller than the data size of the 3D illustration data. By using thumbnails with a small data size when searching for illustrations based on language information, it is possible to minimize the time lag when retrieving images.

The data storage in the selection tendency storage unit 209 will be described with reference to FIG. 4. When the receiving unit 201 receives the language information recognition result, it supplies the language information recognition result to the context analysis unit 207. The context analysis unit 207 analyzes the context based on the language information recognition result. The selection information in the thumbnail selection unit 204 is stored as the user's selection tendency in the selection tendency storage unit 209. Based on the analysis result of the context analysis unit 207 and the storage result of the selection tendency storage unit 209, the priority determination unit 208 determines the priority when displaying the two-dimensional thumbnail data, and supplies the determination result to the thumbnail display unit 203. The thumbnail display unit 203 displays the two-dimensional thumbnail data on the display in descending order of priority according to the determination result of the priority determination unit 208.

An example of the priority may be a priority determined in response to a foreign language as language information. The priority determination unit 208 may determine a priority such that the thumbnails that appear are changed depending on the type of language. For example, when the word "school" is detected in an African language, the priority is determined such that an illustration showing a school that is commonly seen in Africa is preferentially displayed rather than an illustration showing a typical school in Japan.

The method of displaying the illustration will be described with reference to FIG. 4. When the receiving unit 201 receives the language information recognition result, it supplies the language information recognition result to the thumbnail search unit 202. The thumbnail search unit 202 searches for two-dimensional thumbnail data based on the language information recognition result. The thumbnail search unit 202 accesses the thumbnail storage unit 210, searches for thumbnail information 210a using character information corresponding to the language information recognition result as a keyword, and calls up one or more two-dimensional thumbnail data corresponding to the character information (e.g., a keyword) and identification information of the three-dimensional illustration data associated therewith as search results, and supplies them to the thumbnail display unit 203. The thumbnail display unit 203 as the first display control means supplies the searched one or more two-dimensional thumbnail data to the drawing display unit 400 (see FIG. 1) via the connection unit 3. In response to this, the drawing display unit 400 displays the searched one or more two-dimensional thumbnail data in a specific location on the display of the client terminal 1. The thumbnail selection unit 204 as a selection means selects two-dimensional thumbnail data from one or more two-dimensional thumbnail data displayed on the display in response to a selection operation by the user, for example, an operation of selecting two-dimensional thumbnail data displayed on the display with a pointing device, and supplies identification information of three-dimensional illustration data associated with the selected two-dimensional thumbnail data to the illustration search unit 205. The illustration search unit 205 searches for three-dimensional illustration data based on the identification information of the three-dimensional illustration data. The illustration search unit 205 accesses the illustration storage unit 212, retrieves the three-dimensional illustration data corresponding to the identification information as a search result, and supplies it to the illustration display unit 206. The illustration display unit 206 as a second display control means supplies the searched three-dimensional illustration data to the drawing display unit 400 (see FIG. 1) via the connection unit 3. In response to this, the drawing display unit 400 displays the searched three-dimensional illustration data on the display of the client terminal 1.

The drawing operation unit 300 shown in FIG. 1 can be functionally configured as shown in FIG. 8. FIG. 8 is a diagram showing the configuration of the drawing operation unit 300.

The drawing operation unit 300 includes an illustration editing unit 301, a fixation unit 302, a drawing unit 303, a hand-drawn input unit 304, and an output unit 305.

The illustration editing unit 301 as the first editing means (three-dimensional data editing input means) receives three-dimensional illustration data from the illustration display unit 200 and performs three-dimensional editing processing on the three-dimensional illustration data. In the three-dimensional editing processing, the illustration editing unit 301 accepts three-dimensional rotation operations, three-dimensional movement operations, three-dimensional enlargement operations, three-dimensional reduction operations, etc. by the user, and supplies these operation requests to the drawing display unit 400 (see FIG. 1) via the drawing unit 303, the output unit 305, and the connection unit 3. The three-dimensional rotation operation is an operation in which multiple spatial coordinates included in the three-dimensional illustration data are changed so that they are rotated three-dimensionally around a predetermined axis while maintaining their relative positional relationship. The three-dimensional enlargement operation is an operation in which multiple spatial coordinates included in the three-dimensional illustration data are changed so that they move radially away from a predetermined point at an equal distance rate. The three-dimensional reduction operation is an operation in which multiple spatial coordinates included in the three-dimensional illustration data are changed so that they move radially closer to a predetermined point at an equal distance rate. In response to these operation requests, the drawing display unit 400 changes the display format of the three-dimensional illustration data in the drawing unit 303 of the client terminal 1. This allows the position, size, and orientation of the three-dimensional illustration data on the display of the client terminal 1 to be changed three-dimensionally.

The fixing unit 302 as a conversion means converts (fixes) three-dimensional illustration data that can be edited in three dimensions into two-dimensional illustration data that can be edited in two dimensions in response to a predetermined operation (for example, an operation that activates two-dimensional editing, more specifically, an operation that transitions to a state where two-dimensional editing is possible). The two-dimensional illustration data is two-dimensional line drawing data as shown in FIG. 5(d), and is line drawing data that corresponds to three-dimensional illustration data that has been subjected to three-dimensional editing and turned into two dimensions. The two-dimensional illustration data corresponds to a two-dimensional image data format (for example, BMP format). The fixing unit 302 obtains a plurality of plane coordinates in which a plurality of spatial coordinates included in the three-dimensional illustration data are projected onto a plane corresponding to the display screen, and generates two-dimensional line drawing data corresponding to the obtained plurality of plane coordinates, thereby converting the three-dimensional illustration data into two-dimensional illustration data and fixing it to the drawing unit 303.

The two-dimensional data editing input unit 304 as the second editing means (two-dimensional data editing input means) performs two-dimensional editing processing on the two-dimensional illustration data. In the two-dimensional editing processing, the two-dimensional data editing input unit 304 accepts two-dimensional rotation operations, two-dimensional movement operations, two-dimensional enlargement operations, two-dimensional reduction operations, partial erasure operations, operations to add hand-drawn lines, coloring operations, and the like, performed by the user, and supplies these operation requests to the drawing display unit 400 (see FIG. 1) via the drawing unit 303, the output unit 305, and the connection unit 3. The two-dimensional rotation operation is an operation in which multiple planar coordinates included in the two-dimensional illustration data are changed so as to rotate two-dimensionally around a specified point while maintaining their relative positional relationship. The two-dimensional enlargement operation is an operation in which multiple planar coordinates included in the two-dimensional illustration data are changed so as to move radially away from a specified point at equal distance ratios. The two-dimensional reduction operation is an operation in which multiple plane coordinates included in the two-dimensional illustration data are changed so that they approach a predetermined point radially at an equal distance ratio. The partial erasure operation is an operation in which a portion of multiple plane coordinates included in the two-dimensional illustration data is deleted. The operation of adding handwritten lines is an operation in which multiple plane coordinates corresponding to handwritten lines are added to multiple plane coordinates included in the two-dimensional illustration data. The coloring operation is an operation in which multiple plane coordinates associated with a predetermined color attribute are added to multiple plane coordinates included in the two-dimensional illustration data. In response to these operation requests, the drawing display unit 400 changes the display form of the two-dimensional illustration data in the drawing unit 303 of the client terminal 1. As a result, the position, size, and orientation of the two-dimensional illustration data on the display of the client terminal 1 are changed two-dimensionally, a portion of the two-dimensional illustration data is erased, handwritten lines are added to the two-dimensional illustration data, or the two-dimensional illustration data is colored.

Input to the drawing unit 303 is also possible using a mouse, a finger, a stylus pen, gestures, etc. Information from the drawing unit 303 is output to the drawing display unit 400 by the output unit 305. In response to this, the drawing display unit 400 adds line drawings, etc. handwritten input in the drawing unit 303 of the client terminal 1. As a result, the handwritten input line drawings, etc. are displayed on the display of the client terminal 1.

Each of these units is a function or means that is realized by operating according to instructions from the CPU 501 in accordance with a program loaded onto the RAM 503 from the HD 504.

Next, the operation of the visual communication system 4 will be described with reference to FIG. 9. FIG. 9 is a flowchart showing the operation of the visual communication system.

The visual communication system 4 performs a registration process (S1) in which predetermined information is registered in preparation for supporting communication. The visual communication system 4 then waits until a startup request is received from the user (No in S2). When a startup request is received from the user (Yes in S2), the visual communication system 4 starts the program 500a and starts the communication support process (S3). The visual communication system 4 continues to perform the communication support process (S3) until a termination request is received from the user (No in S4). When a termination request is received from the user (Yes in S4), the visual communication system 4 terminates the process.

Next, the details of the registration process (S1) will be explained using FIG. 10. FIG. 10 is a flowchart showing the flow of the registration process.

When program 500a is started, visual communication system 4 waits until a request to register three-dimensional data is received (No in S11). When a request to register three-dimensional data is received (Yes in S11), the three-dimensional data is input to visual communication system 4 (S12). The three-dimensional data is, for example, polygon data (see FIG. 5(a)). The visual communication system 4 converts the three-dimensional data into three-dimensional illustration data (S13). That is, visual communication system 4 converts the three-dimensional data into three-dimensional illustration data. The three-dimensional illustration data is, for example, three-dimensional line drawing data (see FIG. 5(b)). The visual communication system 4 additionally stores the three-dimensional illustration data in illustration storage unit 212. As a result, illustration information 212a (see FIG. 6) is registered in illustration storage unit 212.

The visual communication system 4 thumbnails the three-dimensional illustration data (S14). That is, the visual communication system 4 converts the three-dimensional illustration data into two-dimensional thumbnail data. The two-dimensional thumbnail data is, for example, two-dimensional line drawing data (see FIG. 5(c)). The visual communication system 4 additionally stores the two-dimensional thumbnail data in the thumbnail storage unit 210 in a form associated with identification information of the three-dimensional illustration data. As a result, the two-dimensional thumbnail data and the identification information of the three-dimensional illustration data in the thumbnail information 210a (see FIG. 7) are registered in the thumbnail storage unit 210.

Then, the visual communication system 4 waits until a keyword registration request is made (No in S15). When the visual communication system 4 receives a keyword registration request (Yes in S15), the text information is input in a form associated with the identification information of the three-dimensional illustration data (S16). The visual communication system 4 converts the text information into a keyword (S17). That is, the visual communication system 4 accesses the thumbnail storage unit 210 and additionally writes the text information in the keyword field 210a1 corresponding to the identification information of the three-dimensional illustration data. As a result, the text information (keyword) in the thumbnail information 210a (see FIG. 7) is registered in the thumbnail storage unit 210. Note that it is not necessary to perform both the registration process and the communication support process as shown in FIG. 9, and it is also possible to execute only the communication support process based on the information registered in the thumbnail storage unit 210 and the illustration storage unit 212 at that time.

Next, the details of the communication support process (S3) will be described with reference to Figs. 11 and 12. Figs. 11 and 12 are flowcharts showing the flow of the communication support process. The process of Fig. 11 and the process of Fig. 12 can be performed in parallel with each other.

In the process of FIG. 11, when the program 500a is started and an initial screen is displayed on the display, the visual communication system 4 waits until there is a request to turn on the language information input function via the initial screen (No in S21). When there is a request to turn on the language information input function (Yes in S21), the visual communication system 4 waits until language information is input (No in S22). When language information is input (Yes in S22), the visual communication system 4 determines whether the language information is voice information (S23). If the language information is voice information (Yes in S23), the visual communication system 4 performs voice recognition processing on the voice information to convert it into character information (S24), and the character information is the recognition result. If the language information is not voice information (No in S23), the visual communication system 4 determines whether the language information is a handwritten character image (S25). If the language information is a handwritten character image (Yes in S25), the visual communication system 4 performs text recognition processing on the handwritten character image to convert it into character information (S26), and sets the character information as the recognition result. If the language information is not a handwritten character image, that is, if the language information is character information typed on the keyboard 541 (No in S25), the visual communication system 4 sets the character information as the recognition result, and the process proceeds to S27.

The visual communication system 4 accesses the thumbnail storage unit 210 and searches for two-dimensional thumbnail data using the character information that is the recognition result as a keyword (S27). The visual communication system 4 displays one or more pieces of searched two-dimensional thumbnail data in a specific location on the display (S28).

The visual communication system 4 can rapidly repeat the processes of S22 to S28 until a request to turn off the language information input function is received (No in S29). In other words, the visual communication system 4 can rapidly update and display new thumbnail images on the display screen every time language information is received from the user. This allows the user to select one or more thumbnail images that are closest to the scene in their mind when they speak or input text, without having to stop the conversation.

When a request to turn off the language information input function is received (Yes in S29), the visual communication system 4 ends the processing of FIG. 11.

In the process of FIG. 12, the visual communication system 4 waits until 2D thumbnail data is selected from the one or more 2D thumbnail data displayed in S28 (No in S31). When 2D thumbnail data is selected from the one or more 2D thumbnail data displayed in S28 (Yes in S31), the visual communication system 4 accesses the illustration storage unit 212 and searches for 3D illustration data associated with the selected 2D thumbnail data (S32). The visual communication system 4 displays the searched 3D illustration data on the display (S33).

The visual communication system 4 waits until an editing operation is performed on the three-dimensional illustration data (No in S34), and when an editing operation is performed on the three-dimensional illustration data (Yes in S34), it performs a first editing process (S35). The first editing process is a three-dimensional editing process. In the first editing process, the visual communication system 4 accepts three-dimensional rotation operations, three-dimensional movement operations, three-dimensional enlargement operations, three-dimensional reduction operations, and the like. In response to these operation requests, the visual communication system 4 changes the display form of the three-dimensional illustration data displayed on the display (S36). The visual communication system 4 repeats the processes of S34 to S36 until the three-dimensional illustration data is fixed to two-dimensional illustration data (No in S37).

When the three-dimensional illustration data is fixed to two-dimensional illustration data (Yes in S37), the visual communication system 4 waits until an editing operation is performed on the two-dimensional illustration data (No in S38). When an editing operation is performed on the two-dimensional illustration data (Yes in S38), the visual communication system 4 performs a second editing process (S39). The second editing process is a two-dimensional editing process. In the second editing process, the visual communication system 4 accepts two-dimensional rotation operations, two-dimensional movement operations, two-dimensional enlargement operations, two-dimensional reduction operations, partial erasure operations, operations to add hand-drawn lines, coloring operations, and the like. In response to these operation requests, the visual communication system 4 changes the display form of the two-dimensional illustration data displayed on the display (S40). The visual communication system 4 repeats the processes of S38 to S40 until a request to complete editing is received (No in S41). In other words, the first stage of editing processing in the loop of S34 to S37 and the second stage of editing processing in the loop of S38 to S41 can quickly and accurately express an illustration that closely resembles the scene that occurs in the user's mind when the speech or text is entered.

When a request to complete editing is received (Yes in S41), the visual communication system 4 ends the processing of FIG. 12.

Next, the configuration of the screen displayed on the display of the client terminal 1 by the visual communication system 4 (display screen by the visual communication system 4) will be described with reference to FIG. 13. FIG. 13 is a diagram showing the configuration of the display screen by the visual communication system 4.

As shown in FIG. 13, the display screen of the visual communication system 4 includes a drawing result display area S100, an operation palette S200, and a thumbnail display area S300.

The drawing result display area S100 is an area where the results of the drawing display unit 400 (see FIG. 1) are output. The operation palette S200 has functions used to call up each input unit, such as the voice input unit 101, the character input unit 102 (see FIG. 3), and the hand-drawn input unit 304 (see FIG. 8). The thumbnail display area S300 displays the language information recognized by the recognition unit 103 (see FIG. 3) and the two-dimensional thumbnail data output by the thumbnail display unit 203 (see FIG. 4).

The layout and design shape of each area shown in Figure 5 is merely an example and does not limit the scope of rights. For example, eight thumbnails surrounded by circular frames are displayed in S300, but there are no restrictions on the presence or absence of frames, their shape, or the number displayed. In addition, recognized language information does not need to be displayed.

Next, the general operation of the screen displayed on the display of the client terminal 1 by the visual communication system 4 (display screen by the visual communication system 4) will be described with reference to FIG. 14. FIG. 14 is a diagram showing the general operation of the display screen by the visual communication system 4. FIG. 14 shows the operation of calling up two-dimensional thumbnail data based on language information.

When the visual communication system 4 detects an operation of pressing the language information recognition button S201 shown in FIG. 14(a), it transitions to a language information recognition mode. In the language information recognition mode, when the visual communication system 4 acquires language information (first language information) by speech, handwriting, typing, etc., it displays the language information recognition result and one or more pieces of two-dimensional thumbnail data linked to the recognition result in the thumbnail display area S300. In addition, when the visual communication system 4 detects an operation of pressing the two-dimensional data editing button S202, it transitions to a drawing result editing mode in which the drawing result display area S100 in FIG. 13 can be edited.

For example, FIG. 14(a) shows the visual communication system 4 recognizing handwritten characters drawn in the drawing result display area S100, and displaying the character information of the recognition result and one or more pieces of two-dimensional thumbnail data associated with the character information in the thumbnail display area S300.

When the visual communication system 4 further acquires language information (second language information) by speech, handwriting, typing, etc., at least a portion of the one or more two-dimensional thumbnail data displayed in the thumbnail display area S300 is changed, and the language information recognition result and the one or more two-dimensional thumbnail data linked to the recognition result are updated and displayed.

For example, when the visual communication system 4 recognizes an utterance such as that shown in FIG. 14(b) in the state shown in FIG. 14(a), the visual communication system 4 changes the state of the thumbnail display area S300 in real time according to the character information of the recognition result. FIG. 14(b) shows the state in which the visual communication system 4 updates and displays in the thumbnail display area S301 based on the character information of the new recognition result and one or more pieces of two-dimensional thumbnail data associated with it.

Specifically, in accordance with the thumbnail image corresponding to the newly input language information (second language information), at least a portion of the display of the thumbnail image already displayed is changed and displayed on the display (display means).

Changing at least a portion of the display of a thumbnail image that is already displayed means, for example, changing the position or size of a thumbnail image that is already displayed in order to add a thumbnail image that corresponds to newly input language information to the thumbnail image that is already displayed, deleting a thumbnail image that corresponds to newly input language information in order to display it in place of the image that is already displayed, or changing the image around the thumbnail image that is already displayed.

In addition to speech, the visual communication system 4 may change the display in the thumbnail display area S300 each time it recognizes new handwritten or typed characters.

In addition, if the 2D thumbnail data cannot be displayed in the thumbnail display area S300, the visual communication system 4 may display the 2D thumbnail data in the thumbnail display area S300 in order from oldest to newest, replacing the oldest.

In FIG. 14, the switch to switch to the language information recognition mode is the language information recognition button S201, but the method is not limited to this and can be, for example, speaking specific words or inputting commands instead of using a button.

As illustrated in FIG. 14, the visual communication system 4 can quickly update and display new thumbnail images on the display screen every time language information is received from the user. This allows the user to select one or more thumbnail images that are closest to the scene in their mind when they speak or input text, without having to stop the conversation.

Next, the flow of the two-stage illustration editing process by the visual communication system 4 will be explained with reference to FIG. 15. FIG. 15 is a diagram showing the flow of the two-stage editing process.

In FIG. 15(a), the visual communication system 4 displays the character information LI of "person" and the two-dimensional thumbnail data SM1 to SM6 associated therewith in the thumbnail display area S300 in response to the linguistic information "person."

When the two-dimensional thumbnail data SM5 is selected, the visual communication system 4 calls up the three-dimensional illustration data linked to the two-dimensional thumbnail data SM5 and displays it in the drawing result display area S100, as shown in FIG. 15(b).

In three-dimensional editing processing, the visual communication system 4 accepts three-dimensional rotation operations, three-dimensional movement operations, three-dimensional enlargement operations, three-dimensional reduction operations, and the like, and in response to these operation requests, changes the display form of the three-dimensional illustration data three-dimensionally, as shown in FIG. 15(c).

When a predetermined triggering operation (e.g., pressing a button for a two-dimensional editing operation (e.g., the two-dimensional data editing button S202 shown in FIG. 14(a)) is received, the visual communication system 4 fixes the three-dimensional illustration data to two-dimensional illustration data, as shown in FIG. 15(d).

In two-dimensional editing processing, the visual communication system 4 accepts two-dimensional rotation operations, two-dimensional movement operations, two-dimensional enlargement operations, two-dimensional reduction operations, partial erasure operations, operations to add hand-drawn line drawings, coloring operations, and the like, and in response to these operation requests, changes the display form of the two-dimensional illustration data two-dimensionally, as shown in FIG. 15(e).

As shown in FIG. 15, the first stage of editing (FIG. 15(b) and FIG. 15(c)) and the second stage of editing (FIG. 15(e)) can quickly and accurately express an illustration that closely resembles the scene that appears in the user's mind when the speech or text is input.

Next, the detailed operation of the screen (display screen by visual communication system 4) displayed on the display of client terminal 1 by visual communication system 4 will be described with reference to Figs. 16 to 23. Figs. 16 to 23 are diagrams each showing the detailed operation of the display screen by visual communication system 4. Fig. 14 shows the operation of calling up two-dimensional thumbnail data based on language information, selecting a thumbnail, and performing illustration editing.

When it is detected that the voice input button S2011 in the operation palette S200 shown in FIG. 16(a) has been pressed, the visual communication system 4 turns on the language information recognition function, and displays an icon S3021 indicating that voice input is in a standby state in the thumbnail display area S300, as shown in FIG. 16(b). When the key input button 2012 in the operation palette S200 is pressed, typing input via the keyboard becomes possible, and an icon indicating that typing input is in a standby state is displayed. These icons S3021 and the icon indicating that typing input is in a standby state are examples of language information recognition state display icons.

When it is detected that "people and vehicles" has been spoken, the visual communication system 4 displays the text information LI1 of "people and vehicles" and the two-dimensional thumbnail data SM11 to SM18 linked to it in the thumbnail display area S300, as shown in FIG. 17. Note that the thumbnail data linked to the text information may be thumbnail data linked to the entire text information of "people and vehicles," or may be a group of thumbnail data broken down into keywords such as "people" and "vehicles" and linked to each keyword.

When it detects that "animals" has been further uttered, the visual communication system 4 updates and displays the text information LI2 of "people and vehicles and animals" and the two-dimensional thumbnail data SM21 to SM28 linked to it in the thumbnail display area S300, as shown in FIG. 18. That is, each time the visual communication system 4 detects the input of linguistic information, it updates and displays the two-dimensional thumbnail data in real time. Note that the thumbnail data linked to the text information may be thumbnail data linked to the entire text information of "people and vehicles and animals", or it may be a group of thumbnail data linked to each keyword that has been broken down into keywords such as "people", "vehicles", and "animals".

As illustrated in Figures 17 and 18, the visual communication system 4 can quickly update and display new thumbnail images on the display screen every time language information is received from the user. This allows the user to select one or more thumbnail images that are closest to the scene in their mind when they speak or input text, without having to stop the conversation.

When it is detected that the 2D thumbnail data SM14 has been selected on the screen of FIG. 17, the visual communication system 4 calls up the 3D illustration data IL1 linked to the 2D thumbnail data SM14 and displays it in the drawing result display area S100, as shown in FIG. 19.

In the three-dimensional editing process, the visual communication system 4 accepts three-dimensional rotation operations, three-dimensional movement operations, three-dimensional enlargement operations, three-dimensional reduction operations, and the like, and in response to these operation requests, changes the display form of the three-dimensional illustration data IL1 three-dimensionally, as shown in FIG. 20.

When the visual communication system 4 detects that any of the buttons for two-dimensional editing (the add button S2013, the color button S2014, the eraser button S2015) has been pressed, the three-dimensional illustration data IL1 is fixed to two-dimensional illustration data IL2, as shown in FIG. 21. As an example, in this embodiment, the system determines that the button for two-dimensional editing has been pressed as a request to turn off the language recognition function, and transitions to a state in which two-dimensional editing is possible.

In two-dimensional editing processing, the visual communication system 4 accepts two-dimensional rotation operations, two-dimensional movement operations, two-dimensional enlargement operations, two-dimensional reduction operations, partial erasure operations, operations to add hand-drawn line drawings, coloring operations, and the like, and in response to these operation requests, changes the display form of the two-dimensional illustration data IL2 two-dimensionally as shown in Figures 21, 22, and 23.

In FIG. 21, the retouch button S2013 is used to add handwritten line drawings of the scenery that will form the background of the two-dimensional illustration data IL2, and the color button S2014 is used to add color to the three-dimensional illustration data IL2.

In addition, in Fig. 21, the language information recognition status display icon, character information, and thumbnail data that were displayed in the thumbnail display area S300 in Figs. 19 and 20 are not displayed. The timing for hiding the various information in the thumbnail display area S300 can be appropriately selected, such as when it is detected that any of the buttons for two-dimensional editing processes (the Add button S2013, the Color button S2014, the Eraser button S2015) has been pressed, when fixation is performed, when two-dimensional editing processes are started by the user, etc.

Even after performing two-dimensional editing, the voice input button S2011 or the key input button S2012 can be pressed to turn on the language information recognition function and display various information in the thumbnail display area S300, allowing new three-dimensional illustration data to be displayed and edited on the image that has been two-dimensionally edited, and new fixed two-dimensional illustration data to be added.

In FIG. 22, the eraser button S2015 is used to delete part of the line drawing in the two-dimensional illustration data IL2'.

In FIG. 23, line art is added to the two-dimensional illustration data IL2" by using the Add button S2013.

As shown in Figures 19 to 23, the first stage of editing (Figures 19 and 20) and the second stage of editing (Figures 21 to 23) allow an illustration that closely resembles the scene that occurs in the user's mind when the speech or text is input to be accurately expressed in a short period of time.

As described above, in this embodiment, the visual communication system 4 searches for and displays illustrations that correspond to the language information received from the user, and enables a two-stage editing process while changing the display form of the displayed illustration. This can support accurate information transmission in communication situations.

The concept of the visual communication system 4 may be applied to an online conference system 24. The online conference system 24 may be configured as shown in FIG. 24. FIG. 24 is a diagram showing the configuration of an online conference system 24 according to a modified example of the embodiment, and shows an example in which an online conference tool is used as the drawing display unit 400.

The online conference system 24 has multiple client terminals 21a, 21b, a communication management server 22, and a connection unit 23. The connection unit 23 connects the multiple client terminals 21a, 21b and the communication management server 22 so that they can communicate with each other. The multiple client terminals 21a, 21b can share a display screen realized by the drawing display unit 400.

Each client terminal 21a, 21b has a language information input unit 100a, 100b, an illustration display unit 200a, 200b, and a drawing operation unit 300a, 300b. The functions and operations of the language information input unit 100a, 100b, the illustration display unit 200a, 200b, and the drawing operation unit 300a, 300b are similar to the functions and operations of the language information input unit 100, the illustration display unit 200, and the drawing operation unit 300 in the embodiment, respectively.

The communication management server 22 has a communication management unit 600 in addition to a drawing display unit 400 and a memory unit 500. The functions and operations of the drawing display unit 400 and the memory unit 500 are similar to those of the drawing display unit 400 and the memory unit 500 in the embodiment, respectively.

The communication management unit 600 transmits the audio and camera images received from each of the multiple client terminals 21a and 21b, which are conference participants, to the other client terminals and manages them. The drawing display unit 400 draws and displays the images by superimposing them on the camera images of the online conference. The communication management unit 600 transmits the superimposed images to the other client terminals.

The overlay may be applied to a portion of the image, or to the entire image. In addition to the drawn image, S200, S300, etc. may also be overlaid at the same time.

For example, as shown in Figs. 25 to 27, a camera image showing a user and a drawing made by a gesture of the user acquired by a motion acquisition device can be superimposed. Fig. 25 is a diagram showing a camera image captured by the camera 520 of the client terminal 21a in a modified embodiment, and illustrates a camera image showing the user of the client terminal 21a. Fig. 26 is a diagram showing an image superimposed on the camera image of Fig. 25 by the drawing display unit 400 in a modified embodiment. Fig. 26 shows an example in which the drawing result display area S100 and the thumbnail display area S300 are superimposed, but the operation palette S200 is not superimposed, among the drawing result display area S100, the operation palette S200, and the thumbnail display area S300. Fig. 27 is a diagram showing an image displayed on another client terminal 21b in a modified embodiment. In this case, handwriting with the fingertip of a user on his/her display at a certain client terminal is displayed as an image in which a drawing is superimposed on the camera image by the user's fingertip at the other client terminal.

In this way, visual communication using illustrations can be achieved through gesture control while talking.

In the embodiment described above, the visual communication system 4 and the online conference system 24 are examples of an image editing system or an image display system. Also, the client terminal 1, the client terminal 21a, the client terminal 21b, the server 2, and the communication management terminal 22 are examples of an image editing device or an image display device.

Each function of the embodiments described above can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and devices such as an ASIC (Application Specific Integrated Circuit), DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array), and conventional circuit modules designed to execute each function described above.

The correspondence table in the specification may be one generated by the learning effect of machine learning. Here, machine learning is a technology for enabling a computer to acquire human-like learning capabilities, and refers to a technology in which a computer autonomously generates algorithms required for judgments such as data identification from learning data that is previously loaded, and applies these to new data to make predictions. The learning method for machine learning may be any of supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and deep learning, or may be a combination of these learning methods; any learning method for machine learning is acceptable.

The program 500a executed by the visual communication system 4 or the online conference system 24 may be provided in advance in a ROM or the like. Alternatively, the program 500a may be provided by recording it in an installable or executable file format on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a digital versatile disk (DVD). Alternatively, the program 500a may be provided by being stored on a computer connected to a network such as the Internet and downloaded via the network. The program 500a may also be provided or distributed via a network such as the Internet.

Reference Signs List 1 Client terminal 2 Server 3 Connection unit 4 Visual communication system 21a, 21b Client terminal 22 Communication management server 23 Connection unit 24 Online conference system

Patent No. 6339529

Claims (9)

a first input means for receiving voice input information, text input information or motion input information;
a recognition means for recognizing character information from the voice input information, the text input information, or the action input information;
a search means for searching for a plurality of thumbnail images, each of which is associated with image data, by using the recognized character information;
a display means for displaying the retrieved thumbnail images;
a first editing means for performing a first editing process on first image data that is associated with a thumbnail image selected from among the plurality of thumbnail images displayed on the display means and has a first data format;
a second editing means for performing a second editing process on second image data having a second data format different from the first data format among the image data displayed on the display means;
a conversion means for converting the first data format into the second data format,
The display means includes:
when the first editing process is performed by the first editing means, a display of the first image data is changed to a display of image data after the first editing process;
when the second editing process is performed on the second image data converted by the conversion means, a display of the second image data is changed to a display after the second editing process.
Image editing device. When none of the plurality of thumbnail images displayed on the display means is selected, and the first input means receives second voice input information, second text input information, or second action input information, and second character information is recognized by the recognition means for the second voice input information, the second text input information, or the second action input information, the search means searches for a plurality of second thumbnail images each associated with image data, using the recognized second character information, and the display means displays the searched second thumbnail image in place of the plurality of thumbnail images displayed on the display means.
The image editing device according to claim 1 . a second input means for receiving a selection operation for designating a thumbnail image to be selected from the plurality of thumbnail images;
a selection means for selecting image data linked to a thumbnail image designated by the selection operation from among a plurality of image data linked to a plurality of thumbnail images displayed on the display means;
Further equipped with
The first editing means performs the first editing process on the first image data when the first image data is selected by the selection means from among a plurality of image data linked to a plurality of thumbnail images displayed on the display means.
The image editing device according to claim 1 . 2. The image editing device according to claim 1, wherein the conversion means converts the first data format into the second data format in response to a state in which the second editing process by the second editing means is enabled. the first editing means performs the first editing process on the three-dimensional image data displayed on the display means;
the second editing means performs the second editing process on the two-dimensional image data displayed on the display means;
4. The image editing device according to claim 1 , wherein the conversion means converts a three-dimensional image data format into a two-dimensional image data format. 6. The image editing device according to claim 5, wherein the conversion means determines a plurality of planar coordinates projected onto a plane corresponding to the display means from a plurality of spatial coordinates contained in the three-dimensional image data, and generates two -dimensional image data including the plurality of planar coordinates. the first editing process includes a process corresponding to a three-dimensional editing operation;
The image editing device according to claim 1 , wherein the second editing process includes a process corresponding to a two-dimensional editing operation. receiving voice or text input information;
recognizing character information from the voice information or the text input information;
using the recognized character information to search for a plurality of thumbnail images, each of which is associated with an image data;
displaying the retrieved thumbnail images on a display means;
a step of performing a first editing process on first image data that is associated with a thumbnail image selected from the plurality of thumbnail images displayed on the display means and has a first data format;
a step of changing a display of the first image data to a display of image data after the first edit processing when the first edit processing is performed in the step of performing the first edit processing;
converting the first data format into a second data format;
a step of performing a second editing process on second image data having the second data format among the image data displayed on the display means;
a step of changing a display of the second image data to a display after the second editing process when the second editing process is applied to the second image data converted in the converting step in a step of applying the second editing process;
Image editing methods including. receiving voice or text input information;
recognizing character information from the voice information or the text input information;
using the recognized character information to search for a plurality of thumbnail images, each of which is associated with an image data;
displaying the retrieved thumbnail images on a display means;
a step of performing a first editing process on first image data that is associated with a thumbnail image selected from the plurality of thumbnail images displayed on the display means and has a first data format;
a step of changing a display of the first image data to a display of image data after the first edit processing when the first edit processing is performed in the step of performing the first edit processing;
converting the first data format into a second data format;
a step of performing a second editing process on second image data having the second data format among the image data displayed on the display means;
a step of changing a display of the second image data to a display after the second editing process when the second editing process is applied to the second image data converted in the converting step in a step of applying the second editing process;
An image editing program that causes a computer to execute the following:

Images