JP4239970B2 - Information terminal equipment - Google Patents

Description

  The present invention relates to an information terminal device. The information terminal device of the present invention relates to an information terminal device such as a videophone or a video conference that transmits a created image based on a face image captured by a user as an image with a facial expression.

  In image communication, it is known that the user's camera image affects the quality of communication with each other. Conventionally, several methods have been adopted in this field. As a first method, there is a method of displaying an image of the user himself / herself. In this method, a picture-in-picture method for displaying as a small window in a screen displaying an image from the other party and a screen division method for displaying the screen in one region divided into a plurality of regions have been used.

  As a second method, there is a user interface of the image communication terminal device of Patent Document 1, and this device performs various operations on the received counterpart image when the user is outside the shooting range of the camera. By changing and displaying the display state later, it is possible to determine whether or not the user is within the shooting range of the camera without displaying the local image, and that the user has deviated from the shooting range of the camera. A method of notification is disclosed.

JP-A-8-251561

  However, in the display by the two methods mentioned in the first method described above, it is difficult to confirm the shooting situation because only the user's own image is displayed, and the adjustment of the shooting situation also allows the user to see the image actually displayed. There is a problem that there is nothing but to judge.

  In addition, when the user is within the shooting range in the second method, there is a problem that it is impossible to confirm what position the user is in. This method makes it possible to adjust the position of the user within the shooting range, but receives and displays only the image of the other party without displaying the user image. Other shooting conditions such as the distance between users cannot be confirmed.

  An object of the present invention is to provide an information terminal device that can eliminate such drawbacks of the prior art, easily realize confirmation of shooting conditions, and improve user convenience.

  In order to solve the above-described problems, the present invention provides an image transmission system that communicates using an information terminal device that handles images as communication information. This device captures a subject as a video signal and outputs it as image data. Means, feature extraction means for extracting feature data representing an expression from image data, and expression synthesis means for generating a character composite image by deforming character data as a basic image selected by a user based on the feature data , Transmission / reception means for transmitting / receiving data to / from the communication partner information terminal device, output means for displaying or outputting image data from the communication partner information terminal device, and what kind of situation the user is photographing based on the characteristic data Guide information generating means for generating advice information indicating the expression information, and the expression synthesis means reflects the advice information. It generates a character synthesis image, and outputs the character synthesis image generated from the transmitting and receiving means.

  The information terminal device of the present invention outputs the subject captured by the image input means to the feature extraction means as image data, generates advice information based on the feature data by the guide information generation means, and the advice information is output by the facial expression synthesis means. By generating the reflected character composite image and outputting it via the transmission / reception means, the user can check the shooting situation with the generated character composite image.

  In order to solve the above-described problems, the present invention is an image transmission system that communicates using an information terminal device that handles images as communication information. This device takes a subject as a video signal and outputs it as image data. Image input means, feature extraction means for extracting feature data representing facial expressions from image data, transmission / reception means for transmitting / receiving data to / from a communication partner information terminal device, and display or output of image data from the communication partner information terminal device Output means, guide information generating means for generating advice information indicating what kind of shooting situation the user is based on the characteristic data, and notification converting means for converting the advice information into notification information, and output means Is characterized by receiving notification information and outputting it.

  The information terminal device of the present invention outputs the subject captured by the image input means to the feature extraction means as image data, generates advice information based on the feature data by the guide information generation means, and provides advice information by the notification conversion means. By converting to notification information and receiving and outputting the notification information by the output means, the user's own shooting situation can be easily grasped.

  Next, an embodiment of an information terminal device according to the present invention will be described in detail with reference to the accompanying drawings.

  In this embodiment, the information terminal device of the present invention is applied to the communication terminal device 10. The illustration and description of parts not directly related to the present invention are omitted. In the following description, the signal is indicated by the reference number of the connecting line in which it appears.

  The communication terminal device 10 is connected to the Internet or the like via a communication line. The communication terminal device 10 may include the same components or different components depending on the communication form. It will be disclosed sequentially in the following examples.

  As shown in FIG. 1, the communication terminal device 10 includes an image input unit 12, a changeover switch 14, a facial expression feature extraction unit 16, a projection output unit 18, a facial expression synthesis unit 20, a guidance information generation unit 22, an operation unit 24, a character A management unit 26, a notification control unit 28, and a transmission / reception unit 30 are included.

  The image input unit 12 condenses incident light 32 from a subject through a two-dimensionally arranged light receiving element through an optical system (not shown), and converts the analog image signal thus obtained into a digital image by photoelectric conversion. It has a function of converting it into an image signal and taking it into the communication terminal device 10 as image data. The image input unit 12 sets the user's face as a subject and outputs the face image as image data 34 to the changeover switch 14.

  The changeover switch 14 has a function of switching the output destination of the supplied image data 34. Although not shown, the change-over switch 14 outputs the image data 34a and 34b, for example, to either the facial expression feature extraction unit 16 or the sound output unit 18 according to the selection of the operation unit 24, for example.

  The facial expression feature extraction unit 16 has a function of obtaining feature data representing a facial expression based on the supplied image data 34a. The facial expression feature extraction unit 16 outputs the obtained feature data 36 to the facial expression synthesis unit 20 and the guidance information generation unit 22, respectively. The facial expression feature extraction unit 16 extracts emotion data based on voice data (not shown), and outputs facial expression data 36 for displacing or moving the facial part so as to reflect the extracted emotion data and feature data. Although not shown, the facial expression feature extraction unit 16 has a function of reducing an image in accordance with an operation instruction signal from the operation unit 24 and outputting the image together with sound to the projection output unit 18 as a projection signal.

  The projection sound output unit 18 has a function of outputting an image and sound (not shown). The projection sound output unit 18 includes a monitor 38 and a notification output unit 40. On the monitor 38, for example, an image from a camera, a character image indicating a basic model, a background image, a preview image, a character image with expression, and the like are displayed. The notification output unit 40 displays, for example, the selected transmission image, facial expression extraction certainty, guidance content, and the like.

  The facial expression synthesizer 20 receives the feature data 36 representing the facial expression and the character image 42, which is the selected basic model, and moves the feature points of the character image of the basic model image to generate an image reflected in the facial expression. It has a function. This function can be realized by application software. This image becomes a two-dimensional image or a three-dimensional image according to the supplied character image. The facial expression composition unit 20 outputs the character image data 44 reflecting the facial expression of the user (speaker) to the transmission / reception unit 30.

  The guidance information generation unit 22 has a function of generating guidance information based on the obtained feature data 36, and includes a guidance information determination unit 46 as shown in FIG. The guidance information is information representing each of a current determination result obtained by determining success or failure of the extraction process based on the feature data 36 that is a result of the extraction process and an extraction process success rate for a certain period. The guidance information generation unit 46 outputs the generated guidance information 48 to the notification control unit 28.

  Returning to FIG. 1, the operation unit 24 supplies an operation instruction signal 50 selected by a user operation to a control unit and a character management unit 30 (not shown).

  The character management unit 26 downloads character image data and background data from the character management center 16 via the transmission / reception unit 34 from the signal line 52 and stores them, and stores thumbnail data 54 of the character image data and background data as a projection sound output unit. It has a function to output to 18. This is for use in confirming a final image obtained by combining these when the user selects character image data and background data. The user selects character image data and background data according to the request by an operation instruction signal 50 from the operation unit 24, and outputs the selected character image data and background data to the facial expression synthesis unit 16 as content 42.

  The character management unit 26 does not always have to manage the character image data and the background data, and may only manage either the character image data or the background data.

  The notification control unit 28 has a function of generating information 56 to be notified to the user from the supplied guidance information 48. The notification control unit 28 outputs the notification information 56 to the projection sound output unit 18. The notification information 56 is information corresponding to the number of lamps to be lit corresponding to the stage of the guidance information 48 to be supplied. As shown in FIG. 3, the notification control unit 28 associates the guidance information 48 and the notification information 56 with, for example, the extraction process success rate and the number of lamps to be lit.

  The sound output unit 18 described above displays the state of the extraction process according to the notification information 56 supplied. For example, as shown in FIG. 4, in addition to a character string display unit 58 for displaying character strings such as “success” and “failure” based on the determination result, and a shape such as a figure in addition to symbols “O” and “X”. It is preferable to provide a symbol display unit 60 for displaying and a stage display unit 62 for notifying the situation with these sizes. The character string or figure may be notified by changing the color. In addition, the user may be notified by voice output. For example, in addition to outputting a beep sound, a character or character string such as “success”, “failure”, “○” or “×” may be read out and output, or a pre-registered character or character string may be output. It may be read out and output. It is desirable that audio output can be arbitrarily set to output / non-output.

  Returning to FIG. 1 again, the transmission / reception unit 30 has a function of transmitting the projection data 44 from the facial expression synthesis unit 20 and receiving the projection data sent from a communication terminal device (not shown). The received sound data 64 is output to the sound output unit 18 as the sound data 66. Further, as described above, the transmission / reception unit 30 has a function of performing data communication between the character management unit 26 and a character management center (not shown).

  The operation of image communication in the communication terminal apparatus 10 shown in FIG. 1 will be briefly described. The user uses the communication terminal device 12 to capture a face image. As shown in FIG. 5, the communication terminal device 10 acquires a face image via the image input unit 12 (step S10). Image data 34 of the face image is supplied to the facial expression feature extraction unit 24 via the changeover switch 14.

  Next, the facial expression feature extraction unit 16 determines whether or not to transmit an image to the communication partner (step S12). Although not shown, this determination is made in response to the control signal. The control signal is generated based on, for example, an operation instruction signal 50 supplied from the operation unit 24 in response to an operation by the user. When the image data 34 is not transmitted immediately by the changeover switch 14 (YES), the image data 34 is output to the facial expression feature extraction unit 16 and proceeds to extraction processing (to step S14). If the image data 34 is to be transmitted immediately (NO), the process proceeds to the image data 34 transmission process (not shown in FIG. 1) (to step S16). At this time, the communication terminal device 10 may supply the image data 34 to the monitor 38 and display the face image on the monitor 38.

  In step S14, facial features are extracted from the image data 34. Next, it is determined whether or not to generate a character image (step S18). This determination is also made according to the operation. If a character image is to be generated (YES), the process proceeds to image generation processing (to step S20). If a character image is not generated (NO), the process proceeds to guidance information generation processing (to step S22).

  In the image generation process, the facial expression synthesis unit 20 generates a character image (step S20). A character image with a background reflecting a facial expression in the character image and the background content 42 is used. In this process, guidance information 48 is also generated based on the supplied feature data 36.

  In the guidance information generation process, the guidance information generation unit 22 generates an extraction process success rate and the like, and outputs it to the notification control unit 28. Although the guidance information generation process is shown separately as a procedure, it may be performed when generating a background character image. Thereby, the user can confirm the photographing situation.

  Next, the generated character image with facial expression or face image data is transmitted (step S16). The image output unit 18 displays the supplied image data including the received image data 66 and the like (step S24). After this display, the process is terminated.

  On the other hand, the notification control unit 28 generates notification information 56 corresponding to the supplied guidance information 48 and outputs the notification information 56 to the projection sound output unit 18 (step S26). The user responds, for example, how to acquire the next image according to the notification displayed on the sound output unit 18. In this notification, not only the number of blinking lamps but also the audio output may be increased or decreased in stages according to the extraction processing success rate. After this display, the process is terminated.

  In this way, by displaying the notification information 56 corresponding to the guidance information 48 and notifying the user of the shooting status, the user can easily check the shooting status at the time of use, reducing the burden on the user. can do. In addition, even if the user is appropriately shooting with the camera, the user can check the shooting situation, so the convenience can be greatly enhanced.

  Next, as another embodiment of the communication terminal device of the image communication system 10 to which the information terminal device of the present invention is applied, only the notification display by guidance will be described. In addition, common constituent elements and steps are denoted by the same reference numerals, and description thereof is omitted to avoid complicated description. As shown in FIG. 6, the communication terminal apparatus 10 of the present embodiment is provided with a storage 68 in the configuration of FIG. The storage 68 stores a guidance database for managing the correspondence of notification information. The storage 68 is connected to the notification control unit 28 via a signal line 70.

  Further, not only the feature data 36 but also the image data 34 are supplied to the guidance information generation unit 22. As shown in FIG. 7, the guidance information generation unit 22 includes an extraction data determination unit 72, an image brightness / darkness determination unit 74, a face distance determination unit 76, an open / close determination unit 78, a face angle determination unit 80, and an integration determination unit 82. The extracted data determination unit 72, the face distance determination unit 76, the open / close determination unit 78, and the face angle determination unit 80 each input the supplied feature data 36, and output the data 84 as a determination result to the integrated determination unit 82. Yes. The image contrast determination unit 74 receives the supplied image data 34 and outputs data 84 as a determination result to the integrated determination unit 82.

  The extracted data determination unit 72 corresponds to the guidance information determination unit 46. The image contrast determination unit 74 performs various determinations such as determination by calculating a histogram for an arbitrary region based on the extraction processing result from the captured image data 34, determination by a value obtained by adding the density value of the arbitrary region and dividing by the number of pixels. By applying various methods, it is possible to determine the contrast of an image. By this determination, the user can grasp in advance the brightness of an appropriate shooting environment.

  The face distance determination unit 76 makes a determination based on the distance between a plurality of free combinations such as the left eye and right eye, the left mouth edge and the right mouth edge, and the size of the face area based on the extraction process result. An appropriate face distance of the user can be determined by applying various methods such as determination. By this determination, the user can grasp an appropriate distance from the image input unit 12 in advance.

  The open / close determination unit 78 applies various methods such as determination based on variation of the density value in the eye region and determination based on template matching from the feature data 36 that is the extraction processing result and the captured image data 34, and the left eye, Whether the right eye is open or closed can be determined. This determination improves the face extraction accuracy by notifying the user in advance that the eyes are closed when it is thought that face feature extraction cannot be performed stably. Can be done. In this case, the open / close determination unit 78 may be connected to input the image data 34.

  The face angle determination unit 80 calculates the distance between the eyes and the corners of the left and right eyes of the feature data 36 and the distance between the left edge and the right edge of the mouth, the distance between the upper and lower eyelids of each of the left and right eyes, and the distance between the eyebrows. Based on the calculation of distance from the center of the mouth, the inclination of the straight line connecting the pupils of the left eye and the right eye and the horizontal, and the inclination of the straight line connecting the left end of the mouth and the right end of the mouth and the horizontal, and the result of the extraction process The face angle can be determined by applying various methods such as determination based on the inclination of the face region. By this determination, it is possible to perform appropriate communication by informing the user in advance.

  The integrated determination 82 has a function of integrating the determination results 84 described above, and generates guidance information 48. The integrated determination unit 214 outputs the guidance information 48 to the notification control unit 28.

  The notification control unit 28 generates notification information 56 using the guidance DB of the storage 60 based on the received guidance information 48. The notification control unit 28 transmits to the sound output unit 18. The sound output unit 18 can display a notification corresponding to the received notification information 56. In the guidance DB, for example, correspondence between the guidance information 48 and the notification information 56 shown in FIG. 8 is managed.

  The operation of this embodiment will be briefly described. As shown in FIG. 9, the operation procedure is a step of generating notification information 56 using the guidance DB of the storage 68 based on the guidance information 48 between step S22 and step S26 shown in FIG. 5 (step S28). Is provided.

  By using the guidance DB to generate and display the notification information 56 and providing appropriate guidance to the user, the user can know the shooting situation in detail when using the facial expression image communication system. The burden can be greatly reduced. Further, in the case of an inappropriate shooting situation, not only the cause is notified, but also the shooting situation can be grasped even when the shooting is properly performed, so the convenience can be greatly improved.

  Next, another embodiment of the communication terminal device 10 of the image communication system to which the information terminal device of the present invention is applied will be described. The communication terminal device 10 of the present embodiment includes a storage 86 in the character management unit 26 as shown in FIG. The storage 86 stores the basic model character data and background data in a database. The character management unit 26 outputs the content 42 to the facial expression synthesis unit 20 in response to a request from the user.

  The communication terminal device 10 shown in FIG. 10 is a case where communication is performed with the communication terminal device 88 of the communication partner via the IP network 90. The communication terminal device 88 includes at least a transmission / reception unit 92, a notification control unit 94, a storage 96, and a sound output unit 98. The transmission / reception unit 92, the notification control unit 94, and the sound output unit 98 in the communication terminal device 88 have the same functions as the transmission / reception unit 30, the notification control unit 28, and the sound output unit 18 used in the communication terminal device 10. .

  The communication terminal devices 10 and 88 can transmit and receive guidance information. That is, the transmission / reception units 30 and 92 have a function of handling data in a format including image information and guidance information. The transmission / reception unit 30 receives the guidance information 48 from the notification control unit 28 and outputs guidance information from the other party to the notification control unit 28.

  The storage 96 is a guidance database that stores guidance information and notification information in association with each other and has a management function.

  The transmission / reception unit 92 supplies the guidance information 100 received from the communication terminal device 10 to the notification control unit 28. The notification control unit 28 accesses the notification information corresponding to the guidance information 100 to the guidance DB of the storage 96, acquires the corresponding notification information 102, and outputs the generated notification information 104 to the projection sound output unit 98.

  The operation of communication terminal apparatus 10 is shown in FIG. First, a captured image is acquired (step S10). Next, it is determined whether or not to transmit (step S12). Regardless of transmission, facial expression features are extracted as feature data 36 (step S14). When transmitting (NO), guidance information 48 is generated based on the extracted feature data 36 (step S22). Along with the image data 42 to be transmitted, the guidance information 48 is transmitted to the communication terminal device 88 (step S30). After this transmission, the same image as the image transmitted to the sound output unit 18 is displayed (step S24). Thereafter, the process proceeds to a notification information generation process (to step S28).

  On the other hand, if it is determined whether or not transmission is possible, and if it is not immediately transmitted (YES), feature data 36 is generated (step S14). Next, it is determined whether or not to generate a character image (step S18). If a character image is to be generated (YES), the process proceeds to character image generation processing (to step S20). If a character image is not generated (NO), the process proceeds to guidance information generation processing (to step S22).

  In the character image generation process, the character image selected by the user is given a facial expression, and a combined image combined with the selected background is generated as a character image (step S20). The feature data 36 is supplied not only to the facial expression synthesis unit 20 but also to the guidance information generation unit 22. The guidance information generation unit 22 generates guidance information 48 based on the feature data 36 and the image data 34.

  The communication terminal device 10 supplies the generated character image and guidance information to the transmission / reception unit 30 from each of the facial expression synthesis unit 20 and the notification control unit 28, and transmits it to the communication terminal device 88 (step S32). After the transmission, the communication terminal apparatus 10 displays a character image on the projection sound output unit 18 (step S34).

  In the determination that the character image is not generated (NO), the guidance information 48 is generated as described above. The generated guidance information 48 is transmitted from the notification control unit 28 to the communication terminal device 88 via the transmission / reception unit 30 (step S36).

  After step S24, S34 or S36, the notification control unit 28 generates notification information 56 based on the supplied guidance information 48. The generated notification information 56 is output to the sound output unit 18 (step S28). The projection output unit 18 displays the supplied notification information 56 on the notification output unit 40 (step S26). After operating in this way, the processing of the communication terminal device 10 ends.

  As shown in FIG. 12, the communication terminal device 88 receives information, images, character images, and guidance information from the communication terminal device 10 (step S38). Of the received information, guidance information 100 is output to the notification control unit 94. Next, notification information corresponding to the guidance information 100 is generated (step S28). The notification information 104 is generated by reading the notification information corresponding to the guidance information 100 from the guidance DB of the storage 96 by the access from the notification control unit 94 and outputting it to the sound output unit 98.

  The projection output unit 98 displays the supplied notification information 104 (step S26).

  In image communication using a character image, for example, even if a captured image from a camera is not transmitted to a communication partner, and only a character image is transmitted, the user of the communication partner is notified by notifying the guidance information 100. The presence of the communication partner can be confirmed, the anxiety about whether or not the communication partner is actually can be removed, and the communication can be concentrated.

  Next, a configuration example of a main part will be described as another embodiment of the communication terminal device 10 of the image communication system to which the information terminal device of the present invention is applied. In the present embodiment, the communication terminal device 10 causes the facial expression synthesis unit 20 to input guidance information 48 and newly provides a storage 106, as shown in FIG. The storage 106 stores a model synthesis database (DB). The model synthesis DB is a database that stores guidance information 48 and model synthesis information in association with each other. The model synthesis information means information that is reflected in the generation of the character image. The storage 106 outputs the model synthesis information 108 read corresponding to the guidance information 48 to the facial expression synthesis unit 20.

  In the communication terminal device 10, the facial expression composition unit 20 uses the model composition DB by this connection, and generates the character image after reflecting the received model composition information in the same manner as the received guidance information.

  Here, the monitor 38 of the projection sound output unit 18 has at least a liquid crystal display or the like, and also includes an audio output device such as a speaker, so that an audio signal corresponding to the guidance information 48 is added to the character image. , The added sound can be output.

  Here are some ways to reflect guidance information in model synthesis information. For example, when the guidance information 48 indicates the darkness of the shooting situation, the entire image is expressed as blackish, and the selected character is expressed as being in a dark place. In addition, when the guidance information 48 indicates a shooting situation away from the camera, that is, far away, the character is displayed in a small size so that the character is in a far place. In addition, as model synthesis information, an object with a special expression effect such as a character background change corresponding to some guidance information, tears, sweating, or display of a character string in a balloon, or a sound effect sound, etc. May be added to the character image.

  The operation will be briefly described. As shown in FIG. 14, the communication terminal apparatus 10 supplies the guidance information 48 to the facial expression composition unit 20 after obtaining the guidance information 48. The facial expression synthesis unit 20 accesses the storage 106 to the guidance information 48 and acquires the corresponding model synthesis information 108 (step S40).

  The facial expression synthesis unit 20 generates an image so that the model synthesis information 108 is reflected on the basic character image (content) 42 supplied in accordance with the selection. The facial expression synthesis unit 20 outputs the generated character image to the transmission / reception unit 30. In the transmission / reception unit 30, the face image or the generated character image is transmitted to the communication terminal device 88 (step S16). Although not clearly appearing as a signal line in FIG. 13, the communication terminal device 10 displays the face image or the generated character image on the sound output unit 18 (step S24). After this display, one image communication operation is terminated.

  By generating the character image based on the model synthesis information reflecting the guidance information, the user can confirm the shooting state together with the generation of the character image, and the burden on the user can be greatly reduced. Moreover, the display part which displays the notification accompanying guidance information becomes unnecessary, and the user can focus on communication more because the visual attention location reduces.

  Next, another embodiment of the communication terminal device 10 of the image communication system to which the information terminal device of the present invention is applied will be described using the main configuration. In this embodiment, the components of the communication terminal device 10 are simplified as much as possible, and the server 110 is provided with a character image generation function. The server 100 may be provided in a management server such as a character management center.

  As shown in FIG. 15, the server 110 has the same reference numerals as those used in the communication terminal device 10. The server 110 includes a transmission / reception unit 30, a facial expression feature extraction unit 16, a facial expression synthesis unit 20, a guidance information generation unit 22, and a storage 106. In particular, the storage 106 stores not only the model synthesis DB but also a character image and a background. Therefore, the storage 106 outputs the data 108 including the character image and background according to the user's selection and the model synthesis information corresponding to the guidance information 48 to the facial expression synthesis unit 20.

  The facial expression synthesis unit 20 generates a character image according to the supplied data 108. The generated character image is supplied from the transmission / reception unit 30 of the server 110 to the communication terminal devices 10 and 88 via the IP network 90. The communication terminal device 10 in FIG. 15 receives the supplied character image 66 and outputs it to the sound output unit 18. In the sound output unit 18, for example, a character image selected by the communication partner and the user is displayed on the monitor 38.

  The operation of the communication terminal apparatus 10 in the image communication system is shown in FIG. The communication terminal device 10 acquires the captured image data 34 (step S10). It is determined whether or not to transmit the image data 34 (step S12a). When not transmitting (NO), the image data 34 is sent to the sound output unit 18 via the changeover switch 14, and the process proceeds to the display process (to step S24). In the case of transmission (YES), the image data 34 is transmitted from the transmission / reception unit 30 to the server 110 (step S42). With this transmission, the image communication system proceeds to management processing (to subroutine SUB).

  The management process is a process in the server 110. As a result, a character image is generated and transmitted to the communication terminal device 88 of the communication partner not shown in FIG. Server 110 also receives image data from communication terminal device 88 and generates a character image. The server 110 transmits at least a character image and guidance information supplied from a communication partner among the generated character images. The server 110 may return the character image selected by the user.

  Communication terminal apparatus 10 receives the character image and guidance information supplied from server 110 (step S44). The communication terminal device 10 supplies the received character image to the projection sound output unit 18 and displays it (step S24). When communication is performed through the server 110 as described above, image communication can be easily realized by a communication terminal device having a simplified configuration.

  FIG. 17 shows the management process as a subroutine. The server 110 receives the image data 34 taken by each communication terminal device (step S38). Next, based on the received image data 34, feature data 36 indicating facial features is extracted by the facial feature extraction unit 16. The feature data 16 is supplied to the facial expression synthesis unit 20 and the guidance information generation unit 22, respectively.

  The guidance information 22 is generated by the guidance information generation unit 22 based on the image data 34 and the feature data 36 (step S22). The generated guidance information 48 is supplied to the facial expression synthesis unit 20. The facial expression synthesis unit 20 accesses the model synthesis DB of the storage 110 based on the supplied guidance information 48, reads the character image and background selected by the user, and the corresponding model synthesis information, and the facial expression synthesis unit 20 (Step S40).

  The facial expression composition unit 20 generates a character image as a composite image based on the supplied information (step S34). The server 110 transmits the composite image and the guidance information 48 to the communication terminal devices 10 and 88 of the communication partner (step S46). Although the transmission to the communication terminal device 10 is performed for image confirmation by the user, it may not be transmitted to itself. After this transmission, the server 110 finishes the management process for generating an image and proceeds to return. Each communication terminal apparatus receives and displays the composite image and guidance information 48 supplied as described above.

  In the present embodiment, the guidance information 48 is intentionally transmitted, but it is clear from the above description that it is not necessary to transmit it.

  When operated in this way, a character image reflecting the guidance information is generated, and a heavy load process such as facial feature extraction or character image generation is executed by the server 236, and the communication terminal can be used without a notification display unit. The devices 10 and 88 can be replaced by general-purpose videophone devices and the like, and an image communication system with a guidance function can be realized more easily.

  By configuring as described above, the communication terminal apparatus 10 outputs the subject image captured by the image input unit 12 to the facial expression feature extraction unit 16 as image data 34a, and provides guidance information 48 based on the feature data 36. A character composite image 44 generated by the guide information generation unit 22 and reflected by the advice information by the facial expression synthesis unit 16 is generated and output via the transmission / reception unit 30. By confirming, the burden on the user can be reduced.

  The guide information generation unit 22 includes an extraction data determination unit 72, an image light / dark determination unit 74, a face distance determination unit 76, an open / close determination unit 78, and a face angle determination unit. By integrating the extracted data determination unit 72 and at least one of these determination units and generating the guidance information 48, the user can know the shooting situation in detail, and the usability can be greatly improved.

  The communication terminal device 10 outputs the subject image captured by the image input unit 12 to the facial expression feature extraction unit 16 as image data 34a, and generates guidance information 48 based on the feature data 36 by the guide information generation unit 22. The notification control unit 28 converts the guidance information 48 into notification information 56, and the projection output unit 18 receives and outputs the notification information 56, so that the user's own shooting situation can be easily grasped and the burden on the user is reduced. can do.

  The guide information generation unit 22 includes an extraction data determination unit 72, an image light / dark determination unit 74, a face distance determination unit 76, an open / close determination unit 78, and a face angle determination unit. By integrating the extracted data determination unit 72 and at least one of these determination units and generating the guidance information 48, the user can know the shooting situation in detail, and the usability can be greatly improved.

  Further, the communication terminal device 10 manages the guidance information 48 and the notification information 56 in association with each other in the storage 68, accesses the storage 68 with the notification control unit 28, and acquires the notification information 56 corresponding to the guidance information 48. Thus, it is converted and output to the sound output unit 18 so that the user can know the shooting situation accurately.

  The notification control unit 28 outputs the guidance information 48 to the transmission / reception unit 30, supplies the guidance information 48 to the communication terminal device 88 of the communication partner, and transmits to the communication terminal device 88 at least the transmission / reception unit 92, the notification control unit 94, and the video. By including the sound output unit 98, it is possible to confirm the presence of the communication partner by displaying (outputting) the notification information 104, and it is possible to concentrate on communication excluding anxiety.

  The server 110 includes a transmission / reception unit 30, a display feature extraction unit 16, a facial expression synthesis unit 20, and a guidance information generation unit 22, and the character synthesis image 44 is generated by the facial expression synthesis unit 20 based on character data, feature data, and guidance information. At least, by outputting to the information terminal device of the communication partner, the processing load on the terminal side can be reduced and the notification display can be eliminated. As a result, it can be easily used with a general-purpose video phone or the like.

  The information terminal device may be a mobile phone, a PDA (Personal Digital Assistant), a video phone, a camera interphone, or the like as a personal computer or a portable information terminal.

It is a block diagram which shows the schematic structure of the Example in the communication terminal device to which the information terminal device which concerns on this invention is applied. It is a figure which shows the structure of the guidance information generation part in the communication terminal device of FIG. It is a figure which shows a response | compatibility with the guidance information and notification information which the communication terminal device of FIG. 1 handles. It is a figure which shows the example of a notification display displayed on the projection output part of the communication terminal device of FIG. It is a flowchart explaining the operation | movement procedure in the communication terminal device of FIG. It is a block diagram which shows the schematic principal part structure of the other Example in the communication terminal device to which the information terminal device which concerns on this invention is applied. It is a block diagram which shows the other structural example in the guidance information generation part of FIG. It is a figure which shows a response | compatibility with guidance information at the time of using the guidance information generation part of FIG. It is a flowchart explaining the operation | movement procedure of the communication terminal device using the guidance information generation part of FIG. It is a block diagram which shows the principal part structure of the other Example in each communication terminal device of the image communication system to which the information terminal device which concerns on this invention is applied. 11 is a flowchart for explaining an operation procedure in the transmission-side communication terminal device of FIG. 10 to which the information terminal device according to the present invention is applied. 12 is a flowchart for explaining an operation procedure in the receiving communication terminal apparatus with respect to the transmitting communication terminal apparatus in FIG. It is a block diagram which shows the principal part structure of the other Example in the communication terminal device of the image communication system to which the information terminal device which concerns on this invention is applied. 14 is a flowchart for explaining an operation procedure in the communication terminal apparatus of FIG. It is a block diagram which each shows the structure of the communication terminal device and server in the image communication system to which the information terminal device which concerns on this invention is applied. 16 is a flowchart for explaining an operation procedure in the image communication system of FIG. 16 is a flowchart for explaining an operation procedure of a server in the image communication system of FIG.

Explanation of symbols

10 Communication terminal equipment
12 Image input section
14 selector switch
16 Facial feature extraction unit
18 Projection output section
20 facial expression synthesis
22 Guidance information generator
24 Operation unit
26 Character Management Department
28 Notification controller
30 Transceiver

Claims (6)

In an image transmission system that communicates using an information terminal device that handles images as communication information, the device includes:
Image input means for capturing a subject as a video signal and outputting it as image data;
Feature extraction means for extracting feature data representing the facial expression of the image data;
Facial expression synthesis means for transforming character data as a basic image selected by the user based on the feature data and generating a character composite image;
A transmission / reception means for transmitting / receiving data to / from a communication partner information terminal device;
Output means for displaying or outputting image data from the information terminal device of the communication partner;
Guide information generating means for generating advice information indicating what kind of situation the photographing situation of the user is based on the feature data;
The information terminal device characterized in that the facial expression synthesis means generates a character composite image reflecting the advice information and outputs the character composite image generated from the transmission / reception means. 2. The apparatus according to claim 1, wherein the guide information generation unit is a functional block that determines a success rate of extraction of the feature data in providing the advice, and a first that detects light and darkness of the image from the image data. A functional block, a second functional block for detecting the distance between the face and the imaging means from the feature data, a third functional block for detecting blinking of eyes and opening / closing of the mouth, and a fourth functional block for detecting line of sight, and An information terminal apparatus comprising: an integrated determination unit that integrates at least one of the determination and the functional block to generate the advice information. In an image transmission system that communicates using an information terminal device that handles images as communication information, the device includes:
Image input means for capturing a subject as a video signal and outputting it as image data;
Feature extraction means for extracting feature data representing facial expressions from the image data;
A transmission / reception means for transmitting / receiving data to / from a communication partner information terminal device;
Output means for displaying or outputting image data from the information terminal device of the communication partner;
Guide information generating means for generating advice information indicating what kind of shooting situation the user is based on the feature data;
Notification conversion means for converting the advice information into notification information,
The output means receives and outputs the notification information ;
The notification conversion means outputs the advice information to the transmission / reception means, supplies the advice information to a communication partner information terminal device,
The communication terminal information terminal apparatus includes at least the transmission / reception means, the notification conversion means, and the output means . The apparatus according to claim 3, wherein the guide information generation unit is a functional block that determines a success rate of extraction of the feature data when providing the advice, and a first that detects light and darkness of the image from the image data. A functional block, a second functional block for detecting the distance between the face and the imaging means from the feature data, a third functional block for detecting blinking of eyes and opening / closing of the mouth, and a fourth functional block for detecting line of sight, and An information terminal apparatus comprising: an integrated determination unit that integrates at least one of the determination and the functional block to generate the advice information. 5. The apparatus according to claim 3, further comprising a storage unit that manages the advice information and the notification information in association with each other,
The information conversion device characterized in that the notification conversion means accesses the storage means, acquires the notification information corresponding to the advice information, converts it, and outputs it to the output means. In the image transmission system to which the information terminal device according to claim 3, 4 or 5 is connected, the system includes a server device,
The server device includes guide information generating means for generating advice information indicating what kind of shooting situation of the user is based on the transmission / reception means, the feature extraction means, the facial expression synthesis means, and the feature data,
The server device generates a character composite image obtained by deforming character data as a basic image selected by the user based on the character data, the feature data, and the advice information, and includes at least the communication A server device that outputs to a partner information terminal device.

Images