JP5548964B2 - Electronic data editing apparatus, electronic data editing method and program - Google Patents

Description

  The present invention relates to an electronic data editing apparatus, an electronic data editing method, and a program having an electronic data editing (creating) function such as a portable terminal with a mail creating function.

  For example, since a mobile terminal with a mail creation function such as a mobile phone can arbitrarily edit (create) mail data (text data) such as text data, an electronic data editing apparatus having an electronic data editing (creation) function can be used. It is an example. In the following, if the mobile terminal with this mail creation function is a “mobile phone”, the mail that can be created with this mobile phone is a text-based mail or a mixed pictogram mail. The pictogram mixed mail can use various pictograms (a wide variety of graphic symbols such as a heart symbol prepared in advance), and can express emotions that are difficult to convey only by letters.

  However, in the mixed pictogram mail, an appropriate pictogram must be selected from the list and the pictogram must be inserted into the mail, and this operation is performed manually.

  Therefore, a technique is known in which when a mail is created, a user's voice is picked up by a microphone, an emotion is judged, and a pictograph specified from the emotion is automatically inserted (for example, see Patent Document 1).

JP 2006-277567 A

However, in the above prior art, it is possible to automatically insert data such as pictograms that require time for manual input into editing data (electronic data) such as e-mail. Since a certain amount of audio data is required and it takes a certain amount of time from voice data collection to emotion judgment, emotions cannot be judged quickly, and the operability when editing electronic data is degraded. is there.
In addition to the case of inputting pictograms with a mobile phone, there is a problem in operability even when a name consisting of a character string arbitrarily input by a user is added to image data taken with a digital camera. In other words, a normal digital camera does not have a keyboard that is easy to input such as a personal computer or an advanced input support function, and it is necessary to input characters using cursor keys. It was troublesome to input.

  Accordingly, an object of the present invention is to provide an electronic data editing apparatus, an electronic data editing method, and a program capable of improving editing efficiency when data that takes time for manual input is inserted into electronic data being edited. is there.

An electronic data editing apparatus according to the present invention is an electronic data editing apparatus having an e-mail transmission function, wherein facial expressions detected in image data are identified and classified into a plurality of groups. Expression type determination means for determining which of the types corresponds, selection means for selecting insertion data that can be inserted into an e-mail based on the expression type determined by the expression type determination means, and a transmission target The facial expression type is determined by the facial expression type determination means for the image data attached to the electronic mail, and the insertion data selected by the selection means is inserted into the electronic mail based on the determined facial expression type. Transmission control means for transmitting an e-mail, and attached image determination means for determining whether image data is attached to the e-mail to be transmitted. The transmission control unit transmits an e-mail as it is when the attached image determination unit determines that the image data is not attached, and when it is determined that the image data is attached, The facial expression type is determined by the facial expression type determination unit for data, and the insertion data selected by the selection unit based on the determined facial expression type is inserted into the electronic mail and the electronic mail is transmitted. Features.
An electronic data editing apparatus according to another aspect of the present invention is an electronic data editing apparatus having an e-mail transmission function, wherein a human facial expression detected in image data is identified, and a plurality of facial expressions are identified. A facial expression type determining means for determining which facial expression type is classified into a group, and a selecting means for selecting insertion data that can be inserted into an e-mail based on the facial expression type determined by the facial expression type determining means And determining the expression type by the expression type determination means for the image data attached to the e-mail to be transmitted, and inserting the insertion data selected by the selection means based on the determined expression type A transmission control means for inserting into an e-mail and sending an e-mail, and a mode setting for selectively setting an emotion mark automatic insertion mode and an emotion character automatic insertion mode. And when the automatic setting mode of the emotion mark is set by the mode setting unit, the selection unit is image data representing emotion based on the facial expression type determined by the facial expression type determination unit. When a certain emotion mark is selected and the emotion character automatic insertion mode is set by the mode setting means, an emotion character that is a character representing an emotion based on the facial expression type determined by the facial expression type determination means is selected. And when the automatic setting mode of the emotion mark is set by the mode setting unit, the transmission control unit adds the emotion mark selected by the selection unit to a part of the image data attached to the e-mail. When the electronic mail is synthesized and the emotion setting automatic insertion mode is set by the mode setting means, the selection means Characterized in that it send an e-mail the selected emotion character is inserted in the subject line or body of the e-mail.
An electronic data editing method according to the present invention is a method for controlling an electronic data editing apparatus having an e-mail transmission function, in which a computer of the electronic data editing apparatus detects a human face detected in image data. Can be inserted into an e-mail based on the facial expression type determination process that identifies facial expressions and determines which facial expression type is classified into multiple groups and the facial expression type determined by the facial expression type determination process Selection processing for selecting insertion data, and facial expression type determination by the facial expression type determination processing for image data attached to an email to be transmitted, and the selection processing based on the determined facial expression type A transmission control process that inserts the insertion data selected by e-mail into the e-mail and transmits the e-mail; and image data in the e-mail to be transmitted. An attached image determination process for determining whether or not an image is attached, and the transmission control process transmits an e-mail as it is when the attached image determination process determines that no image data is attached. If it is determined that the image data is attached, the facial expression type is determined by the facial expression type determination process for the image data, and is selected by the selection process based on the determined facial expression type. The insertion data is inserted into the electronic mail and the electronic mail is transmitted.
According to another aspect of the present invention, there is provided an electronic data editing method for controlling an electronic data editing apparatus having an e-mail transmission function, which is detected in image data by a computer of the electronic data editing apparatus. A facial expression type determination process that identifies facial expression of a human face and determines which of the facial expression types classified into a plurality of groups, and an electronic expression based on the facial expression type determined by the facial expression type determination process A selection process for selecting insertion data that can be inserted into an e-mail and a determination of an expression type by the expression type determination process for image data attached to an e-mail to be transmitted, and based on the determined expression type Then, a transmission control process for inserting the insertion data selected by the selection process into the electronic mail and transmitting the electronic mail is executed.
In addition, the program according to the present invention enables a computer of an electronic data editing apparatus having an e-mail transmission function to identify facial expressions of a human face detected in image data and to classify the facial expressions into a plurality of groups. A facial expression type determining means for determining which one of these is applicable, a selecting means for selecting insertion data that can be inserted into an e-mail based on the facial expression type determined by the facial expression type determining means, and an electronic to be transmitted The facial expression type is determined by the facial expression type determination means for the image data attached to the mail, and the insertion data selected by the selection means based on the determined facial expression type is inserted into the electronic mail to be electronic Transmission control means for transmitting mail and attached image determination means for determining whether image data is attached to the e-mail to be transmitted When the transmission control means determines that the image data is not attached by the attached image determination means, the transmission control means transmits an e-mail as it is and determines that the image data is attached. Performs the facial expression type determination by the facial expression type determination means for the image data, inserts the insertion data selected by the selection means based on the determined facial expression type into the electronic mail, and sends the electronic mail. It is characterized by transmitting.
According to another aspect of the present invention, there is provided a program for identifying a facial expression of a human face detected in image data and classifying the computer into an electronic data editing apparatus having an electronic mail transmission function. A facial expression type determining means for determining which of the facial expression types to be applied, a selecting means for selecting insertion data that can be inserted into an e-mail based on the facial expression type determined by the facial expression type determining means, and transmission The facial expression type is determined by the facial expression type determination unit for the image data attached to the target electronic mail, and the insertion data selected by the selection unit based on the determined facial expression type is added to the electronic mail. It is characterized by functioning as transmission control means for inserting and transmitting electronic mail.

  According to the present invention, it is possible to provide an electronic data editing apparatus, an electronic data editing method, and a program capable of improving editing efficiency when data that takes time for manual input is inserted into electronic data being edited.

It is a block diagram of a mobile phone with a camera. 3 is a conceptual configuration diagram of an emotion determination unit 13. FIG. It is a figure which shows the flow of a mail preparation and transmission process. It is a conceptual diagram which shows the automatic synthetic | combination mode of an emotion mark, and the automatic insertion mode of an emotion character. It is a figure which shows a 1st modification. It is a selection instruction conceptual diagram of a desired emotion determination result. It is a figure which shows the flow of a dictionary registration process. It is a conceptual diagram of a dictionary data table. It is a figure which shows the flow of dictionary data table utilization. It is a figure which shows the flow of the judgment data registration process, and the flow of the emotion specific process using the data for judgment. It is a conceptual diagram of the data table for judgment.

  Hereinafter, embodiments of the present invention will be described by taking a camera-equipped mobile phone as an example with reference to the drawings.

  First, terms in this embodiment are defined. The e-mail or e-mail data refers not only to text-based data such as text data, but also to data that includes additional information such as format information and optional information other than characters (image information, etc.) in addition to character information. . This is because, for example, text mail is generally used frequently in mobile phones, but some mobile phones and personal computers use non-text mail (electronic data such as HTML format) that can embed characters and images. ) Can be edited and created. In addition, “editing” or “editing process” in the present embodiment means that the electronic data that has already been created (e.g., the above-mentioned mail data) is modified or modified, and that new electronic data can be created. means. “Inserted data” refers to data that is inserted or synthesized into electronic data to be edited. For example, when the electronic data to be edited is “image data”, it can be inserted by image synthesis or the like. If the image data is image data or the electronic data to be edited is text data such as text data, it is text data (for example, text data) that can be additionally inserted at a specified position of the text data. However, these are merely examples for explanation, and any data can be used as long as it can be inserted into electronic data to be edited.

  FIG. 1 is a block diagram of a camera-equipped mobile phone. In this figure, a camera-equipped mobile phone (hereinafter simply referred to as “mobile phone”) 1 performs voice communication or data communication including mail via an antenna 2 with the nearest mobile phone base station (not shown). A communication unit 3 to perform, an imaging unit 4 including an imaging device such as a CCD or CMOS that captures a still image or a through image of a subject and generates image data thereof, a numeric keypad for both telephone number / character input, an on-hook key, and an off-hook Operation unit 5 including keys, shutter keys, cursor keys and various function keys, display of incoming / outgoing telephone numbers, electronic viewfinder during still image shooting or through image shooting, confirmation / playback display of captured images, creation of e-mail, Display unit 6 composed of a liquid crystal display or the like used for browsing, voice call microphone 7 and speaker 8, image data, and sent / received mail Data and other data such as a telephone directory, an input / output unit 11 that mediates exchange of image data and the like with an external device 10 such as a personal computer, and a primary battery or a secondary battery And a power supply unit 12 including a human face is extracted from given image data, and facial expressions (laughing face, crying face, angry face, expressionlessness,...) Are determined and It includes an emotion determination unit 13 that generates emotion data indicating the determination result, and a control unit 14 that is configured by a microcomputer including a CPU 14a, a PROM 14b, a RAM 14c, an I / O 14d, and the like, and controls the operations of the above-described units.

  FIG. 2 is a conceptual configuration diagram of the emotion determination unit 13. In this figure, as described above, the emotion determination unit 13 extracts “a person's face from the given image data, and expresses the facial expression (laughing face, crying face, angry face, no expression,... Etc.) and generate emotion data indicating the determination result ”. Here, “emotion” accurately refers to the human psychological state, but in practice, it is impossible or difficult to grasp the psychological state, and here, there is a certain correlation with the psychological state. “Facial expression” is defined as emotion. That is, it is assumed that “emotion = facial expression”.

  As a known technique for judging emotion (≈ psychological state) from facial expressions, for example, “Method of Estimating Psychological State from Facial Expression” described in Japanese Patent Application Laid-Open No. 2007-65969 is known. The method may also be used for the determination unit 13.

  By the way, the principle of the method described in this publication is that “the Fourier description that quantitatively represents the shape of the left and right eyes and mouth using a Fourier descriptor from a human image, and represents the shape of these mouth and left and right eyes, respectively. By calculating and normalizing the determination score for a plurality of preset basic emotions for each shape of the mouth and the left and right eyes by a linear discriminant using a child, the above basic emotions in each shape are calculated. Is calculated using a multiple regression equation that takes into account the importance of each facial expression element, and the human psychological state is estimated from the result. "

  The emotion determination unit 13 may be given image data G (this image data G may be a still image or a through image captured by the imaging unit 4 or may be a captured image stored in the storage unit 9. A face extraction unit 15 that extracts a human (human) face from a still image or a still image input from the external device 10; and a first determination unit 16 that determines whether a face is extracted. When it is determined that there is face extraction, the extracted facial expression is identified (for example, a smile, a crying face, an angry face, etc.) and the accuracy value for each facial expression (the “probability” of the expression) Of the facial expression identification unit 17 for setting the facial expression identification unit 17, the second determination unit 18 that determines the accuracy value for each facial expression in the facial expression identification unit 17, and the second determination unit 18 A selection unit that outputs a selection signal corresponding to the determination result 9, and an emotion data output unit 20 for outputting removed one emotion data from a plurality of emotion data according to the selection signal.

  Here, the facial expression identification unit 17 has a plurality of facial expression classifiers for each facial expression. Specifically, the illustrated example includes a smile classifier 17a for identifying a smile, a crying face classifier 17b for identifying a crying face, and an angry face classifier 17c for identifying an angry face. However, these are only examples. A classifier corresponding to another facial expression may be provided, or any of the classifiers illustrated may be omitted or replaced. Alternatively, it may be a single configuration that identifies only one facial expression (for example, a smile).

  The “smile”, “crying face”, and “angry face” in the above examples are examples of “expression types” for classifying different facial expressions of the same person. It is not a classification but a classification of facial expressions commonly used by a plurality of persons.

  The emotion data output unit 20 includes the number of facial expressions identified by the facial expression identification unit 17 plus one data output unit. That is, in the illustrated example, the facial expression identification unit 17 includes three classifiers (smiling classifier 17a, crying face classifier 17b, and angry face classifier 17c). A smile data output device 20a for outputting, a crying face data output device 20b for outputting crying face data, and an angry face data output device 20c for outputting angry face data. Is provided with an expressionless data output device 20d for outputting expressionless data when the above does not apply.

  The face data taken out from these four data output devices (smile data output device 20a, crying face data output device 20b, angry face data output device 20c and expressionless data output device 20d) are designed to imitate each expression. It may be a facial expression icon, or may be an emoticon or a pictograph. Alternatively, it may be identification information such as code data for specifying those facial expression icons, emoticons or pictograms.

  In such a configuration, when arbitrary image data G is given to the emotion determination unit 13, first, the face extraction unit 15 extracts a human face, and then the first determination unit 16 determines whether or not there is face extraction. Is done. If there is no face extraction, it is determined that the image data does not include a person's face, and data indicating that fact is output.

  On the other hand, when it is determined that there is face extraction, the facial expression identification unit 17 identifies the facial expression of the extracted face. For example, when the extracted face is a smile, a large value is given to the accuracy value of the smile classifier 17a, and a small value is given to the accuracy value of the other classifiers (crying face classifier 17b and angry face classifier 17c). Is given. The second determination unit 18 compares the respective accuracy values, the selection unit 19 outputs a selection signal for extracting emotion data (smile data) corresponding to the facial expression having the largest accuracy value, and the emotion data output unit 20 In accordance with this selection signal, one emotion data out of a plurality of emotion data, in this case, smile data (smile icon, smile pictogram or emoticon or identification information for specifying them) of the smile data output device 20a. Take out and output.

  As described above, the emotion determination unit 13 extracts the face of a person from the given image data G, and determines the facial expression (laughing face, crying face, angry face, no expression, etc.). Thus, emotion data indicating the determination result can be generated. In addition, when the accuracy value of each facial expression is equivalent, it is regarded as an expressionless and data of the expressionless data output unit 20d (expressionless icon, expressionless pictogram or emoticon or identification information for specifying them) Is output.

  FIG. 3 is a diagram showing a flow of mail creation and transmission processing. This flow is a part of a control program stored in advance in the PROM 14b of the control unit 14, and is read into the RAM 14c of the control unit 14 in response to a user event (an operation event for mail creation or transmission by the user). It is executed by the CPU 14a of the control unit 14.

  This flow can be broadly divided into a text editing block B1 executed immediately after the start of the flow and an emotion mark insertion block B2 executed subsequently. First, in the text editing block B1, it is determined whether or not text data for the mail subject has been input (step S1). If it has been input, the text data is inserted at the specified position (cursor position) of the mail subject. (Step S2). Next, it is determined whether or not text data for the mail body has been input (step S3), and if it has been input, the text data is inserted at the designated position (cursor position) of the mail body (step S4).

  Next, in the emotion mark insertion block B2, first, the presence / absence of image data for mail attachment is determined (step S5). Here, the image data for mail attachment is typically a face photograph (still image) taken by the image pickup unit 4 during mail creation, but other than this, for example, in the storage unit 9 It may be a stored face photograph taken in the past. In short, any suitable face photo for extracting emotions may be used. It may also be a face photo of another person.

  If there is no image data for mail attachment, the process leaves the emotion mark insertion block B2 to determine completion of mail creation (mail editing completion operation by the user) (step S13). If B1 is executed and completed, the mail is transmitted (step S14), and then the flow ends.

  On the other hand, when there is image data for mail attachment, the image data is sent to the emotion determination unit 13, and the emotion determination unit 13 detects a face (step S6). If no face is detected (step S7). NO) exits the emotion mark insertion block B2 and determines that the mail creation is complete (the mail editing completion operation by the user) (step S13). If not completed, the text editing block B1 is executed again and completed. Then, after sending the mail (step S14), the flow ends.

  On the other hand, if a face is detected (YES in step S7), emotions are determined by recognizing facial expressions (step S8), and then it is determined whether the emotion mark automatic synthesis mode is selected. (Step S9). The emotion mark automatic composition mode is a mode selected in advance by the user, and when this mode is selected, the automatic combination of the emotion mark to the image data attached to the mail is allowed. .

  When the emotion mark automatic synthesis mode is selected, an emotion mark corresponding to the determined emotion is generated and synthesized at the designated position in the attached image (step S10), and then the emotion character automatic insertion mode is selected. It is determined whether or not it is performed (step S11). The emotion character automatic insertion mode is a mode selected in advance by the user, and when this mode is selected, the automatic insertion of emotion characters in the mail subject or mail text is allowed. .

  If the emotion character automatic insertion mode is selected, a pictograph corresponding to the determined emotion is generated and combined with the specified subject position (cursor position) of the subject of the mail or the text (step S12). If the emotion mark automatic composition mode is not selected, step S10 is passed, and if the emotion character automatic insertion mode is not selected, step S12 is passed.

  FIG. 4 is a conceptual diagram showing an emotion mark automatic synthesis mode and an emotion character automatic insertion mode. As shown in (a), in the emotion mark automatic composition mode, the emotion mark 22 is placed at a predetermined position (for example, one of the corners indicated by a dotted line) of the target image data 21 (image data attached to an email). Synthesize. Also, as shown in (b), in the emotional character automatic insertion mode, emoticons (or emoticons) 24 and 25 are inserted at designated positions (cursor positions) in the subject or body of the mail 23.

  As described above, according to the present embodiment, the face character (or face pictogram) 24 corresponding to his / her emotion at that time can be obtained simply by taking a still image of his / her face using the image pickup unit 4 during mail creation. 25 can be automatically inserted in the subject and body of the mail 23.

  At this time, the still image shooting of the imaging unit 4 is completed almost simultaneously with the shutter operation, and the face extraction and emotion determination in the emotion determination unit 13 are also completed almost instantly. Compared to emotion judgment based on, a series of processes from emotion judgment to emoji or emoticon insertion can be completed in a very short time. It is possible to provide a mobile terminal with a mail creation function that can be reflected. In addition, since emotion determination is performed from image data attached to an email, it is not necessary to prepare image data for emotion determination again, thereby reducing labor.

  In addition, according to the present embodiment, as shown in FIG. 4A, since an emotion mark can be automatically synthesized at a predetermined position of image data attached to an email, the emotion of the sender is given to the recipient of the email. Can be visualized, and the certainty of emotion transmission can be improved.

  In the above description, emoticons (characters are combined to make a face) 24 and 25 are inserted in the subject and body of the mail 23, but the present invention is not limited to this. For example, graphic data such as a face icon or a pictograph may be inserted if permitted by the specification of the mail system.

  The technology of the present application is not limited to the above-described embodiment, and various modifications and development examples are included within the scope of the idea. For example, the following may be adopted.

  FIG. 5 is a diagram showing a first modified example, and more specifically, is a diagram showing a main part flow replacing step S5 to step S8 of the emotion mark insertion block B2 of FIG. In this first modification, first, it is determined whether image data for mail attachment (hereinafter referred to as image data for attachment) has been photographed (step S15), and when image data for attachment has been photographed, After the image data for attachment is stored in the storage unit 9 (step S16), it is determined whether new image data (hereinafter referred to as emotion determination image data) has been captured (step S17).

  If the image data for attachment has not been photographed, step S16 is passed to determine whether or not the image data for emotion determination has been photographed (step S17). If the emotion determination image data has not been shot, the process passes through the emotion mark insertion block B2 and proceeds to step S13 (email creation completion determination). If the emotion determination image data has been shot, the emotion determination image data is acquired. The image data is sent to the emotion determination unit 13, and the emotion determination unit 13 detects a face from the image data for emotion determination (step S18). If no face is detected (NO in step S19), an emotion mark is inserted. The process leaves block B2 and proceeds to step S13 (mail creation completion determination).

  When a face is detected (YES in step S19), the facial expression is recognized to determine an emotion (step S20), and then an emotion mark indicating the determined emotion is displayed on the screen (step S21). . Then, the user's selection instruction is determined (step S22). If there is no selection instruction, the repetition is determined (step S23), and if there is a repetition, step S17 (photographing determination of image data for emotion determination) is performed again. If there is no repetition, the emotion mark insertion block B2 is skipped and the process proceeds to step S13 (email creation completion determination). If there is a selection instruction, the emotion determined at the timing when the selection instruction is given. Adopt (step S24).

  Thereafter, similarly to the emotion mark insertion block B2 of FIG. 3, it is determined whether or not the emotion mark automatic synthesis mode is selected (step S9). If it is selected, an emotion mark corresponding to the determined emotion is generated and combined with the designated position in the attached image data (step S10), and then the emotion character automatic insertion mode is selected. It is determined whether or not (step S11). If it is selected, a pictograph corresponding to the determined emotion is generated and combined with the designated subject position (cursor position) of the subject of the mail or the text (step S12).

  In this first modification, image data attached to an email (image data for attachment) and image data for emotion determination (image data for emotion determination) are separated, so image data attached to the email ( Arbitrary data can be used for the attached image data, and the degree of freedom of the attached image data is increased.

  In addition, when the desired emotional determination is obtained while repetitively capturing the image data for emotion determination or while capturing a through image, it is only necessary to indicate the selection result of the emotion determination result. Emotion determination can be performed through trial and error, and emoticons and emotion marks that match the desired emotion expression can be selected and combined with image data for attachment or inserted into an email.

  FIG. 6 is a conceptual diagram for selecting a desired emotion determination result. In this figure, a confirmation display area 26 for emotion determination image data is provided on the screen of the display unit 6, and each of one or several emotion determination image data taken retrospectively. An icon indicating the emotion determination result is displayed. For example, in the illustrated example, an expressionless icon 27, a crying face icon 28, and a smiling face icon 29 are displayed in order from the top. The user only has to select the icon of the intended emotion. For example, in this figure, the smile icon 29 where the cursor 30 is located can be selected and designated as the intended icon.

  In this example, it is assumed that the image data for emotion determination is acquired by shooting several times, that is, the image data for emotion determination is a plurality of still image data obtained by shooting the same subject several times. For example, a through image may be used. By changing the facial expression during the acquisition of the through image, the expressionless icon 27, the crying face icon 28, the smile icon 29, etc. can be displayed, and any of them can be selected arbitrarily.

  In the above description, emotion marks (for example, emoticons and pictograms) determined from facial expressions are inserted. However, the present invention is not limited to this. For example, text data corresponding to facial expressions is registered in a dictionary in advance. Alternatively, text data may be read from the dictionary and inserted into the mail whenever necessary.

  FIG. 7 is a diagram showing a flow of dictionary registration processing. In this flow, first, text data (words, short sentences, etc.) to be registered is input (step S31), and then the face is continuously photographed (through image photographing) using the imaging unit 4, and the through image data is obtained. Obtain (step S32). Then, the emotion determination unit 13 is used to determine the emotion from the face in the through image data (step S33), and the determination result is displayed on the display unit 6 to determine the presence / absence of the user's registration instruction (step S34). ). If there is no registration instruction (NO in step S35), the process returns to step S32 again. If there is a registration instruction (YES in step S35), the emotion information (code) determined at the timing of the registration instruction and step S31 After associating with the text data input in step (b) and registering it in the dictionary data table in the PROM 14b or the storage unit 9 (step S36), the flow ends. It should be noted that the “registration instruction” by the user is not only performed by the key operation of the operation unit 5, but other than that, for example, when the facial expression is not changed for a predetermined time toward the imaging unit 4 “Registration instruction”, or “cancellation instruction” when the facial expression is changed during the process. This is preferable because a confirmation or cancellation operation by the user is unnecessary.

  FIG. 8 is a conceptual diagram of the dictionary data table. In this figure, the dictionary data table 31 has an emotion field 31a and a text data field 31b. The emotion field 31a stores information on emotions (laugh, cry, anger, expressionless, etc.), and the text data field 31b stores text data for each emotion. In the illustrated example, the emotion field 31a stores “laugh” indicating a laughing face, “crying” indicating a crying face, “angry” indicating an angry face, and “nothing” indicating no expression. This is for convenience of illustration. Actually, identification information (code) corresponding to each emotion is stored.

  FIG. 9 is a diagram showing a flow of using the dictionary data table, which replaces the mail subject input processing (steps S1 and S2) and the mail text input processing (steps S3 and S4) in the flow of FIG.

  In this flow, first, it is determined whether or not text data has been input (step S41). If it has been input, the text data is inserted at the current cursor position (step S42). Next, it is determined whether to acquire through image data (step S43). If not acquired, the process proceeds to step S5 in FIG. 3. If acquired, emotion information (code) is obtained from the face in the through image data. Determination is made (step S44).

  Then, with reference to the dictionary data table 31 of FIG. 8, the text data registered in association with the emotion information (code) determined in step S44 is read, and the text data is “temporarily” inserted at the current cursor position. (Step S45).

  Next, a confirmation operation or cancellation operation by the user is determined (step S46), and if cancelled, the process proceeds to step S5 in FIG. 3, and if confirmed, the temporarily inserted text data is confirmed and inserted (step S47). Then, the process proceeds to step S5 in FIG. It should be noted that the “confirm” or “cancel” operation by the user is not only performed by the key operation of the operation unit 5, but also, for example, the facial expression is changed toward the imaging unit 4 for a predetermined time. It may be “confirmed” if the user has not been selected, or “cancel” if the expression is changed in the middle.

  According to this flow, emotion information (code) and text data are associated with each other and registered in the dictionary data table 31. After that, it is determined from the image data only by photographing a face as necessary. The text data related to the emotion information (code) can be extracted from the dictionary data table 31 and automatically inserted at the designated position of the mail subject or the mail text. For this reason, text data corresponding to the emotion at that time can be quickly inserted into the mail, and the efficiency of mail creation can be improved.

  In the above explanation, emotions are specified from the face included in the image data. However, in reality, there are individual differences in facial expressions, so emotions must be specified for application to unspecified humans. Some kind of ingenuity is necessary in terms of reliability. The following are the ideas.

  FIG. 10 is a diagram showing a flow of determination data registration processing and a flow of emotion identification processing using the determination data. As shown in (a), in the judgment data registration process, first, a personal name to be registered is input (step S51), and information representing an emotion to be registered is input (step S52). Next, imaging is performed to acquire image data (step S53), and the captured image data or feature data extracted from the image data (feature point data in the image) is input to the personal name in step S51. Are registered in a data table for determination (described later) in association with the emotion (information representing) input in step S52 (step S54), and then the flow ends.

  Next, as shown in (b), in the emotion identification process, the face detected in the captured image is collated with registration information in a determination data table (described later) to identify registration data having the highest degree of matching. (Step S61). Next, after identifying the personal name and emotion corresponding to the identified registration data (step S62), the flow is terminated.

  FIG. 11 is a conceptual diagram of the determination data table. The determination data table 32 includes a user field 32a, an emotion field 32b, and an image data or feature data field 32c. The user field 32a stores the personal name input in step S51, the emotion field 32b stores information representing the emotion input in step S52, and the image data or feature data field 32c stores image data. Alternatively, feature data extracted from the image data is stored.

  According to the flow of the determination data registration process and the emotion identification process using the determination data, the determination data table 32 stores emotions (information representing each person) in advance and image data or feature data of the individual. When the emotion is actually determined, the emotion of each individual is specified by collating with the registered information in the determination data table 32, so that it is not affected by individual differences and is not specified by an unspecified human. It is possible to improve the reliability of emotion identification when applied to.

  In the above description, e-mail is created with a camera-equipped mobile phone as an example. However, the present invention is not limited to this. For example, a user arbitrarily inputs image data taken with a digital camera. You may apply when adding the name which consists of character strings. In addition, it is desirable that a camera function such as an imaging unit is present, but image data can be taken in from an external device such as a personal computer as necessary. In short, any electronic device having a function of acquiring image data from its own imaging unit or an external device and a function of editing electronic data may be used.

1 Mobile phone with camera (electronic data editing device)
3 Communication unit (transmission means)
4 Imaging unit (imaging means)
14 Control unit (identifying means, inserting means, reading means, image attaching means, specifying means)
15 Face extraction unit (extraction means)
17 Facial expression identification unit (determination means)
19 Selection part (selection means)
31 Dictionary data table (storage means)
32 Judgment data table (registration means)

Claims (9)

An electronic data editing device having an e-mail transmission function,
Facial expression type determining means for identifying facial expressions detected in the image data and determining which of the facial expression types classified into a plurality of groups;
Selection means for selecting insertion data that can be inserted into an e-mail based on the expression type determined by the expression type determination means;
A facial expression type is determined by the facial expression type determination unit for image data attached to an electronic mail to be transmitted, and insertion data selected by the selection unit based on the determined facial expression type is the electronic mail. A transmission control means for sending an e-mail inserted into
An attached image determining means for determining whether image data is attached to the e-mail to be transmitted;
The transmission control means transmits an e-mail as it is when the attached image determination means determines that the image data is not attached, and when it is determined that the image data is attached, the image data A facial expression type is determined by the facial expression type determination means, and insertion data selected by the selection means based on the determined facial expression type is inserted into the electronic mail, and an electronic mail is transmitted. An electronic data editing device. An electronic data editing device having an e-mail transmission function,
Facial expression type determining means for identifying facial expressions detected in the image data and determining which of the facial expression types classified into a plurality of groups;
Selection means for selecting insertion data that can be inserted into an e-mail based on the expression type determined by the expression type determination means;
A facial expression type is determined by the facial expression type determination unit for image data attached to an electronic mail to be transmitted, and insertion data selected by the selection unit based on the determined facial expression type is the electronic mail. A transmission control means for sending an e-mail inserted into
Mode setting means for selectively setting the emotion mark automatic insertion mode and the emotion character automatic insertion mode,
The selection means selects an emotion mark that is image data representing an emotion based on the facial expression type determined by the facial expression type determination means when the emotion setting automatic insertion mode is set by the mode setting means When the emotion character automatic insertion mode is set by the mode setting means, an emotion character that is a character representing an emotion is selected based on the expression type determined by the expression type determination means,
The transmission control unit combines the emotion mark selected by the selection unit with a part of the image data attached to the e-mail when the mode setting unit sets the emotion mark automatic insertion mode. Send an e-mail, and if the mode setting means is set to the emotion character automatic insertion mode, send the e-mail by inserting the emotion character selected by the selection means into the subject or body of the e-mail An electronic data editing apparatus characterized by: Face determination means for determining whether or not a face is included in the image data;
When the transmission control means determines that the image data is attached by the attached image determination means, the transmission control means further determines whether or not a face is included in the image data by the face determination means, When it is determined that the face is not included, the e-mail is transmitted as it is. When it is determined that the face is included, the expression type determination unit determines the expression type for the image data. The electronic data editing apparatus according to claim 1 , wherein the electronic data editing apparatus transmits the electronic mail by inserting the insertion data selected by the selection means into the electronic mail based on the determined facial expression type. The selection means selects an emotion character that is a character representing an emotion based on the expression type determined by the expression type determination means,
4. The electronic data editing according to claim 1, wherein the transmission control unit transmits the e-mail by inserting the emotion character selected by the selection unit into a subject or body of the e-mail. apparatus. The selection means selects an emotion mark that is image data representing an emotion based on the facial expression type determined by the facial expression type determination means,
5. The transmission control unit according to claim 1, wherein the transmission control unit combines the emotion mark selected by the selection unit with a part of image data to be attached to the email and transmits the email. Electronic data editing device. A text data storage unit that stores in advance any text data associated with each of the plurality of facial expression types that can be determined by the facial expression type determination unit;
The selection means selects text data corresponding to the expression type determined by the expression type determination means from among a plurality of text data stored in the text data storage means,
6. The electronic data editing according to claim 1, wherein the transmission control means inserts the text data selected by the selection means into a subject or body of the electronic mail and transmits the electronic mail. apparatus. Imaging means;
An input means for inputting arbitrary text data in accordance with a user operation;
The acquisition of new image data by the imaging unit, the input of arbitrary text data by the input unit, the facial expression type determined by the facial expression type determination unit for the acquired new image data, The electronic data editing apparatus according to claim 6, further comprising: a registration unit that associates input arbitrary text data with each other and additionally registers the text data in the text data storage unit. A method of controlling an electronic data editing apparatus having an e-mail transmission function,
In the computer of the electronic data editing device,
A facial expression type determination process that identifies facial expressions detected in the image data and determines which of the facial expression types classified into a plurality of groups;
A selection process for selecting insertion data that can be inserted into an e-mail based on the expression type determined by the expression type determination process;
The facial expression type is determined by the facial expression type determination process for the image data attached to the electronic mail to be transmitted, and the insertion data selected by the selection process based on the determined facial expression type is the electronic mail. A transmission control process that inserts into and sends an email,
An attached image determination process for determining whether or not image data is attached to the e-mail to be transmitted;
The transmission control process transmits an e-mail as it is when it is determined by the attached image determination process that no image data is attached, and if it is determined that the image data is attached, this image data A facial expression type is determined by the facial expression type determination process, and insertion data selected by the selection process based on the determined facial expression type is inserted into the electronic mail, and an electronic mail is transmitted. Electronic data editing method. A computer of an electronic data editing device having an e-mail transmission function,
Facial expression type determining means for identifying facial expressions detected in the image data and determining which of the facial expression types classified into a plurality of groups;
Selection means for selecting insertion data that can be inserted into an e-mail based on the expression type determined by the expression type determination means;
A facial expression type is determined by the facial expression type determination unit for image data attached to an electronic mail to be transmitted, and insertion data selected by the selection unit based on the determined facial expression type is the electronic mail. A transmission control means for sending an e-mail inserted into
Function as attached image determination means for determining whether image data is attached to the e-mail to be transmitted;
The transmission control means transmits an e-mail as it is when the attached image determination means determines that the image data is not attached, and when it is determined that the image data is attached, the image data A facial expression type is determined by the facial expression type determination means, and insertion data selected by the selection means based on the determined facial expression type is inserted into the electronic mail, and an electronic mail is transmitted. Program.

Images