JP5518919B2 - Face registration device, program, and face registration method - Google Patents

Description

  Embodiments described herein relate generally to a face registration device, a program, and a face registration method.

  2. Description of the Related Art Conventionally, there has been disclosed a reception guidance system that can apply a face recognition technique to a humanoid robot to identify a visitor and respond individually.

  In general face recognition technology, the registration of a face requires input of a face image and a name / ID associated with the face image. Therefore, it is particularly unspecified in various facilities such as banks, hospitals, stores, etc. In an environment with many people, offline registration is essential.

  Therefore, in the prior art, when a robot having an arm portion shakes hands with a visitor, a face image is taken and a face is registered.

JP 2005-193351 A

  However, in the prior art, a face image is captured using a special environment in which a robot shakes hands with a visitor, and this is not a smooth face image acquisition method.

The face registration device according to the embodiment includes an image pickup unit that picks up an image of a situation around the device, a face detection unit that detects a human face area from an image picked up by the image pickup unit, and a predetermined call content at a predetermined timing. Voice output means for outputting the calling voice. Further, reaction detection means for detecting a face image that has reacted to the content of the call based on a change in the face area of the person by the face detection means before and after the output of the call voice by the voice output means and the reaction detection When there are a plurality of face images that have reacted to the call content detected by the means, a selection is made to select one face image from among the plurality of face images that have reacted to the call content detected by the reaction detection means Selection means for executing processing, and face registration means for registering the face image detected by the reaction detection means or the face image selected by the selection means in association with the contents of the call.

The program according to the embodiment includes a computer that controls a face registration apparatus that includes an imaging unit that captures surrounding conditions, a face detection unit that detects a human face area from an image captured by the imaging unit, and a predetermined timing at a predetermined timing. And function as voice output means for outputting a call voice related to the call contents. And a reaction detection means for detecting a face image that reacts to the content of the call based on a change in the face area of the person by the face detection means before and after outputting the call voice by the voice output means. When there are a plurality of face images that react to the call content detected by the reaction detection means, one face image among the plurality of face images that react to the call content detected by the reaction detection means Selection means for executing selection processing, face registration means for registering the face image detected by the reaction detection means or the face image selected by the selection means, and the contents of the call in association with each other Make it work.

The face registration method according to the embodiment is a face registration method executed by a face registration apparatus that includes an imaging unit that images a surrounding situation, and the face detection unit detects a human face area from an image captured by the imaging unit. A step of detecting, and a step of outputting a call voice related to a predetermined call content at a predetermined timing by the voice output means. A step of detecting a face image that reacts to the content of the call based on a change in the face area of the person by the face detection unit before and after the output of the call voice by the voice output unit; When the selection means has a plurality of face images that react to the call content detected by the reaction detection means, among the plurality of face images that react to the call content detected by the reaction detection means, A step of executing a selection process for selecting one face image, and the face registration unit associates the face image detected by the reaction detection unit or the face image selected by the selection unit with the call content. Registering.

It is a perspective view which shows the external appearance of the face registration apparatus concerning embodiment. It is a block diagram which shows the structure of the electrical equipment system of information processing apparatus. It is a block diagram functionally showing the configuration of the assist device. It is a functional block diagram which shows the function structure of a person detection part. It is a schematic diagram which shows the example of data of calling data DB. It is a functional block diagram which shows the function structure in a face registration process. It is a flowchart which shows the flow of a face registration process. It is a functional block diagram which shows the function structure of a face detection means.

  FIG. 1 is a perspective view illustrating an appearance of a face registration device 1 according to the embodiment. The face registration device 1 is used in various facilities such as banks, hospitals, stores, and the like, and is an information terminal that performs face registration of a customer while facing the customer, for example. Note that the face registration device 1 is not necessarily in a state of facing the customer, as long as it can be detected by the person detection unit 29 (see FIG. 3) and the face detection unit 81 (see FIG. 6) described later. It ’s enough. Such a face registration device 1 includes an information processing device 2 that is a main body of customer face registration and a humanoid or animal-type anthropomorphic assist device 3 called a mascot robot or the like.

  First, the information processing apparatus 2 will be described. The information processing device 2 also functions as a point service device, for example. As shown in FIG. 1, the information processing apparatus 2 has an assist device 3 placed on the upper surface of a housing 4.

  In addition, the information processing apparatus 2 includes a display device 5 configured by an LCD (Liquid Crystal Display) or an organic EL display capable of displaying predetermined information as a color image on the housing 4, and the display device 5. For example, a resistive touch panel 6 that is placed on the display surface, a card reader 7 that exchanges data with a membership card or mobile phone that is a non-contact type wireless IC card, such as various tickets, etc. Issuance port 8 and the like for issuing. The card reader 7 establishes wireless communication with a non-contact IC card or a mobile phone, and reads information from the non-contact IC card or the mobile phone. As an example, a non-contact IC card or a mobile phone can store and save electronic money or a membership number having a value equivalent to cash. In FIG. 1, an antenna (not shown) is built in the card reader 7, and wireless communication with a non-contact IC card or a mobile phone is established through this antenna. The card reader 7 may be a code reader that reads a barcode or a two-dimensional code written on a card.

  Such an electrical system of the information processing apparatus 2 is configured as shown in FIG. Here, FIG. 2 is a block diagram showing the configuration of the electrical system of the information processing apparatus 2.

  As shown in FIG. 2, the information processing apparatus 2 includes an information processing control unit having a computer configuration including a CPU (Central Processing Unit), a ROM (Read Only Memory) storing a control program, a RAM (Random Access Memory), and the like. 11 and a memory unit 12 composed of a nonvolatile ROM, HDD (Hard Disk Drive), etc., and online with each other via the communication unit 14 connected via the bus 13. It is comprised so that communication can be performed.

  The information processing control unit 11 is connected to the display device 5, the touch panel 6, and the card reader 7 described above via the bus 13 and the I / O device control unit 15, and further connected to the printer 9. The printer 9 is built in the housing 4, and prints various tickets and the like and issues them from the issuing port 8 under the control of the information processing control unit 11. The display device 5 displays an image and a message as visible guidance information for the user under the control of the information processing control unit 11.

  The memory unit 12 also functions as a DB that stores a face image, face feature information, and various types of information obtained by face registration processing described later in association with each other.

  Next, the assist device 3 will be described. Here, FIG. 3 is a block diagram functionally showing the configuration of the assist device 3. As shown in FIGS. 1 and 3, the assist device 3 mainly includes a head portion 20 and a body portion 21.

  Further, as shown in FIGS. 1 and 3, the assist device 3 includes a camera unit 23 in the head 20, a microphone 24, a speaker 25, a communication unit 26, and an operation unit 27 in the body unit 21. ing. Further, the assist device 3 centralizes the image processing unit 28, the person detection unit 29, the voice recognition unit 30, the operation control unit 31, the storage unit 32, and these hardware in the body unit 21. And an overall control unit 33 for controlling the above. The overall control unit 33 has a computer configuration including a CPU, a ROM that stores a control program, a RAM, and the like.

  The assist device 3 may be sold or transferred in a state where the program is stored in the ROM, or the program stored or transferred in the state stored in the storage medium or by communication via the communication line is arbitrary. May be installed in the assist device 3. As the above storage medium, any type of media such as a magnetic disk, a magneto-optical disk, an optical disk, or a semiconductor memory can be used.

  The camera unit 23 functions as an imaging unit having an imaging element such as a CCD sensor, and images the situation around the assist device 3. The image processing unit 28 performs image processing on the video captured by the camera unit 23 and converts it into a digital image.

  The person detection unit 29 is responsible for identifying visitors in various facilities such as banks, hospitals, stores, and the like, and detects a person standing in front of the face registration device 1 from the image processed by the image processing unit 28. As a person detection method, for example, a technique disclosed in Japanese Patent Laid-Open No. 2005-165447 can be used. Schematically, as shown in FIG. 4, the person detection unit 29 includes a face area detection unit 51, a face feature extraction unit 52, a collation calculation unit 53, and a result output unit 54. The person detection unit 29 may be realized by the CPU of the overall control unit 33 being executed according to a program.

  The face area detection unit 51 detects a human face area from the image input by the image processing unit 28. The face feature extraction unit 52 extracts face feature information in the face area detected by the face area detection unit 51.

  The collation calculation unit 53 obtains the similarity between both pieces of information by collating the face feature information extracted by the face feature extraction unit 52 with the personal face feature information of each individual stored in the memory unit 12. Various kinds of information held in the memory unit 12 are output in correspondence with the degree of similarity equal to or higher than a predetermined value and the personal face feature information obtained with the degree of similarity higher than the predetermined value. The result output unit 54 outputs the calculation result of the collation calculation unit 53.

  As shown in FIGS. 1 and 3, the operation mechanism 40 is formed in a part of the body portion 21 of the assist device 3, and the operation control unit 31 controls driving of the operation mechanism 40. Here, the operation mechanism 40 is a wing-shaped structure, for example, and can perform an operation of flapping in the vertical direction under the control of the operation control unit 31.

  In addition, an operation mechanism 50 is formed in a part of the body portion 21 of the assist device 3, and the operation control unit 31 also controls driving of the operation mechanism 50. Here, the movement mechanism 50 is a mechanism that moves the head 20 in the horizontal direction and the vertical direction, for example. The movement mechanism 50 controls the movement of the head 20, the vertical movement of the head 20, It is possible to execute a combined vertical and horizontal operation of the unit 20.

  The speaker 25 outputs a voice message or notification sound to the user. The communication unit 26 is provided to exchange information with the information processing apparatus 2. The microphone 24 collects sounds around the assist device 3 and voices of people. The operation unit 27 is used by a user to input information by key operation based on information output from the speaker 25 or the like.

  The voice recognition unit 30 receives a voice signal input through the microphone 24 and generates a voice recognition result of characters and word strings corresponding to the collected voice of the person. The voice recognition unit 30 recognizes the utterance content of the user by comparing the voice signal input through the microphone 24 with a language dictionary. The voice recognition unit 30 includes a dictionary memory that stores dictionaries for each language (Japanese dictionary, English dictionary, and Chinese dictionary) corresponding to three languages, Japanese, English, and Chinese.

  The storage unit 32 stores a call data database (hereinafter referred to as call data DB) 60 for storing call contents. FIG. 5 is a schematic diagram illustrating a data example of the challenge data DB 60. As shown in FIG. 5, the call data DB 60 is used when executing a face registration process to be described later, and for each call data assigned with a data number, Japanese, English, and Chinese. Three languages are set. For example, call data translated into English and Chinese is also set for call data “I will explain XX. Please gather if you are interested” in Japanese. Note that the call data may be subjected to speech synthesis processing for converting text information into a speech signal, or may be a reproduction of a speech signal prepared in advance. Since the speech synthesis technology has already been established and its software is commercially available, the description thereof will be omitted.

  Next, functions executed by the face registration device 1 will be described. As described above, the face registration device 1 is an information terminal that performs customer face registration while facing the customer. Schematically, the face registration apparatus 1 assumes the face registration of the person concerned by calling at the reception of various facilities such as banks, hospitals, and stores.

  Here, FIG. 6 is a functional block diagram showing a functional configuration in the face registration process, and FIG. 7 is a flowchart showing a flow of the face registration process.

  The program executed by the CPU of the overall control unit 33 of the assist device 3 includes each unit (face detection means 81, voice output means 82, reaction detection means 83, face registration means 84, selection means 85) as shown in FIG. As the actual hardware, the CPU reads the program from the ROM and executes it, so that the above-described units are loaded on the RAM, and the face detection means 81, the sound output means 82, the reaction detection means 83, Face registration means 84 and selection means 85 are generated on the RAM.

  As shown in the flowchart of FIG. 7, the assist device 3 of the face registration device 1 is acquired via the camera unit 23 and the image processing unit 28 provided in the head 20 at regular intervals when the power is turned on. From the captured image, the face detection means 81 detects a person's face (step S1).

  The face detection unit 81 detects the face of a person standing in front of the face registration device 1 from the image processed by the image processing unit 28. As a human face detection method, for example, a technique disclosed in Japanese Patent Laid-Open No. 2005-165447 can be used. Schematically, as shown in FIG. 8, the face detection unit 81 has a face area detection unit 811, a face feature extraction unit 812, and an individual face feature creation unit 813. Further, as a method for detecting a face area, a person area may be detected in advance, and the face may be obtained from the head of the person area even if the face cannot be captured. As a person area detection method, for example, the technique disclosed in Japanese Patent Laid-Open No. 2009-75868 can be used.

  The face area detection unit 811 detects a human face area from the image input by the image processing unit 28. The face feature extraction unit 812 extracts face feature information in the face area detected by the face area detection unit 811. The personal face feature creation unit 813 creates facial feature information for each individual.

  Next, the voice output unit 82 acquires the calling content of the predetermined data number from the calling data DB 60 at a predetermined timing, and outputs the calling voice through the speaker 25 (Yes in step S2). As a result, the assist device 3 of the face registration device 1 calls the user for a call content such as “explain Mr. XX” and “please explain if you are interested”. Note that the output order and output timing of the call contents may be determined in advance or may be determined randomly. Further, the contents of the call such as “Mr. XXX” may be output by the user appropriately inputting from the operation unit 27 by key operation.

  After outputting the call voice in step S2, the face detection means 81 detects the face of the person again from the captured image (step S3), and the reaction detection means 83 responds to the call contents when the call voice is output. A face image is detected (step S4).

  For example, the reaction detection unit 83 takes a difference between the facial feature information of the person detected in step S1 and the facial feature information of the person detected in step S3, and calls for a person who has changed more than a predetermined threshold. And a person who has newly detected a face is detected as a person who has walked in response to the call.

  Note that the reaction detection method in the reaction detection unit 83 may not be facial feature information, but may be image change by optical flow, extraction of a change region by image difference information, or the like.

  When the reaction detection unit 83 cannot detect the face image of the face that has reacted to the contents of the call (No in step S5), the reaction detection unit 83 returns to step S1 and the camera unit 23 provided in the head 20 at a constant interval. And the face detection means 81 detects a person's face from the captured image acquired via the image process part 28 (step S1).

  On the other hand, the reaction detection means 83 can detect a face image of the face that has reacted to the call content (Yes in step S5), or if only one person has reacted to the call content (step S5). In step S6, the face registration unit 84 registers the face image detected by the reaction detection unit 83 in association with the call content in the memory unit 12 (step S7). For example, in the case of outputting a voice saying “I will explain XX. Please gather people who are interested”, the person of the face image detected by the reaction detection means 83 is called “I am interested in XX”. Register the contents as preference information.

  The voice output means 82 may change the language as the contents of the call. When calling “Look here” in a certain language (eg, Chinese), it is assumed that the person who reacts is a person who speaks a certain language (eg, a person who speaks Chinese). Therefore, the voice output means 82 speaks in a specific language, and the reaction detection means 83 detects the presence or absence of reaction detection. In this case, the face registration unit 84 registers the face image detected by the reaction detection unit 83, the face feature information, the calling content, and the calling language in the memory unit 12 in association with each other.

  On the other hand, when the reaction detection unit 83 can detect the face image of the face that has reacted to the contents of the call (Yes in step S5), and there are a plurality of persons who have reacted to the contents of the call ( In step S6, all of the face images of a plurality of names may be selected, but the selection means 85 selects one of the face images of the names of the persons who reacted to the contents of the call detected by the reaction detection means 83. A selection process for selecting a face image may be executed (step S8).

  As an example, the selection unit 85 uses a change in face area (a change in size, a change in direction, and a change in position) for face selection. For example, a face whose face is enlarged, whose face is in front, or in the middle of the captured image is selected. Further, the selection means 85 may select only the face image of the person who has responded to the contents of the call detected by the reaction detection means 83 and has the largest change.

  As another example, for example, when the voice output means 82 outputs a voice saying “I will explain XX. Please gather people who are interested”, the selection means 85 has one of the detected faces centered on the image. The head 20 provided with the camera unit 23 is operated via the operation mechanism 50 so that the position of the camera unit 23 provided in the head 20 and the target person's eyes is adjusted in this embodiment. (Operate to the position where the target person's face image is the center of the image)), the voice output means 82 again outputs a voice (for example, “Are you interested in XX?”) When the correct answer is recognized in the voice recognition of the voice recognition unit 30 for the voice signal input via the microphone 24 (for example, “Yes”), the detected face image at the center of the image is selected.

  When one face image is selected from the face images of the plurality of faces that have reacted to the content of the call detected by the reaction detection unit 83 as described above, the face registration unit 84 is selected by the selection unit 85. The face image and the contents of the call are associated with each other and registered in the memory unit 12 (step S7).

  As described above, according to the face registration device of the present embodiment, a call is made based on a change in the face area of a person from images captured by the camera unit 23 before and after the output of a call sound relating to a predetermined call content at a predetermined timing. By detecting face images that have reacted to the content and registering the detected face image and the call content in association with each other, it is easier to use services that use face information that could not be used unless you registered it Can be handled automatically. For example, it can be used for calling at reception systems such as banks, hospitals, and clinics, collecting customer information in signage such as stores, providing services according to the number of customers visiting the store, and detecting suspicious persons at ATMs.

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

DESCRIPTION OF SYMBOLS 1 Face registration apparatus 23 Imaging means 30 Voice recognition part 50 Operation | movement mechanism 81 Face detection means 82 Voice output means 83 Reaction detection means 84 Face registration means 85 Selection means

Claims (8)

Imaging means for imaging the surroundings of the device;
Face detection means for detecting a human face area from an image captured by the imaging means;
Voice output means for outputting a call voice according to a predetermined call content at a predetermined timing;
Reaction detection means for detecting a face image that reacts to the content of the call based on a change in the face area of the person by the face detection means before and after the output of the call voice by the voice output means;
When there are a plurality of face images that have reacted to the call contents detected by the reaction detection means, one face image is selected from among the plurality of face images that have reacted to the call contents detected by the reaction detection means. A selection means for executing a selection process to select;
Face registration means for registering the face image detected by the reaction detection means or the face image selected by the selection means in association with the call content;
A face registration device comprising: The reaction detection means detects a person who has changed more than a predetermined threshold for a change in the face area of the person before and after the output of the call voice as a person turned around in response to the call content,
The face registration device according to claim 1. The reaction detection means detects a person whose face area of the person is newly detected after outputting the call voice as a person who has reacted to the call content,
The face registration device according to claim 1. The selection means uses a change in the face area of the person for selection;
The face registration device according to claim 1 . The selection means is limited to selecting the face image of the face that has reacted to the call content detected by the reaction detection means, with the largest change.
The face registration device according to claim 1 . An operation mechanism for moving the position of the imaging means;
A voice recognition unit that performs voice recognition of an input voice signal;
With
The selection means operates the imaging means so that one of the face images reacted to the call content detected by the reaction detection means is positioned at a substantially center of the image, and again by the audio output means When the desired answer is voice-recognized through the voice recognition unit, the face image is selected.
The face registration device according to claim 1 . A computer that controls a face registration device that includes an imaging unit that images surrounding circumstances.
Face detection means for detecting a human face area from an image captured by the imaging means;
Voice output means for outputting a call voice according to a predetermined call content at a predetermined timing;
Reaction detection means for detecting a face image that reacts to the content of the call based on a change in the face area of the person by the face detection means before and after the output of the call voice by the voice output means;
When there are a plurality of face images that have reacted to the call contents detected by the reaction detection means, one face image is selected from among the plurality of face images that have reacted to the call contents detected by the reaction detection means. A selection means for executing a selection process to select;
Face registration means for registering the face image detected by the reaction detection means or the face image selected by the selection means in association with the call content;
Program to function as. A face registration method that is executed by a face registration device that includes an imaging unit that images surrounding circumstances,
A step of detecting a face area of a person from an image captured by the imaging means;
Audio output means for outputting a call audio according to a predetermined call content at a predetermined timing;
A step of detecting a face image that reacts to the content of the call based on a change in the face area of the person by the face detection unit before and after the output of the call voice by the voice output unit;
When there are a plurality of face images that the selection means has reacted to the call contents detected by the reaction detection means, one of the plurality of face images that have reacted to the call contents detected by the reaction detection means Executing a selection process of selecting a face image of
A step in which a face registration unit registers the face image detected by the reaction detection unit or the face image selected by the selection unit in association with the call content;
Including face registration method.

Images