JP6270784B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method, and more particularly, to an image processing apparatus and an image processing method capable of photographing a front surface and a back surface of a camera and reproducing and displaying the images obtained here.

  Conventionally, a camera has been a device for taking a still image. However, in recent years, since image signals have been digitized, moving images can be shot by performing inter-frame compression on images that have been acquired continuously. It has become possible to use as a device. For this reason, recent digital cameras are capable of shooting moving images as well as still images.

  In this way, digital cameras can shoot both still images and movies, but for users who have only taken still images, they do not know when to shoot movies, Even when I started shooting, I didn't know when to stop shooting, so it was often unsightly and sluggish.

  In recent years, image processing techniques have advanced and various proposals have been made. For example, Patent Document 1 proposes an image processing apparatus that reflects a user's facial expression in a game. That is, this image processing apparatus changes the scene to be displayed according to the user's facial expression, and displays a composite image in which a person other than the user is replaced with the user's head, thereby making it possible to immerse the video content. The experience is further improved.

  Patent Document 2 proposes a video camera that reflects a photographer's emotion in a photographed image. That is, the grip portion of the video camera is provided with a sensor. Based on the sensor output, the video camera is reddish when the user's tension is high, and bluish when the user is relaxed. The color of the image has been changed.

JP 2007-300562 A Japanese Patent Laid-Open No. 4-98799

  As described above, various image processing techniques have been proposed, but the image processing is not image processing for performing shooting reflecting the relationship between the photographer and the subject. That is, the image processing apparatus disclosed in Patent Document 1 is an image process for a display image in a game machine, and does not change a captured image. In addition, the video camera disclosed in Patent Document 2 changes the color tone according to the user's facial expression, and does not relate to image switching that can be enjoyed by the viewer during viewing.

  The present invention has been made in view of such circumstances, and provides an image processing apparatus and an image processing method capable of reflecting a photographer's state that cannot be read only by a subject image. Objective.

In order to achieve the above object, an image processing apparatus according to a first aspect of the present invention is provided with a display unit that displays an image of a subject and a photographer who is provided on the same surface as the display unit and observes the image of the subject . A shooting unit that outputs image data, an avatar information unit that stores an avatar image to be displayed by replacing the photographer, and a digitizing unit that digitizes the photographer 's gesture based on a change in the image data; depending on the value of the gesture by the upper Symbol quantifying section, and an image processing unit for switching the avatar images corresponding to the photographer which is stored in the avatar information portion from the image of the subject displayed image in the display unit Have .

The image processing apparatus according to the second aspect based on the first invention, the photographing direction of the photographing unit, a second imaging unit for imaging the different directions, the image of the subject, the It is the image image | photographed by the 2nd imaging | photography part.

An image processing apparatus according to a third invention is the image of the subject and the image of the avatar image captured by the imaging unit and the second imaging unit and processed by the image processing unit in the second invention. A recording unit for recording data;

According to a fourth aspect of the present invention, there is provided an image processing method in which an image of a subject is displayed on a display unit, and a photographer who observes the image of the subject is photographed by a photographing unit provided on the same surface as the display unit. And quantifying the photographer 's gesture based on the change in the image data, and displaying the display image on the display unit from the subject image to the avatar information unit according to the digitized numerical value of the gesture . Is switched to the avatar image corresponding to the photographer stored in the above.

  According to the present invention, it is possible to provide an image processing apparatus and an image processing method capable of reflecting a photographer's state that cannot be read only by a subject image.

It is a block diagram which shows the structure of the camera concerning 1st Embodiment of this invention. In the camera concerning 1st Embodiment of this invention, it is a figure which shows the combination of the front imaging | photography part 1a and a back imaging | photography part, (A) shows an example provided with both in one camera, (B) shows two (C) is a figure which shows the example of the combination of two cameras and the computer which cooperates with this. In the first embodiment of the present invention, it is a diagram showing a situation where a photographer is photographing the movement of a subject as a moving image, (a) shows a state of photographing, and (b) to (c) are taken. FIG. In 1st Embodiment of this invention, it is a figure which shows the change of the back image which image | photographed the photographer, (a) thru | or (d) show a time change. In 1st Embodiment of this invention, it is an example which shows the synthesis | combination of a front image and a back image, and is a figure which shows a mode that a front image and a back image are completely switched. It is a flowchart which shows the operation | movement of the camera control concerning 1st Embodiment of this invention. In 1st Embodiment of this invention, it is a figure explaining digitization of the photographer's facial expression which is a back image, (a) shows the example of a smile, (b) shows the example of a troubled face. It is a flowchart for detecting the smile degree R concerning 1st Embodiment of this invention. In the first embodiment of the present invention, it is a diagram for explaining the detection of the orientation of the photographer's face, (a) shows a state of facing the front, (b) is a diagram showing a state of facing sideways It is. It is a flowchart for determining the gesture concerning 1st Embodiment of this invention. In 1st Embodiment of this invention, it is a figure explaining the method of the photographer's display of a back image, (A) is an example which uses a back image as it is, (B) shows the example which uses an avatar. FIG. It is a flowchart which shows the operation | movement of the camera control concerning the 1st technique relevant to this invention. In the 1st technique relevant to this invention, it is a figure explaining the detection of a motion amount, (A) shows the whole front image, (B) is the elements on larger scale of a front image. It is a flowchart for detecting the amount of motion concerning the 2nd technique relevant to the present invention. It is a block diagram which shows the structure of the camera concerning the 2nd technique relevant to this invention. It is a flowchart which shows the operation | movement of the camera control concerning the 2nd technique relevant to this invention.

  Hereinafter, a preferred embodiment will be described using a camera to which the present invention is applied according to the drawings. The camera according to the first embodiment of the present invention includes an imaging unit for imaging the front of the camera, an imaging unit for imaging the rear of the camera, and two imaging units for acquiring moving images. In response to the operation of the release button, acquisition of moving image data is started and recorded on a recording medium. In addition, the camera in the first embodiment determines the photographer's facial expression and gesture, and when the aspect changes, the image of the photographer's face is captured by an imaging unit for capturing the rear of the camera, and By combining with the image data in front of the camera, the photographer's state that cannot be read only with the subject image is also reflected in the image.

  FIG. 1 is a block diagram showing the configuration of a camera according to the first embodiment of the present invention. A camera 10 is a digital camera that captures an image of a subject. The camera 10 includes a front photographing unit 1a, a rear photographing unit 1b, an image recording control unit 2, a mode determining unit 3, an operation unit 4, an avatar information unit 5, an image composition unit 7, and a display unit 8.

  The front photographing unit 1a is disposed in front of the main body of the camera 10 and is a photographing unit mainly for photographing a subject, and acquires a front image. As shown in FIG. 2A, the rear photographing unit 1b is disposed behind the main body of the camera 10, and is a photographing unit that mainly photographs the photographer's face, and acquires a rear image. Each of the front photographing unit 1a and the rear photographing unit 1b includes an optical system for forming a subject image and an image sensor that photoelectrically converts the subject image, and processes an image signal output from each image sensor. A processing unit or the like is provided, and each image data is output. In addition, you may comprise the front imaging | photography part 1a with a zoom lens.

  The image recording control unit 2 performs overall control of the camera 10 according to a stored program, and records and controls moving image data output from the front photographing unit 1a and the rear photographing unit 1b. The image recording control unit 2 includes an auxiliary recording unit 2a that temporarily stores image data output from the front photographing unit 1a and an auxiliary recording unit 2b that temporarily stores image data output from the rear photographing unit 1b. .

  In the present embodiment, the moving image data of the front image photographed by the front photographing unit 1a is always recorded. However, in a predetermined state, the moving image data of the rear rear image is synthesized and this synthesized image is recorded. . In order to combine the images, moving image data is temporarily stored in the auxiliary recording units 2a and 2b, respectively.

  The mode determination unit 3 determines the mode of the photographer's image (front image) and the subject's image (rear image). That is, the aspect determination unit 3 inputs image data from the front imaging unit 1a and the rear imaging unit 1b, and performs determination based on the image data. As the mode determination, a photographer's facial expression and gesture, whether or not there is a change in the subject, and the like are performed.

  The operation unit 4 is an operation member such as a release button or a zoom button provided on the camera 10. When the release button is operated, the camera 10 starts moving image shooting, and when the release button is operated again, the moving image shooting is stopped. The operation state of the operation unit 4 is transmitted to the image recording control unit 2. As will be described later, the avatar information section 5 stores information related to avatars in order to display a photographer by replacing it with an avatar.

  As shown in FIG. 2A, the display unit 8 includes a display device such as a liquid crystal monitor disposed on the back surface of the camera 10, and reproduces and displays a moving image recorded in the image recording control unit 2. Further, the moving image acquired by the front photographing unit 1a is displayed as a so-called live view for observing the subject image.

  The image combining unit 7 combines the image acquired by the front image capturing unit 1a and the image acquired by the rear image capturing unit 1b. The composite image generated by the image composition unit 7 is displayed on the display unit 8. The auxiliary recording units 2a and 2b described above are temporary storage units for synthesizing images using the image data acquired by each imaging unit.

  The image composition in the image composition unit 7 is performed by inputting the determination result of the aspect determination unit 3 and performing composition processing such as image switching based on the photographer's facial expression and the like, changes in the subject image, and the like. In the present embodiment, the moving image of the front image acquired by the front photographing unit 1b is always recorded. However, when a change in the aspect is detected by the aspect determining unit 3, the subject image (front image) acquired by the front photographing unit 1a is detected. The moving image of the photographer's face image (back image) acquired by the rear photographing unit 1b is combined with the moving image and recorded.

  In addition, as image synthesis, in addition to synthesizing the rear image with the front image like a parent-child image (see FIG. 3C), image processing such as black and white inversion of the front image (see FIG. 3D), Various processes such as a switching process (see FIG. 5) for completely switching images are included.

  Further, in the present embodiment, the front photographing unit 1a and the rear photographing unit 1b are provided integrally with the camera 10, but this is not limiting, and only one photographing unit is provided as shown in FIG. Two cameras 10a and 10b may be prepared, and the two cameras may be linked. In this case, the imaging unit of the camera 10a is used as the front imaging unit 1a, the imaging unit of the camera 10b is used as the rear imaging unit 1b, and the control unit (image recording control unit 2) of one camera is used as the other imaging unit. The camera is configured to control the photographing unit. The control communication in this case may be wired or wireless.

  Further, as shown in FIG. 2C, two cameras 10a and 10b each having only one photographing unit are prepared, and a system is constituted by a computer 20 for controlling both. That is, the imaging unit of the camera 10a is used as the front imaging unit 1a, the imaging unit of the camera 10b is used as the rear imaging unit 1b, and the computer 20 controls both cameras. Any device equipped with a wireless LAN or the like can exchange control signals and image data.

  Next, the operation in the present embodiment will be described. First, the use state of the camera in the present embodiment will be described with reference to FIGS. FIG. 3 shows a situation where the photographer is photographing the movement of the subject as a moving image. As shown in FIG. 3A, the photographer 30 is chasing the subject 31 while looking at the display unit 8 provided on the back surface of the camera 10. At the same time, the facial expression of the photographer 30 can be monitored by the rear photographing unit 1b provided on the back surface.

  At this time, the photographer 30 is chasing the subject 31 so hard that the expression hardly changes as shown in FIGS. The image at this time is not particularly interesting and does not need to be recorded. However, if an unexpected happening occurs in the subject, the facial expression changes as shown in FIG. 4C, and sometimes the other part is seen as shown in FIG. 4D.

  Such a change in the photographer's facial expression, etc. indicates that something happened, and there are cases where the facial expression is more interesting than the subject. In the present embodiment, such a photographer's image is used effectively to change the moving image.

  For example, when the photographer's facial expression changes, the moving image can be displayed by superimposing the photographer's facial expression on the front image (FIG. 3B) (FIG. 3C), Change the front image according to the facial expression of the photographer. As a method of changing the front image, for example, in the case of a surprised or troubled expression, the image may be changed to black and white to increase the sense of urgency (FIG. 3D).

  Further, as image synthesis, as shown in FIG. 3C, not only two images may be juxtaposed, but also completely switched as shown in FIG. That is, when the subject 31 is photographed by the front photographing unit 1a (FIG. 5A) and the facial expression of the photographer 30 changes, the rear image of the photographer 31 photographed by the rear photographing unit 1b. When the situation changes, it may be switched again to the front image of the subject 31 by the front photographing unit 1a (FIG. 5C).

  Next, the operation of the camera 10 in the first embodiment of the present invention will be described using the flowchart shown in FIG. In the present embodiment, as described with reference to FIGS. 3 to 5, when the photographer's facial expression is changed, the rear photographing unit 1 b can change the subject image or the photographer's face image. Are combined with the front image from the front photographing unit and recorded.

  If the camera control flow is entered, it is first determined whether or not the camera 10 is set to the shooting mode (S1). The camera 10 can be set in a shooting mode and a playback mode by a photographer. If the result of determination in step S1 is that a shooting mode has been selected, it is determined whether or not to perform shooting (S2). Whether or not to perform this shooting is determined by detecting whether or not the release button of the operation unit 4 has been pressed. In the present embodiment, when the release button is pressed once, moving image shooting starts, and when the release button is pressed again, moving image shooting ends.

  If the result of determination in step S2 is that shooting has not been made, processing returns to step S1. On the other hand, if it is photographing, front photographing and recording are performed (S11). In this step, the front photographing unit 1 a photographs the subject in front of the camera 10 and records it in the image recording control unit 2.

  Subsequently, gesture / expression determination is performed (S12). In step S12, the aspect determination unit 3 that inputs the image data from the front photographing unit 1a and the rear photographing unit 1b determines whether the photographer has a facial expression, for example, a smile or a troubled face. The determination of the smile level and the determination of the gesture will be described later with reference to FIGS. In this step, the direction of the photographer's face is also determined.

  Subsequently, it is determined whether or not there is a change in the orientation and facial expression of the photographer (S13). In this step, based on the result of the determination in step S12, it is determined whether or not there is a change in facial expression or face orientation. If there is no change as a result of the determination in step S13, the process returns to step S11 and the front photographing and recording are continued.

  If the result of determination in step S13 is that there is a change, it is next determined whether or not there is a change in the front image (S14). Even if there is a change as a result of the determination in step S13, if there is a change in the front image, the front subject image is more important, so the process returns to step S11 to continue the front shooting and recording. .

  On the other hand, if the result of determination in step S14 is that there is no change in the front image, recording is performed by the rear camera (S15). That is, when there is a change such as a change in the photographer's facial expression and there is no change in the front subject image (monotonic), in addition to shooting by the front shooting unit 1a, shooting by the rear shooting unit 1b The image data at this time is also recorded.

  Subsequently, it is determined whether or not the operation is finished (S16). In this step, it is determined whether or not the release button in the operation member 4 has been operated again. If the result of this determination is that the operation has not ended, it is determined whether or not there is a change in the front screen (S17). If there is no change in the front screen as a result of this determination, the process returns to step S15 in order to continue the photographing of the photographer by the rear photographing unit 1b.

  On the other hand, if the result of determination in step S17 is that there is a change in the front screen, recording by the rear camera (rear photographing unit 1b) is terminated (S19). The recording of the image data from the rear photographing unit 1b is finished, but the photographer's image monitoring by the rear photographing unit 1b is continued in order to determine the gesture / expression in step S12.

  Subsequently, a special effect is applied to the front image (S20). Special effects here include generation of a black-and-white inverted image as shown in FIG. 3 (d), and generation of a composite image of the subject image and the photographer's image on the parent-child screen as shown in FIG. 3 (c). The complete switching of images as shown in FIG. When the special effect is applied, the process returns to step S11 and the above-described processing is performed.

  If the result of determination in step S <b> 16 is that the operation has ended, shooting is ended (S <b> 18). That is, in this step, processing for ending the photographing operation is performed. When the photographing is finished, this flow is finished, and the process is executed from step S1.

  If the result of determination in step S1 is not shooting mode, it is playback mode. In this case, it is determined whether or not a playback image has been selected (S3). In this step, it is determined whether or not a reproduction image is selected from a plurality of moving images by a reproduction operation member (not shown) of the operation unit 4. If the result of this determination is that a replay image has not been selected, this flow is terminated and executed again from step S1.

  If the result of determination in step S3 is that a playback image has been selected, it is determined whether or not playback is to be performed (S4). If the result of determination is that playback is not performed, processing returns to step S3. On the other hand, in the case of reproduction, the image is reproduced (S5). In this step, the moving image recorded in the image recording control unit 2 is reproduced.

  When shooting a video from step S11 to step S20, if the scene is a monotonous scene and the photographer's facial expression or the like changes, the back image captured by the rear imaging unit 1b is inserted, so only the subject image is inserted. The photographer's situation that cannot be read with can be reflected in the image. When the image reproduction is finished, the process is finished, and the process is executed again from step S1.

  When a back image is inserted, some users may have resistance because their face is reproduced and displayed as it is. Therefore, an image image (avatar image) that is a part of the photographer may be displayed. That is, as shown in FIG. 11A, when the expression of the photographer 30a changes like the photographer 30b, the avatar 33a to the avatar 33b are changed according to the expression change, as shown in FIG. To change.

  The avatar information section 5 described above has avatar image data for recording / reproducing the avatars 33a and 33b. As will be described later, the avatar image digitizes facial expressions and the like, and therefore, the avatar image corresponding to this numerical value may be read and used.

  Next, a method for digitizing the facial expression into the numerical value R will be described with reference to FIGS. Fig. 7 compares a person's smile (Fig. 7 (a)) with a troubled or surprised face (Fig. 7 (b)). When both are compared, the eyes and mouth are characteristic. I understand.

  In other words, when the eyes are compared, it can be seen that if there are many white eyes above the line 61 connecting the centers of the eyes of both eyes, it is close to a smile. The mouth can also be determined based on the same concept. That is, if the area of the lower lip is larger than the line 62 connecting both ends of the mouth, the degree of smile increases. In the present embodiment, the higher the score is the more smiles are made, and the lower the score when surprised or troubled. This makes it easier for a control means such as a microcomputer to make a determination.

  A method of obtaining the smile level R using the determination method shown in FIG. 7 will be described using the flowchart shown in FIG. First, the eyes and mouth are detected (S151). In this method, a face is detected by the face detection unit 5d based on the output of the image sensor 3, and the eye and mouth are extracted from the detected face by pattern recognition.

  Subsequently, in steps S152 to S154, the smile level is detected from the area of the white of the eye. First, the area of the white eye above the line 61 connecting the eyes of both eyes is measured, and this value is set as EA (S152). Similarly, the area of the white eye below the line 61 is measured, and this value is set to EB (S153). Next, the difference between EA and EB obtained in this way is normalized by the sum of these, that is, (EA−EB) / (EA + EB) is obtained to obtain a numerical value RE (S154). It is determined that the larger the numerical value RE, the higher the smile degree.

  Next, in steps S155 to S157, the smile level is detected from the area of the lips. First, the area of the lips above the line 62 connecting both ends of the mouth is measured, and this value is set as LA (S155). Similarly, the area of the lips below the line 62 is measured, and this value is set as LB (S156). The difference between LA and LB thus obtained is normalized by the sum of these, that is, (LB−LA) / (LA + LB) is obtained to obtain a numerical value RL (S157). It is determined that the larger the numerical value RL, the higher the smile degree.

  Subsequently, the smile level R is obtained by adding the values RE and RL obtained in steps S154 and S157 (S158). The larger the value R, the closer to smile. Further, in order to increase the accuracy of determination of the smile level, the state near the arrow 64 in FIG. 7A is determined (S159). That is, if there is a shadow at the edge of the mouth and teeth are visible, the probability of smile is high, and this is determined. As a result of the determination, if teeth are visible and there is a shadow at the mouth end, one point is added to smile level R (S160).

  However, if there is a wrinkle between the eyebrows, it cannot be said that it is a smile, so determination is made in step S161. That is, it is determined whether or not there is a wrinkle between the eyes near the arrow 65 in FIG. 7B (S161). If the eyelid is detected between the eyebrows as a result of the determination, one point is deducted from the smile level R (S162). In this way, the numerical value (R) can be obtained such that the closer to a smile, the higher the score.

  In this way, the expression of the photographer's face can be quantified by the flow for obtaining the smile level R. If the face is constantly monitored by the rear photographing unit 1b and the numerical value of the facial expression is recorded, if the photographer's facial expression changes due to changes in the subject etc., the value of the smile level R changes. Judgment can be made.

  In the present embodiment, the degree of smile is determined based on whether there are many white eyes above the line connecting the centers of the eyes of both eyes. However, the present invention is not limited to this. Good. That is, in FIG. 7, the position ET at the highest point of the heel with respect to the width EH of the heel is obtained, and the smile degree can also be obtained by the ratio, ET / EH. When ET / EH increases, it becomes close to a smile, and when this ratio decreases and the inclination of the upper eye increases, it tends to move away from the smile. The facial expression may be digitized as appropriate by other methods.

  In addition, depending on the situation change, what appears on the photographer's face is not only a change in facial expression such as a smile level, but also the direction in which the photographer is facing. That is, as shown in FIG. 9, from the state of facing the front (FIG. 9 (a)), it may turn to the side (FIG. 9 (b)). The change in the orientation of the face can be determined by obtaining the center position of the face and the center position of the eyes, and quantifying (Δx) these time changes (image changes in continuous shooting).

  This change in face orientation will be described using the gesture determination flow shown in FIG. If this flow is entered, first, the center coordinates of the face are obtained (S251). Subsequently, eye center coordinates are obtained (S252). There are various methods for obtaining the center coordinates. For example, a face template may be prepared, and a center position obtained in advance for a template having a high degree of coincidence may be used.

  Subsequently, the previous coordinates and the coordinates obtained this time are compared (S253). That is, the shift amount of the face center coordinates and the shift amount of the eye center coordinates (Δx in FIG. 9) are obtained. Next, it is determined whether or not the deviation amount obtained in step S253 has changed by a predetermined amount (S256). The predetermined amount may be an amount that can be said to have changed the orientation of the photographer's face.

  As a result of the determination in step S256, if it is determined that the predetermined amount has changed, the presence of a gesture is stored (S257). As a result of the determination in step S256, if it is determined that the predetermined amount has not changed, or if the presence of a gesture is stored in step S257, the process returns to the original flow.

  As described above, the control unit (image recording control unit 2) of the camera 10 can perform the facial expression determination (smile degree R) flow illustrated in FIG. 8 and the gesture determination flow illustrated in FIG. 10 in parallel. In steps S12 and S13, the photographer's face orientation and facial expression change can be determined. Note that the determination here is not limited to the flow described above, and other elements may be determined.

Next, a first technique related to the present invention will be described with reference to FIGS. In the first embodiment of the present invention, the subject image and the image of the photographer's face are recorded such that the front image is always recorded, the orientation and expression of the photographer's face changes at the time of photographing, and does not change to the front image. Accordingly, the rear image by the rear photographing unit 1b is recorded, and image synthesis is performed using both images. In the present technology , both the front image by the front photographing unit 1a and the rear image by the rear photographing unit 1b are always recorded at the time of photographing, and both are reproduced according to the form of the subject image and the image of the photographer's face at the time of reproduction. It is determined whether or not to synthesize an image using an image.

The configuration of the present technology is the same as the configuration of the first embodiment shown in FIG. 1, and only the camera control flow shown in FIG. 6 is replaced with the flowchart shown in FIG. Explained.

  When the flow shown in FIG. 12 is entered, it is determined whether or not the photographing mode is set, as in the first embodiment (S1). If the result of this determination is that a shooting mode has been selected, it is determined whether or not to perform shooting (S2). As a result of this determination, if shooting is not performed, the process returns to step S1.

  If the result of determination in step S2 is that shooting is to be performed, shooting and recording are performed on the front and back surfaces (S31). In this step, the front photographing unit 1a and the rear photographing unit 1b respectively shoot and both image data are recorded in the image recording control unit 2. Subsequently, the amount of front movement is recorded (S32). The amount of front motion is determined by detecting a motion vector. Details will be described later with reference to FIGS. 13 and 14.

  Subsequently, back face expression digitization recording is performed (S33). In this step, the facial expression of the photographer, such as the smile level R described above, is digitized and recorded. When the back facial expression is digitized and recorded, it is determined whether or not the operation is finished as in step S16 (S34). If the result of this determination is that photographing end operation has not been performed, processing returns to step S31 and photographing / recording on the front and back surfaces is continued.

  On the other hand, if the result of determination in step S34 is that the operation has ended, a shooting end operation is performed as in step S18 (S35). When the photographing end operation is performed, the operation ends, and the process is executed again from step S1. In this way, at the time of shooting, both front and back images are recorded simultaneously by repeatedly executing steps S11 to S34. At this time, a numerical value of the amount of frontal motion and the expression is recorded simultaneously.

  If the result of determination in step S1 is not shooting mode, it is determined whether or not a playback image has been selected as in the first embodiment (S3). If the result of this determination is that it has not been selected, processing returns to step S1. On the other hand, if it is selected, as in the first embodiment, it is determined whether or not to start reproduction (S4). If the result of this determination is that playback has not started, processing returns to step S3.

  On the other hand, if the result of determination in step S4 is that reproduction has started, the front image is reproduced (S5). As described above, both the front image and the rear image are recorded in the image recording control unit 2, and the front image is read out from this and reproduced and displayed on the display unit 8.

  Next, it is determined whether or not there is a change in the numerical value representing the facial expression of the back image (S41). In this step, while reproducing the front image, the back face expression value recorded together with the image data in step S33 is detected, and it is determined whether or not this value has changed. If the result of determination in step S41 is that there is a change in the back face expression value, it is next determined whether or not the amount of frontal motion is less than or equal to a predetermined value (S42). In this step, while reproducing the front image, the amount of frontal motion recorded together with the image data in step S32 is detected, and it is determined whether this amount of motion has become a predetermined value or less.

  If the result of determination in step S42 is that the amount of front movement is less than or equal to a predetermined value, an avatar corresponding to the rear camera image is displayed (S46). That is, as a result of the determination in steps S41 and S42, there is no change in the front image taken by the front photographing unit 1a, and the facial expression of the photographer has changed in the rear image taken by the rear photographing unit 1b. In this case, an avatar having a facial expression corresponding to a change in facial expression of the photographer is read from the avatar information unit 5 and reproduced and displayed.

  By displaying the avatar, when the movement of the front screen is monotonous, it is possible to display an avatar that explains the situation based on the image with a change on the back as appropriate, and it is possible to prevent the viewer from getting bored Become. If an avatar is displayed, it will return to step S5 and will reproduce | regenerate a front image.

  As a result of the determination in step S41, if there is no change in the numerical value representing the rear display, or if the result of determination in step S42 is that the amount of frontal motion is greater than or equal to a predetermined value, it is determined whether or not the reproduction has ended ( S44). In this step, an operation state of a reproduction end operation member (not shown) in the operation unit 4 is detected and determined. If the result of this determination is that playback has not ended, processing returns to step S5 and playback of the front image is performed.

  If the result of determination in step S44 is that playback has ended, playback end operation is performed (S45). When the reproduction end operation is performed, the operation ends and the process is executed again from step S1.

  In addition, in this embodiment, when step S41 and S42 were passed by Yes, the avatar corresponding to a back image was displayed, but for the user who wants to see a front image, it is simultaneous with a back image. You may make it instruct | indicate reproduction | regeneration of the recorded front image.

  In this embodiment, the avatar is displayed in step S46. However, as in the first embodiment, a composite image may be generated and displayed using the photographer's image. It doesn't matter.

  Next, the detection of the motion amount performed in the recording of the motion amount in step S32 will be described with reference to FIGS. FIG. 13A shows the state of the subject 31 in the screen, and the screen portion 21 with light ink in the screen is shown in FIG. 13B. When the child who is the subject 31 raises his / her hand, the pixel of the hand portion is changed from the shadow 32a at the timing T = t1 to the position of the shadow 32b at the timing T = t2, as shown in the figure. . In the present embodiment, the amount of motion is detected by paying attention to the fact that the pattern itself of the shadows 32a and 32b does not change much and only the position changes.

  The flow of the motion amount shown in FIG. 14 is executed every time the front photographing unit 1a takes an image. If the flow for detecting the amount of movement is entered, first, an image at the center of the screen of the previous frame is detected (S301). In this step, the image data of the portion corresponding to the central portion of the screen is read out from the image data obtained by the previous photographing unit 1a.

  Subsequently, the image at the center of the screen of the current frame is detected (S302). In this step, the image data of the portion corresponding to the central portion of the screen is read out from the image data captured by the front photographing unit 1a this time. When this reading is completed, the two images are compared (S303). That is, the degree of coincidence between the image data read in step S301 and the image data read in step S302 is detected.

  Next, based on the comparison result in step S303, it is determined whether or not both images match (S304). If the result of this determination is that the images do not match, one image is shifted (S306). In this step, for example, image data obtained by shifting the image data of the previous frame by one pixel is generated. When shifting the pixels, the pixels are regularly shifted from front to back and from left to right.

  If one of the images is shifted, it is next determined whether or not the shift amount is a predetermined amount or more (S307). When there is a motion amount that is greater than or equal to the predetermined amount, there is a change in the motion of the subject 31, so there is no need to accurately determine the motion amount. If the result of determination in step S307 is that it is not greater than the predetermined amount, processing returns to step S303.

  On the other hand, if the result of determination in step S307 is that it is greater than or equal to a predetermined amount, the shift amount is determined to be maximum (S308). That is, it is determined that the subject 31 has changed due to considerable movement. When this determination is made, the original flow is restored.

  As a result of the determination in step S304, if the two images match as a result of the comparison between the two images, the shift amount when they match is set as the motion amount (S305). That is, in step S306, the image is shifted little by little in the front, rear, left, and right directions, but a shift amount from the first image as a reference is obtained, and this shift amount is set as a motion amount. If the amount of movement is the amount of movement, the original flow is restored.

  In this way, in the movement amount flow, the previous and current images are compared, and one image is shifted until they match. Then, the amount of shift when the values match is set as the amount of motion, and when the amounts of motion do not agree greatly, it is determined that the amount of shift is maximum. By quantifying the amount of movement in this way, it is possible to determine how much the image has moved and to set the level of image switching. Further, the determination amount may be switched depending on the shooting scene or the like. For example, it is better to set a large judgment amount for a subject that moves rapidly, such as an athletic meet scene. On the other hand, for a subject that moves little like a baby, the judgment amount is small. Good.

As described above, in the present technology , in step S31, the front image and the rear image are recorded at the same time, and at the time of reproduction, the front image and the rear photographer's face image are converted into the rear image. The image based on it is displayed. For this reason, a photographer's state that cannot be read only by the image of the subject can be reflected in the image.

Next, a technique related to the present invention will be described with reference to FIGS. In the first and second embodiments of the present invention, image composition such as image switching from the front image to the back image is performed after the change has occurred, but in the present technology , image switching or the like is performed before the change occurs. Like to do. In other words, since it is often better to display the situation where the change is occurring than to change the image after the change has occurred, the image is changed prior to the start of the change.

  FIG. 15 is a block diagram showing the configuration of the present embodiment. The same components as those in the first embodiment shown in FIG. 1 are denoted by the same reference numerals, and detailed description thereof is omitted. The timer unit 9 measures time and outputs the time information to the comparison and information providing unit 6.

  The comparison and information providing unit 6 includes a frontal movement amount M obtained based on the image data of the front image acquired by the front photographing unit 1a and a rear facial expression obtained based on the image data of the rear image acquired by the rear photographing unit 1b. The numerical value (degree of smile) R is compared. In addition, time information is input from the time measuring unit 9, and information such as a change signal indicating the timing of change determination is added to the image data. That is, when the front image is reproduced, the timing for switching to the back image is given to the image data as a change signal.

Next, the operation of the present technology will be described using the flowchart shown in FIG. In step S1 and step S2, as in the first embodiment, it is determined whether or not the shooting mode is selected. In the shooting mode, it is determined whether or not shooting is performed.

  If the result of determination in step S2 is shooting, front and back shooting are performed and image data is recorded (S51). In this step, the front image capturing unit 1a acquires the front image, the rear image capturing unit 1b acquires the rear image, and the image data of both images is recorded simultaneously.

  Subsequently, the front movement amount M is recorded based on the image data of the front image (S52). In this step, the motion amount flow shown in FIG. 14 is executed, and the shift amount when the values match is recorded as M. Note that the amount of motion M at this time is obtained in units of the number of pixels.

  Subsequently, the rear facial expression digitization R is recorded (S53). In this step, using the image data of the back image from the rear photographing unit 1b, the smile level R shown in FIG. 8 is executed to determine and record the smile level R. Next, the rear face expression digitization R of the previous frame is compared, and a difference ΔR is obtained (S54). In this step, the back face expression digitization (smiling degree) R is read last time, and a difference ΔR between this value R and the back face expression digitization R obtained in step S53 is obtained.

  Subsequently, it is determined whether or not the difference ΔR is larger than the front movement amount M / 320 (S55). In this step, the difference ΔR obtained in step S54 is compared with the value obtained by dividing the motion amount M recorded in step S52 by 320 to determine whether or not ΔR> M / 320.

  Here, the movement amount M is divided by 320 for the following reason. For example, when the movement amount M changes by 10% in the horizontal direction of the screen, it can be determined that the movement is considerable. For example, in the case of a camera with 8 million pixels, this movement is 3200 × 2400 pixels, and 320 pixels corresponding to 10% of the number of pixels corresponds to this movement. Therefore, the value obtained by dividing the amount of motion M by 320 pixels is the comparison target in step S55. The design value of 320 may be changed as appropriate depending on the number of pixels.

The difference ΔR in step S55 is a value indicating facial expression change, and the range of this facial expression change is normally −1 to 3, and if there is one or more changes, it is considered a large change. The movement amount M and the expression digitization R are numbers having different dimensions, and are changes that cannot be spoken by the same earthwork. However, in the present technology , the value obtained by dividing the amount of movement M by 320 and the difference ΔR in expression change are compared, and if the change in ΔR is larger, it is determined that the rear image is more interesting during viewing.

  If the result of determination in step S55 is that ΔR is greater than M / 320, a change signal is added to the image file (S58). That is, in this case, since the rear image changes more than the front image, and the rear image is more interesting during viewing, a change signal is given to the image data. Further, time information indicating the timing at which the facial expression changes is attached to the change signal.

  If a change signal is given to the image data in step S58, or if the result of determination in step S55 is that ΔR> M / 320, it is determined whether or not the operation is completed as in step S16 (S56). . If the result of this determination is that the operation has not ended, processing returns to step S51 and the aforementioned operation is executed. On the other hand, if the result of determination in step S <b> 56 is that the operation has ended, shooting is ended as in step S <b> 18 (S <b> 57). When the photographing is finished, the processing is executed again from step S1.

  If the result of determination in step S <b> 1 is not shooting mode, it is determined in step S <b> 3 and step S <b> 4 whether or not playback in playback mode is to be performed, as in the first embodiment. If the result of determination in step S4 is playback, change signal time determination is performed (S61).

  As described above, when the rear face image is more interesting than the front face image, the change signal is given in step S58. Therefore, in step S61, when reproducing the front image, first, time information indicating the timing of the change given to the change signal is detected.

  Subsequently, the front image is reproduced (S62). In this step, the front image recorded in the image recording control unit 2 is reproduced. Subsequently, it is determined whether or not it is a predetermined time before the change signal timing (S63). In step S61, since the timing to change is detected, it is determined in this step whether or not it is a predetermined time before the detected timing, for example, about one second before.

  If the result of determination in step S63 is that it is a predetermined time ago, the rear camera image is reproduced for a predetermined time (S64). In the reproduction of the rear image here, as shown in FIG. 5, the front image is completely switched to the rear image. As a result, the playback of the rear image is started not at the timing when the change ends but at a timing about 1 second before that. In a scene where the result of photographing by the rear photographing unit 1b is more interesting, it is possible to display the change in facial expression and make the image more enjoyable during viewing. When the reproduction of the rear image is finished, the process returns to step S61.

  If the result of determination in step S63 is that the timing of the change signal is not a predetermined time ago, it is determined whether or not playback has ended (S65). In this step, it is determined whether or not the reproduction of the selected reproduction image has been completed, or whether or not an operation member for ending reproduction has been operated. If the result of this determination is that playback has not ended, processing returns to step S61. On the other hand, if the reproduction has been completed, a process for terminating the reproduction is executed (S66). When the reproduction end process is finished, this flow is finished, and the process is executed again from step S1.

As described above, in the present technology , time information for performing image composition such as switching to a background image at the time of shooting is given, and an image such as switching to a background image before the occurrence of a facial expression change based on the time information at the time of playback. I'm trying to compose. For this reason, it is possible to display a situation where a change is occurring, and to play a more interesting moving image.

In this technology , whether the front image or the rear image is interesting is determined by movement and facial expression change.Of course, the facial expression change of the previous subject and the photographer's facial expression change are compared and switched. The movement of the image and the movement of the back face image may be compared and switched.

  Further, the rear image reproduced in step S64 is not completely switched as shown in FIG. 5, but may be a composite display as shown in FIG. 3B, or as shown in FIG. 3C. A black and white reversed image may be used. Further, the change signal may be attached to the image file not at the time of shooting but at the time of editing after shooting.

  As described above, in each embodiment of the present invention, if a movie is shot, monotonous shooting that always occurs is supplemented with a non-monotonic image to reduce it. It is possible to provide a moving image camera and an appreciation device that can be used.

  Further, in each embodiment of the present invention, the form of the subject image and the photographer's face image is detected, and the subject image and the photographer's face image are synthesized according to this form. For this reason, a photographer's state that cannot be read only by the image of the subject can be reflected in the image.

  In each embodiment of the present invention, an example is shown in which shooting is performed with the camera 10 and playback is performed. However, the present invention is not limited thereto, and playback can also be performed on a personal computer, a television, or the like. In the case of enjoying with a personal computer, a TV, or the like, image data of a moving image taken with the camera 10 may be stored.

  In each embodiment of the present invention, the camera 10 was shot and played back with a moving image. However, it is a matter of course that a still image shooting and playback display function may be provided as in a normal camera.

  Furthermore, in each embodiment of the present invention, a digital camera has been described as an apparatus for photographing. However, the camera may be a digital single lens reflex camera or a compact digital camera, such as a video camera or a movie camera. A camera for moving images may be used, and a camera built in a mobile phone, a personal digital assistant (PDA), or the like may of course be used.

  The present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. Moreover, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above embodiments. For example, you may delete some components of all the components shown by embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

DESCRIPTION OF SYMBOLS 1a ... Front imaging | photography part, 1b ... Back imaging | photography part, 2 ... Image recording control part, 2a ... Auxiliary recording part, 2b ... Auxiliary recording part, 3 ... Aspect determination part, 4 ... operation part, 5 ... avatar information part, 6 ... comparison and information giving part, 7 ... image composition part, 8 ... display part, 9 ... timekeeping part, 10 ... Camera 10a ... Camera 10b ... Camera 20 ... Computer 21 ... Screen part 30 ... Photographer 30a ... Photographer 30b ... Photographer 31 ..Subject, 32a ... shadow (T = t1), 32b ... shadow (T = t2), 33a ... avatar, 33b ... avatar, 31 ... subject, 61 ... both eyes A line connecting the pupils, 62 ... a line connecting both ends of the mouth, 64 ... an arrow, 65 ... an arrow

Claims (4)

A display for displaying an image of the subject ;
A photographing unit that is provided on the same surface as the display unit, photographs a photographer who observes the image of the subject, and outputs image data;
An avatar information section for storing an avatar image to be displayed by replacing the photographer;
A digitizing unit that digitizes the photographer 's gesture based on the change in the image data;
Depending on the value of the gesture by the upper Symbol digitizing unit, an image processing unit for switching a display image in the display unit on the avatar image corresponding to the photographer which is stored in the avatar information portion from the image of the subject,
An image processing apparatus comprising: The second photographing unit for photographing a direction different from the photographing direction of the photographing unit,
The image processing apparatus according to claim 1, wherein the image of the subject is an image captured by the second imaging unit. 3. The recording apparatus according to claim 2 , further comprising a recording unit that records the image of the subject and the image data of the avatar image captured by the imaging unit and the second imaging unit and processed by the image processing unit. Image processing apparatus. Display the subject image on the display,
The photographing unit provided on the same surface as the display unit photographs a photographer who observes the image of the subject , outputs image data,
Based on the change of the image data, the photographer 's gesture is digitized,
According to the digitized numerical value of the gesture, the display image on the display unit is switched from the image of the subject to an avatar image corresponding to the photographer stored in the avatar information unit .
An image processing method.