JP7804175B2 - Photographing device, information processing method, and program - Google Patents

Description

This technology particularly relates to a photography device, information processing method, and program that allows users to easily perform photography operations without moving from their standing position.

Photo sticker creation devices are known in the art. These devices photograph a user, allow the user to edit the photographed image, and print the edited image on sticker paper. Photo sticker creation devices are installed in amusement facilities and the like.

The flow of one game provided by the photo sticker creation device typically involves taking a photo of the user in a photo booth, then having the user move to an editing booth, editing the image according to the operations performed in the editing booth, and printing the edited image on sticker paper.

Operation of the photo sticker creation device in the photo booth is mainly performed by using your fingers to operate the screen displayed on a touch panel monitor. In the photo booth, a monitor is installed near the camera, etc., which is used to display live view images, which are images being taken.

Patent Document 1 discloses technology that identifies the three-dimensional position of fingers based on image data and constructs a skeletal model of the user's hand and fingers. The user's gestures are recognized based on the skeletal model, and operations such as turning off the lights in a room by waving one's hand are accepted.

JP 2013-037675 A

To ensure optimal image quality, users are often required to stand away from the camera when taking photos. If they change their position in response to instructions during the photo shoot, it can be difficult for them to operate the monitor display from that position.

This technology was developed in light of these circumstances, allowing users to easily perform shooting operations without having to move from their standing position.

An image capture device according to one aspect of the present technology includes a capture unit that captures a user in a photo booth as a subject; a display unit that displays a live view image representing the image being captured by the capture unit and an operation screen on which information for operation by hand gestures is displayed; a recognition unit that recognizes gesture operations performed by the user based on an image obtained by capturing the user after being guided to a standing position set at a position away from the display unit; and a control unit that executes processing in accordance with the gesture operations.

In one aspect of this technology, a user in a photo booth is photographed as a subject, and an operation screen is displayed that displays a live view image showing the image being photographed and information for operation using hand gestures. Furthermore, gesture operations performed by the user are recognized based on an image obtained by photographing the user after being guided to a standing position set at a location away from the display unit, and processing corresponding to the gesture operations is executed.

This technology allows users to easily perform shooting operations without moving from their standing position.

1 is a perspective view showing an external appearance of a photo sticker creation device according to an embodiment of the present technology; FIG. 2 is a perspective view showing an example of the front configuration of the imaging unit. 1 is a block diagram showing an example of the configuration of a photo sticker creation device; FIG. 2 is a block diagram illustrating an example of a functional configuration of the photo sticker creating device. FIG. 2 is a block diagram showing an example of the configuration of an image capture processing unit. 10 is a flowchart illustrating a photographing process. FIG. 10 is a diagram showing a display example of a guidance screen. FIG. 10 is a diagram showing a display example of a live view screen. FIG. 10 is a diagram showing a gesture operation. FIG. 10 is a diagram illustrating an example of a gesture operation. FIG. 10 is a diagram illustrating an example of timing at which a gesture operation is accepted. FIG. 10 is a diagram illustrating another example of the timing at which a gesture operation is accepted. 10A and 10B are diagrams illustrating examples of gesture operations performed outside the angle of view. FIG. 10 is a diagram showing a display example of a live view screen. FIG. 10 is a diagram showing a display example of a live view screen.

<Configuration of the photo sticker creation device>
FIG. 1 is a perspective view showing the appearance of a photo sticker creating device according to an embodiment of the present technology.

The photo sticker creation device 1 is a game machine that provides a photo creation game. The photo sticker creation device 1 is installed in a store such as a game center.

Users of the photo sticker creation device 1 are primarily high school girls and young women. Multiple users, typically two or three people per group, can enjoy the game with the photo sticker creation device 1. Of course, a single user can also enjoy the game with the photo sticker creation device 1.

In the game using the photo sticker creation device 1, the user takes a photo of themselves as the subject. The user then edits the captured image to combine it with composite images such as handwritten text and stamp images, creating a colorful image. The user then receives a sticker sheet with the edited captured image printed on it, completing the game sequence.

A pre-operation unit 11 is provided on the side of the photo sticker creation device 1, which has a box-shaped housing. The pre-operation unit 11 is a device used for operations before taking a photo, such as selecting the number of users to play with and selecting the photo course. The pre-operation unit 11 is provided with a touch panel monitor 11A. Below the touch panel monitor 11A is provided a coin slot 11B for paying the play fee.

A screen used for various selections is displayed on the touch panel monitor 11A. After making various selections according to the display on the touch panel monitor 11A, the user enters the photo booth through the entrance/exit G and takes their photo. The photo booth is occupied by a photo section 21. The space between the photo section 21 and the background section 22 forms the photo booth. The photo section 21 is equipped with a box-shaped camera unit 31 with a built-in camera.

An editing unit 51 is provided to the left of the pre-operation unit 11. The editing unit 51 is a device used to edit captured images obtained by taking pictures with a camera. Users who take pictures in the photo booth move to the editing booth in front of the editing unit 51 to edit the captured images. A tablet-integrated monitor 61, which is a monitor for editing captured images, is provided on a slope formed above the editing unit 51. A screen used to edit captured images is displayed on the tablet-integrated monitor 61.

Two editing booths are provided, one on either side of the editing unit 51 housing. One is located in the space in front of the editing unit 51 shown in Figure 1, and the other in the space behind the editing unit 51. The photo sticker creation device 1 allows a group of users who have finished taking photos in the photo booth to move to the available editing booth, allowing them to edit in parallel in the two editing booths.

Figure 2 is a perspective view showing an example of the front configuration of the imaging unit 21. The imaging unit 21 is configured to be surrounded by side panels 21A, 21B, and front panel 21C.

A camera unit 31 is provided at a position slightly above the center of the front of the imaging unit 21. The camera unit 31 is equipped with a camera 41 and a touch panel monitor 42.

Camera 41 is, for example, a single-lens reflex camera, and is attached inside camera unit 31 so that the lens is exposed. Camera 41 has an imaging element such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and captures images of users in the photo booth. Camera 41 functions as a photography unit that captures images of users as subjects.

Moving images (hereinafter also referred to as live view images) captured by the camera 41 are displayed in real time on the touch panel monitor 42. Still images captured by the camera 41 at a specified timing, such as when a shooting command is issued, are saved as captured images.

The touch panel monitor 42 is provided below the camera 41. The touch panel monitor 42 is composed of a monitor such as an LCD and a touch panel stacked on top of it. The touch panel monitor 42 functions as a live view monitor that displays moving images captured by the camera 41, and also displays various GUIs (Graphical User Interfaces) and accepts user selection operations.

Above the camera unit 31, an upper strobe unit 32 is provided, with its curved light-emitting surface facing the user. The upper strobe unit 32 emits light from above and in front of the user, primarily toward the user's face and upper body.

Below the camera unit 31 is a lower box strobe unit 33 that illuminates the user from below. Each of the upper strobe unit 32 and lower box strobe unit 33 is equipped with a strobe light source. The strobe light source emits light in synchronization with the timing of capturing a still image, illuminating the user, who is the subject, with strobe light.

A box-shaped portion 34L is provided between the lower box strobe unit 33 and the side panel 21A. Furthermore, a box-shaped portion 34R is provided between the lower box strobe unit 33 and the side panel 21B. The top surfaces of the box-shaped portions 34L and 34R are substantially horizontal and are used as a luggage area for users performing photography work to place their baggage, etc.

As will be described later, the photo sticker creation device 1 allows various operations during photography to be performed by gesture operations. The photo sticker creation device 1 is equipped with a function for recognizing gesture operations by the user. Gesture operations are performed, for example, using the hands. The movement of the user's hand is detected based on the image captured by the camera 41, and, for example, when a predetermined movement is performed, it is recognized as a gesture operation.

When gesture operations are possible, information such as buttons and slide bars that can be operated by gesture operations is displayed on the touch panel monitor 42. The user looks at the display on the touch panel monitor 42, moves their hand, and performs gesture operations to operate the camera during shooting.

<Internal configuration of the photo sticker creation device>
Fig. 3 is a block diagram showing an example of the configuration of the photo sticker creating device 1. In Fig. 3, the same components as those described above are given the same reference numerals, and duplicated explanations will be omitted where appropriate.

As shown in FIG. 3, the photo sticker creation device 1 is configured by connecting a pre-operation unit 11, a photographing unit 21, an editing unit 51, and a printing unit 103 to a PC (Personal Computer) unit 101 via an external input/output interface 102. The PC unit 101 is housed, for example, inside the housing of the editing unit 51.

The CPU (Central Processing Unit) 111, ROM (Read Only Memory) 112, and RAM (Random Access Memory) 113 that make up the PC section 101 are interconnected by a bus 114.

The CPU 111 executes predetermined programs and controls the overall operation of the photo sticker creation device 1. The ROM 112 stores programs and data executed by the CPU 111. The RAM 112 temporarily stores data and programs processed by the CPU 111.

An input/output interface 115 is also connected to the bus 114. A memory unit 116, a communication unit 117, and a drive 118 are connected to the input/output interface 115.

The storage unit 116 is a non-volatile storage medium such as a hard disk or flash memory. The storage unit 116 stores various setting information and other data supplied by the CPU 111. The information stored in the storage unit 116 is read out by the CPU 111 as appropriate.

The communication unit 117 is an interface for a network such as the Internet. The communication unit 117 communicates with external devices under the control of the CPU 111. The communication unit 117 transmits images selected by the user to, for example, a server managed by the manufacturer of the photo sticker creation device 1. Images transmitted from the communication unit 117 are allocated a specified memory area and saved, and can then be displayed on or downloaded to a mobile device that has accessed the server.

Removable media 119, such as an optical disk or semiconductor memory, is appropriately attached to the drive 118. Programs and data read from the removable media 119 by the drive 118 are supplied to the CPU 111, where they are stored in the memory unit 116 or installed.

The external input/output interface 102 is connected to the input/output interface 115. Control of each part by the PC unit 101 is performed via the external input/output interface 102.

The pre-operation unit 11 performs pre-selection processing for the user before photography. The pre-operation unit 11 is composed of a touch panel monitor 11A, a speaker 121, and a coin processing unit 122.

The touch panel monitor 11A displays various selection screens under the control of the PC unit 101 and accepts user operations on the selection screens. Input signals representing the user's operations are supplied to the PC unit 101, and various settings are made.

The coin processing unit 122 detects the insertion of coins into the coin slot 11B. When the coin processing unit 122 detects that a predetermined amount of coins has been inserted, it outputs a start signal to the PC unit 101 to instruct the start of the game.

The photography unit 21 performs photography processing on users in the photography booth. The photography unit 21 is composed of a lighting device 131, a camera 41, a touch panel monitor 42, and a speaker 132.

The lighting devices 131 are the strobe units in the photo booth, and emit light in accordance with the lighting control signal supplied from the PC unit 101. The lighting devices 131 include the strobe units installed in the photo booth, such as the upper strobe unit 32 and the lower box strobe unit 33.

The camera 41 takes pictures in accordance with shutter control by the PC unit 101 and outputs the image data obtained by the pictures to the PC unit 101.

The editing unit 51 performs post-photography editing. The editing unit 51 is composed of a tablet-integrated monitor 61, touch pens 141A and 141B, and a speaker 142. The tablet-integrated monitor 61 and touch pens 141A and 141B are provided on two sides of the editing unit 51 housing.

The tablet built-in monitor 61 displays an editing screen under the control of the PC unit 101 and accepts user operations on the editing screen. An input signal representing the content of the user's operation is supplied to the PC unit 101, and the captured image to be edited is edited.

The printing unit 103 is configured to include a printer 151. A sticker paper unit 152 is attached to the printer 151.

The printer 151 prints an image on sticker paper stored in the sticker paper unit 152 based on print data supplied from the PC section 101. The printer 151 ejects the sticker paper with the printed image from the sticker paper ejection port.

<Example of functional configuration of photo sticker creation device>
Fig. 4 is a block diagram showing an example of the functional configuration of the photo sticker creating device 1. At least some of the functional units shown in Fig. 4 are realized by the CPU 111 in Fig. 3 executing a predetermined program.

The photo sticker creation device 1 implements an information processing unit 201. The information processing unit 201 is composed of a pre-selection processing unit 211, a photography processing unit 212, an editing processing unit 213, a printing processing unit 214, and a transmission processing unit 215.

The pre-selection processing unit 211 executes pre-selection processing by controlling each unit of the pre-operation unit 11. The pre-selection processing involves selecting the number of users to play the game with, selecting a photography course, and so on. Information representing the user's selection is supplied to the photography processing unit 212.

The shooting processing unit 212 performs shooting processing by controlling each unit of the shooting unit 21. During the shooting processing, multiple shots are taken and multiple shot images, such as six, are generated. The shooting processing unit 212 outputs data for the multiple shot images generated by the shooting processing to the editing processing unit 213.

The editing processing unit 213 performs editing processing by controlling each part of the editing unit 51. Editing processing includes editing captured images, printing images on sticker paper, and sending images to an external image management server.

The images sent to the image management server can be viewed by the user on their mobile device, such as a smartphone, or downloaded to their mobile device after the user has finished playing the game on the photo sticker creation device 1.

The editing processing unit 213 outputs print data generated by arranging the edited captured image in a predetermined layout to the print processing unit 214. The editing processing unit 213 also outputs image data for transmission generated based on the edited captured image to the transmission processing unit 215.

The print processing unit 214 outputs the print data to the printer 151 and prints the image on sticker paper.

The transmission processing unit 215 controls the communication unit 117 to communicate with the image management server and transmit (upload) images to be transmitted to the image management server. Images generated in the game using the photo sticker creation device 1 are provided in the form of printed sticker paper, and also in the form of data via the image management server.

Figure 5 is a block diagram showing an example configuration of the imaging processing unit 212.

The photography processing unit 212 is composed of a photography control unit 221, a gesture recognition unit 222, and a control unit 223. Note that Figure 5 mainly shows the configuration related to gesture operations.

The shooting control unit 221 controls the camera 41 to take pictures. For example, the shooting control unit 221 outputs moving images captured by the camera 41 as live view images to the gesture recognition unit 222 and the display control unit 231.

The shooting control unit 221 also starts capturing images at a predetermined timing. After the start of shooting, multiple shots, such as six, are taken at predetermined intervals. Data for, for example, six captured images generated by the multiple shots is supplied to the display control unit 231, where it is used for display on the screen and also supplied to the editing processing unit 213 (Figure 4).

The gesture recognition unit 222 analyzes each frame of the live view image and estimates the skeleton of the user's hand. For example, the gesture recognition unit 222 detects the user's hand movement (gestures using the hand) based on the results of the skeleton estimation, and if it detects a predetermined movement, it recognizes that movement as a gesture operation. Recognition of such a gesture operation is performed at the acceptance timing, which is the timing at which a gesture operation can be accepted.

For example, gestures such as extending the index finger, pushing the hand with the index finger extended forward, and sliding the hand with the index finger extended up, down, left, or right are recognized as gestures. Furthermore, various movements such as spreading the index finger and middle finger to make a peace sign, spreading all five fingers to make a paper sign, and pointing the thumb up are also recognized as gestures.

In this way, a movement that combines hand and finger movements is recognized as a gesture operation using the hand. Information representing the recognition result by the gesture recognition unit 222 is supplied to the display control unit 231 and the processing execution unit 232.

The control unit 223 is composed of a display control unit 231 and a processing execution unit 232.

The display control unit 231 controls the display on the touch panel monitor 42 during shooting. For example, the display control unit 231 displays a live view screen including a live view image on the touch panel monitor 42 based on the live view image supplied from the shooting control unit 221.

The display control unit 231 also displays an operation screen on the touch panel monitor 42, on which information for gesture operations is arranged. Information for gesture operations may be provided on a live view screen, and the live view screen may be used as the operation screen. The display control unit 231 switches the display of the operation screen depending on the content of the gesture operation recognized by the gesture recognition unit 222.

The display control unit 231 may display an image such as a pointer at a position on the operation screen that corresponds to the position of the user's hand or finger performing the gesture operation. This makes it possible to improve visibility during gesture operation.

The processing execution unit 232 executes various processes according to the content of the gesture operation recognized by the gesture recognition unit 222. The processing execution unit 232 executes various processes other than those related to screen display, such as processes for adjusting the angle of view of the camera 41 and processes for changing background music during shooting.

In this way, the control unit 223, which is composed of the display control unit 231 and the processing execution unit 232, functions as a control unit that executes processing in response to gesture operations by the user.

<Shooting process flow>
The photographing process of the photograph sticker creating device 1 will be described with reference to the flowchart of Fig. 6. The process shown in Fig. 6 starts, for example, when a user who has started the game completes the pre-selection work and moves to the photographing booth.

In step S1, the display control unit 231 of the photography processing unit 212 displays a guidance screen on the touch panel monitor 42 that provides guidance on where to stand when taking a photo.

Figure 7 shows an example of the guidance screen display.

In the example in Figure 7, the message "First, stand on the 'bigger line'" is displayed, and below that is an image showing the standing positions provided on the floor of the photo booth. The user adjusts their standing position by following the instructions on the guide screen and checking the marks provided on the floor of the photo booth.

Footprint marks and other symbols indicating the positions of the two users are printed on the floor of the photo booth. The positions at which the users stand during photography are set in advance at a distance from the camera 41 to ensure that the angle of view, brightness, etc. are appropriate. From the position adjusted according to the guidance, it is difficult to touch the touch panel monitor 42 provided near the camera 41 and operate the buttons displayed on the touch panel monitor 42. In this way, the positions at which the users stand during photography are set at a position that makes it difficult for them to reach the touch panel monitor 42 even if they stretch out their hands.

The above-mentioned gesture operation is used to enable you to operate the display on the touch panel monitor 42 from the position you are standing in, even when you have adjusted your standing position according to the guidance. After the guidance on your position is given, capturing of a live view image begins.

In step S2, the display control unit 231 displays a live view screen based on the live view image supplied from the shooting control unit 221.

Figure 8 shows an example of the live view screen.

As shown in Figure 8, a live view image is displayed in a display area 301 formed approximately in the center of the live view screen. In the example of Figure 8, a moving image showing large images of the faces of two users is displayed as the live view image.

Below the display area 301, buttons 311-1 to 311-7, which are operated to select a stamp image, are displayed in a horizontal row. More buttons are provided to the left of button 311-1 and to the right of button 311-7. Each circular button displays an image that is a sample of the stamp image that will be selected when the corresponding button is pressed.

To the right of the display area 301 is an OK button 312, which is operated when finishing selecting the stamp image.

In this way, gesture operations are accepted while information such as buttons used for operation is displayed on the touch panel monitor 42. The timing for accepting gesture operations is after the guidance on standing positions has been provided.

In step S3 of FIG. 6, the gesture recognition unit 222 accepts a gesture operation.

In step S4, the gesture recognition unit 222 analyzes each frame of the live view image and recognizes gesture operations.

In step S5, the control unit 223 executes processing in accordance with the gesture operation. For example, if an operation to select a stamp image is performed by a gesture operation, the display control unit 231 displays the stamp image by superimposing it on the user's face shown in the live view image.

Note that if information such as buttons to be operated when adjusting the angle of view of the camera 41 is displayed on the touch panel monitor 42, the processing of step S5 in response to the gesture operation is performed by the processing execution unit 232 as appropriate.

In step S6, the shooting control unit 221 determines whether shooting has started. If it is determined in step S6 that shooting has not started, the process returns to step S2, and selection of a stamp image using the live view screen continues.

Figure 9 shows a user performing a gesture operation.

In the example of Figure 9, user U1, standing on the left side of camera 41, is performing a gesture operation using both hands, while user U2, standing on the right side of camera 41, is performing a gesture operation using one hand (the right hand).

In this way, the gesture recognition unit 222 recognizes the gesture operations of each of the multiple users who are playing the game together. Processing such as skeletal structure estimation is performed for each user.

Only gesture operations of users standing in a specific position may be recognized, such as gesture operations of a user standing on the left side of camera 41. The standing position of each user is identified based on the image captured by camera 41.

Figure 10 shows an example of gesture operation.

As shown in FIG. 10A, when button 311-4, which represents a stamp image of cat ears, is pressed by a gesture operation by either user U1 or user U2, the stamp image of cat ears is superimposed on the live view image and displayed. For example, the gesture operation to press button 311-4 is performed by moving a hand with the index finger extended to the position corresponding to button 311-4 and then, in that state, pressing forward (moving it toward the touch panel monitor 42).

Note that the hand H pressing button 311-4 in Figure 10A does not represent the user's hand touching and pressing button 311-4 with its finger, but is shown for convenience to indicate a gesture operation performed in a space separate from the touch panel monitor 42. The same applies to the other figures.

When button 311-4 is pressed, the positions of each user's eyes, nose, and other body parts displayed in the live view image are recognized, and a stamp image of cat ears is superimposed on the position of the head. Additionally, a stamp image of cat whiskers is superimposed on the position of each user's nose. Each stamp image continues to be displayed superimposed on the live view image.

In this way, on the live view screen, the user's selection is reflected in real time in the display of the live view image. If a different stamp image is selected, the type of stamp image superimposed on the live view image is switched.

As shown in FIG. 10B, when the OK button 312 is pressed by a gesture operation, the selection of the stamp image is completed. The gesture operation of pressing the OK button 312 is also performed by moving the hand with the index finger extended to the position corresponding to the OK button 312 and then pressing forward in that state. A movement different from the movement of pressing buttons 311-1 to 311-7 may also be recognized as the gesture operation of pressing the OK button 312.

If it is determined in step S6 of FIG. 6 that selection of a stamp image has been completed and that shooting should begin, then in step S7, the gesture recognition unit 222 stops accepting gesture operations.

As mentioned above, the captured image is an image to be provided to the user after editing and other processes have been performed and it is printed on sticker paper or provided to the user via a server that provides an image management service. In the game using the photo sticker creation device 1, gesture operations are accepted before the start of capturing the image to be provided.

In step S8, the shooting control unit 221 controls the camera 41 to capture a photographed image. Just before shooting, a countdown such as "3, 2, 1" is presented by voice or other means, and a still image is captured when the countdown reaches 0. A stamp image is composited with the still image obtained by shooting, and the photographed image is generated.

In step S9, the imaging control unit 221 determines whether imaging has ended. If it is determined in step S9 that imaging has not ended, the process returns to step S7, and imaging of captured images is repeated.

If it is determined in step S9 that shooting has ended because a predetermined number of shots, such as six, have been taken, then in step S10 the display control unit 231 causes an operation screen for gesture operation to be displayed on the touch panel monitor 42. For example, an operation screen is displayed that is used to select the captured image to be used for editing and the type of image processing to be performed on faces appearing in the captured image. The operation screen displays information such as buttons used for gesture operation.

In step S11, the gesture recognition unit 222 resumes accepting gesture operations.

In step S12, the gesture recognition unit 222 analyzes the image captured by the camera 41 and recognizes the gesture operation.

In step S13, the control unit 223 executes processing in response to the gesture operation. For example, if an operation to select a captured image to be edited from among six captured images is performed by a gesture operation, the processing execution unit 232 outputs the selected captured image to the editing processing unit 213 (Figure 4).

When operations using the operation screen are completed, the shooting process ends. In the editing process that takes place after the shooting process, editing is performed on the captured image selected using gesture operations.

As described above, in the photo booth, various selection operations are performed using gesture operations. Even after the user has adjusted their standing position according to the instructions, they can easily perform operations using the display on the touch panel monitor 42 without moving from that position.

In addition, the timing for accepting gesture operations is set to either before shooting begins or after shooting ends. By not accepting gesture operations while shooting, it is possible to prevent the user's hand movements to strike a pose from being recognized as gesture operations.

When taking a photo, the user typically makes various poses with their hands, but if these are recognized as gesture operations, unintended processing may occur. By not accepting gesture operations that occur when taking a pose for a photo, it is possible to prevent such unintended processing from occurring.

The following explains each aspect of gesture control in detail.

<1. Booth accepting gesture control>
Although the gesture operation is accepted in the photography booth in the above embodiment, it may be accepted in another booth formed in the photo sticker creating device 1 .

(1) Pre-booth Gesture operations may be used for operations in the pre-booth, which is a booth in front of the pre-operation unit 11 (FIG. 1). In this case, the pre-operation unit 11 is provided with a camera.

When a gesture operation is accepted, a screen containing information such as buttons for gesture operation is displayed on the touch panel monitor 11A of the pre-operation unit 11. Gesture operations performed by the user while viewing the display on the touch panel monitor 11A are recognized based on images captured by a camera provided in the pre-operation unit 11, and various selections are made as pre-selections.

(2) Editing Booth Gesture operations may be used in the editing booth. In this case, the editing unit 51 is provided with a camera.

When a gesture operation is accepted, a screen containing information such as buttons for gesture operation is displayed on the tablet-built-in monitor 61 of the editing unit 51. Gesture operations performed by the user while viewing the display on the tablet-built-in monitor 61 are recognized based on images captured by a camera provided in the editing unit 51, and various selections related to editing the captured images are made.

<2. Timing of gesture operation acceptance>
As described above, users in the photo booth pose with their hands. Because it is difficult to determine whether a user is posing or performing a gesture operation, the timing for accepting gesture operations is limited to the following times:

(1) Timing before and after photographing FIG. 11 is a diagram showing an example of timing for accepting a gesture operation.

After the user enters the photo booth, the live view screen begins to be displayed at time t1. The live view screen contains information for gesture operations, and gesture operations begin to be accepted as soon as the live view screen begins to be displayed.

Shooting begins at time t2, which is a certain amount of time after the live view screen starts displaying (the time when selection of a stamp image, etc. has finished). When shooting starts, acceptance of gesture operations is stopped. This suspension of acceptance of gesture operations continues until time t3, when the final image has been captured and shooting has ended. Gesture operations are not recognized until the final image has been captured.

At time t3, when the final image capture is completed, the operation screen is displayed on the touch panel monitor 42, and acceptance of gesture operations resumes. Information for gesture operations is displayed on the operation screen. Acceptance of gesture operations continues until time t4, when the display of the operation screen ends.

In this way, gesture operations are accepted before shooting begins and after shooting ends. The timing for accepting gesture operations shown in Figure 11 is the same as the timing described with reference to the flowchart in Figure 6. Only the timing before shooting begins may be set as the timing for accepting gesture operations, and gesture operations may not be accepted after shooting ends. For example, selection after shooting ends is performed by directly touching a button displayed on the touch panel monitor 42 with a finger.

Although gesture operations are accepted before the start of the first shooting and after the end of the final shooting, gesture operations may also be accepted before and after each shooting. In this case, even after the first shooting has started, gesture operations are accepted between each shooting, such as between the first and second shooting.

Figure 12 shows another example of the timing at which gesture operations are accepted.

As shown in FIG. 12, gesture operations are accepted from time t11 to time t12, which is the timing before a certain number of shots are taken. Furthermore, acceptance of gesture operations stops at time t12, when the countdown immediately before shooting begins. As described above, a countdown is presented by audio or other means before an image is taken.

A countdown such as "3, 2, 1" is performed, and after a photograph is taken at time t13, gesture operations are accepted again until time t14, when the countdown for the next photograph begins.

In this way, acceptance of gesture operations may be stopped when the countdown begins immediately before shooting. This also makes it possible to prevent the user's pose from being recognized as a gesture operation. In other words, by posing after the countdown has started, the user can distinguish between gesture operations and hand movements for posing.

(2) Timing during shooting After the first shooting has started, the user may be able to stop the shooting. For example, if the pause button displayed on the live view screen is operated, the shooting is paused. Gesture operations are accepted while the shooting is paused.

This allows users to temporarily stop shooting even after shooting has started, allowing them to use gesture controls to perform operations such as selection. Users can choose to use gesture controls and hand movements to pause.

<3. Gesture operation position>
FIG. 13 is a diagram showing an example of an image being captured while the live view screen is being displayed.

Frame F in Figure 13 indicates the range displayed on the live view screen as a live view image. The range indicated by frame F is cut out from the entire angle of view of the image captured by camera 41 and used as the live view image as described with reference to Figure 8. The user capturing the image can only see the range inside frame F.

When such an image is used as a live view image, hand movements that appear outside the live view image are accepted as gesture movements, while hand movements that appear inside the live view image are recognized as poses (not accepted as gesture movements). In other words, hand movements made outside the field of view of the live view image are accepted as gesture movements, while hand movements made within the field of view of the live view image are not accepted as gesture movements.

In the example of Figure 13, two users are moving their hands with their index fingers extended within the area outside frame F. Hand movements captured in this position are accepted as gesture operation movements. Gesture operations performed within the area outside frame F are accepted even during shooting (after shooting has started), for example.

This allows users to use gesture controls even while shooting. Users can make various selections while viewing the live view image and checking their appearance and the cropping range.

<4. Gesture operation recognition>
(1) Confirmation and Selection Various selections and confirmations are performed by gesture operations using information such as buttons displayed on the screen. For example, when the distance between the base of the index finger and the thumb is equal to or less than a predetermined distance, the movement is recognized as a gesture operation for confirming the selected content.

Also, if a specific finger remains in the same position for a specified period of time, that movement is recognized as a gesture operation that confirms the selection.

Swiping your hand sideways is recognized as a gesture to switch screens.

(2) Trimming FIG. 14 is a diagram showing a display example of a live view screen.

In the example of Figure 14, the entire range of the angle of view of the image captured by camera 41 is displayed as a live view image in display area 301. The user can select the range to use as the captured image by gesture operation. In the example of Figure 14, the range indicated by frame F indicates the currently selected range.

For example, by performing a downward gesture operation as indicated by thick arrow #1, the user can move the range of frame F downward and change the range to be cropped.

Such gesture operations for selecting a trimming range are recognized based on, for example, the skeletal structure of the fingers.

The gesture operation for selecting the trimming area may be recognized based on the relative positions of the thumb and index finger. The gesture operation for selecting the trimming area may be performed with one hand or with both hands.

(3) Enlargement/reduction, zoom in/zoom out When a movement is made such that the distance between the pad of the thumb and the pad of the index finger is equal to or less than a predetermined distance, the movement is recognized as a gesture operation for reducing the size of the image.

When a movement is made that brings the distance between the pad of the thumb and the pad of the index finger greater than a specified distance, that movement is recognized as a gesture operation to enlarge the image.

Similarly, a pinching motion with two fingers is recognized as a pinch-in gesture, and a spreading motion with two fingers is recognized as a pinch-out gesture.

In this way, various operations other than selecting items using buttons and confirming selections can be performed using gesture operations.

(4) BGM Control: Moving a hand with an index finger extended in front of the mouth is recognized as a gesture to lower the volume of the background music during shooting. This allows each user to adjust the volume of the background music.

(5) Others Various operations related to photography, such as adjusting the angle of the camera 41, starting photography, pausing photography, and setting the self-timer shutter, may be performed by gesture operations.

For example, the movement of placing a specific finger, such as an index finger, on a position corresponding to a button displayed on the screen is recognized as a gesture operation, and these operations are performed. By limiting the fingers used for gesture operations to specific fingers, it is possible to reduce erroneous operations. It is also possible to not limit the fingers used for gesture operations.

Also, shooting may be paused by a gesture operation performed outside the field of view of the live view image described above.

<5. Selections made using gestures>
(1) Options When multiple options are available, gesture manipulation is used to select one option. For example, gesture manipulation is used to select retouching or makeup. Retouching is image processing such as adjusting the size of eyes or skin color in an image. Makeup is image processing related to facial makeup, such as changing the color of lips or cheeks.

In addition, the operation of selecting content is performed by gesture operation. The content includes images for synthesis, such as the stamp image described with reference to Figure 8.

Selections regarding retouching and makeup processing are made, for example, when a live view screen such as that shown in FIG. 15 is displayed, and the selections are reflected in real time in the display of the live view image. In the example of FIG. 15, area 302L, which is the area for the user shown on the left side, is formed on the left side of display area 301, and area 302R, which is the area for the user shown on the right side, is formed on the right side of display area 301. Areas 302L and 302R each display buttons, slide bars, and other information for gesture operations.

By using gestures to operate buttons and slide bars, users can select the retouching and makeup processing options, which are then reflected in the live view image display. In this way, users can make various selections while viewing the live view image display. Because the selections are reflected in real time in the live view image display, it becomes easier to imagine the finished image that will be generated as a captured image.

(2) Designation of Position The part of the image to be changed may be selected by a gesture operation of pointing a finger at the live view image displayed on the touch panel monitor 42 .

This allows the user to adjust the appearance of each position on the live view image. In this way, gesture operations can be used to specify positions.

(3) Accepting one user at a time As described with reference to Fig. 15, multiple users may be allowed to perform gesture operations simultaneously, allowing each user to simultaneously make a selection according to their preference.

(4) Changing the Selection Content The selection content may be changed depending on the angle of view of the live view image, such as a full-body image or a close-up image. A full-body image is an image that shows the entire body of the user. A close-up image is an image that shows a large image of the face including the upper body.

<6. Gesture Control by Multiple People>
(1) When the Same Gesture Operation is Performed Simultaneously When a plurality of users perform the same gesture operation simultaneously, the gesture operation is accepted.

(2) When Different Gesture Operations are Performed Simultaneously When multiple users perform different gesture operations simultaneously, each of the gesture operations is accepted.

If multiple users simultaneously perform different gesture operations, a warning may be output to notify the user that the gesture operation cannot be accepted.

(3) When the user who performs the gesture operation is determined in advance The user who performs the gesture operation may be determined in advance. For example, only gesture operations performed by users who have been determined in advance by the photo sticker creating device 1 are accepted.

The user performing the gesture operation may be determined by their standing position, such as accepting a gesture operation from a user standing on the right. In this case, a message such as "The person standing on the right should perform the operation" is displayed when accepting a gesture operation.

The user performing the gesture operation may be determined by the gesture operation, such as by accepting a gesture operation from a user who points their palm toward the camera 41. In this case, a message such as "Please show us the palm of your hand" is displayed when the gesture operation is accepted. By analyzing the palm of the hand pointed toward the camera 41, it becomes easier to detect the skeleton, which can improve the recognition accuracy of subsequent gesture operations.

(4) Individually photographing palms before accepting gesture operations Before accepting a gesture operation, each user may be asked to hold their palm toward camera 41, and the skeleton of each user's palm may be estimated based on the image captured at that time. Estimating the skeleton of each user before accepting a gesture operation makes it possible to improve the accuracy of subsequent recognition.

<7. Standing position when using gestures>
Although gesture operations performed at a standing position set on the floor of the photography booth are accepted in the above embodiment, gesture operations performed at other positions may also be accepted. For example, the position of a chair placed away from the camera 41 is set as the position where the gesture operation is performed. The user will perform the gesture operation while sitting in the chair.

In this way, by setting the position where the gesture operation will be performed in advance, the approximate distance from the camera 41 to the user can be determined, making it possible to improve the recognition accuracy of the gesture operation.

8. Other
Output of Warning If recognition of a gesture operation fails or if the gesture operation is too fast, a warning may be output to notify the user of the failure or the speed of the gesture operation.

- Presentation of sample gesture operations A gesture introduction screen may be displayed to show the user what movements are recognized as gesture operations. The gesture introduction screen may display, for example, an image of a person performing a sample gesture operation.

For example, the gesture introduction screen is displayed on the touch panel monitor 11A of the pre-operation unit 11. This allows the user to check outside the photo booth what movements they should make before entering the photo booth and taking a photo.

A gesture introduction screen may be displayed on the touch panel monitor 42 installed in the photo booth.

Example of Mobile Device Application Operation Gesture operations may be used to operate an application, such as a selfie app, installed on a mobile device such as a smartphone. The selfie app is an application that allows a user to take a picture of themselves (take a selfie) using the smartphone's camera.

The user may be able to register in advance the details of each gesture operation, such as which movements are considered gesture operations. This allows the user to register movements that are easy for them to perform as gesture operations, allowing for smooth shooting and various other operations. Furthermore, because the user registers the movements themselves, they are less likely to forget which movements are paired with which gesture operations.

Different gesture operations may be used when starting video recording and when starting still image recording. This eliminates the need to change the recording mode (video mode or still image mode), such as by operating the screen of a selfie app.

The photo may be taken a predetermined time after the gesture operation of pressing the shutter button is accepted. This allows time to strike a pose after the gesture operation. With most selfie apps, the photo is taken immediately after the shutter is pressed, making it difficult for inexperienced users to strike a pose, but even such users can now take a selfie while striking a pose.

Gesture operations may be used to perform retouching on the screen of a selfie app.

1. Eye Deformation: If a user touches the area around their eyes (for example, under-eye bags) and then moves their index finger and thumb apart as a gesture, a retouching process is performed to widen the width of the eyes in the image.
- If the user touches the area around their eyes (for example, under-eye bags) and then moves their index finger and thumb vertically as a gesture operation, a retouching process is performed to increase the vertical width of the eyes in the image.
When the user performs a gesture operation in which they touch the area around their eyes (for example, under-eye bags), an adjustment bar is displayed.

2. Changing the distance between the left and right eyes When a user performs a gesture operation such as touching the inner corner of their eye, a retouching process is performed to narrow the distance between the left and right eyes in the image.
When a user performs a gesture operation in which they touch the corner of their eye, a retouching process is performed to widen the distance between the left and right eyes in the image.

3. Changing Eye Color When the user blinks (closes their eyes for one second or more) as a gesture operation, the eye color shown in the image is changed.

4. Changing the color of other parts of the face When the user performs a gesture operation in which they touch their cheek, a retouching process is performed to change the color of the blush on the cheeks shown in the image.
When the user performs a gesture operation in which they touch their lips, a retouching process is performed to change the lip color of the lips shown in the image.

5. Changing Hair Color When a user performs a gesture operation by touching their head, a retouching process is performed to change the hair color shown in the image.

6. Other Processing Gesture operations may be used for operations other than retouching processing.
When a user performs a gesture operation by touching their head, image processing is performed to composite content onto the part of the head that appears in the image.
- If the user performs a gesture operation by shaking their head from side to side, the previous operation will be canceled.
Example 1) After the eye transformation process, if the user performs a gesture operation of shaking their head from side to side, the eye transformation process is reset.
Example 2) If the user performs a gesture operation by shaking their head from side to side multiple times, all processing is reset.
Example 3) After the eye transformation process and the cheek color change process, if the user performs a gesture operation of shaking their head from side to side multiple times, the eye transformation process and the cheek color change process are reset.

・Examples of subjects to be photographed Gesture operations are recognized by estimating the skeletal structure of fingers, etc., so it is possible to capture images that do not include a person's face, which increases the degree of freedom in photographic scenes and poses.

Only the shutter operation can be performed by gesture operation. This allows for smoother photography of landscapes, etc.

Examples of Photographic Devices The gesture operations described above can be used in various types of photographic devices with a photo-taking function, such as photo sticker creation devices, smartphones, and cameras.

Regarding the program: The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program that constitutes the software is installed on a computer that is built into dedicated hardware, or on a general-purpose personal computer.

The program to be installed is provided by being recorded on removable media 119 shown in Figure 3, which may be an optical disc (CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc), etc.) or semiconductor memory. It may also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting. The program can be pre-installed in ROM 112 or storage unit 116.

The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or it may be a program that processes in parallel or at the required timing, such as when called.

In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device with multiple modules housed in a single housing, are both systems.

The effects described in this specification are merely examples and are not limiting, and other effects may also occur.

The embodiments of this technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of this technology.

For example, this technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by multiple devices via a network.

Furthermore, each step described in the above flowchart can be performed by a single device, or can be shared and executed by multiple devices.

Furthermore, if one step includes multiple processes, the multiple processes included in that one step can be executed by one device, or can be shared and executed by multiple devices.

1 Photo sticker creation device, 41 Camera, 42 Touch panel monitor, 21 Photography unit, 201 Information processing unit, 212 Editing processing unit, 221 Photography control unit, 222 Gesture recognition unit, 223 Control unit, 231 Display control unit, 232 Processing execution unit

Claims (14)

a photography unit that takes a photograph of a user in a photography booth as a subject;
a display unit that displays an operation screen on which a live view image showing an image being captured by the image capturing unit and a plurality of buttons used to select items, which serve as information for operation by hand gestures, are arranged;
a recognition unit that recognizes, as a gesture operation, an action of pressing a finger at a position corresponding to any one of the buttons, based on an image obtained by photographing the user after guidance on a standing position set at a position away from the display unit; and
and a control unit that executes processing in response to the gesture operation. The imaging device according to claim 1 , wherein the recognition unit recognizes the gesture operation based on a result of estimating a skeleton of the user's hand. The photographing device according to claim 1 , wherein the recognition unit accepts the gesture operation performed before starting to capture an image to be provided to the user. The photographing device according to claim 3 , wherein the recognition unit stops accepting the gesture operation while the image to be provided is being photographed. The imaging device according to claim 3 , wherein the recognition unit accepts the gesture operation performed outside an angle of view of the live view image, and does not accept the gesture operation performed within an angle of view of the live view image. The imaging device according to claim 5 , wherein the control unit temporarily suspends capturing of the image to be provided in response to the gesture operation being performed outside an angle of view of the live view image. The photographing device according to claim 1 , wherein the recognition unit recognizes the content of the gesture operation using one hand or both hands. The photographing device according to claim 1 , wherein the recognition unit recognizes the gesture operations performed by each of the users who play a game together. The photographing device according to claim 1 , wherein the recognition unit recognizes the gesture operation performed by a predetermined user among a plurality of users who share a game. The photographing device according to claim 1 , further comprising: another display unit provided outside the photographing booth, which displays a gesture introduction screen including an image serving as a sample of the gesture operation. The photographing device according to claim 1 , wherein the display unit displays a gesture introduction screen including an image serving as a sample of the gesture operation. The photographing device according to claim 1 , wherein the display unit displays a guide screen that guides the user to a standing position set in the photographing booth before displaying the operation screen. An information processing method for a device including an imaging unit, a display unit, and an information processing unit,
the photographing unit photographs the user in the photographing booth as a subject;
the display unit displays an operation screen on which a live view image showing an image being captured by the image capture unit and a plurality of buttons used to select items, which serve as information for operation by hand gestures, are arranged;
the information processing unit recognizes, based on an image obtained by photographing the user after being guided to a standing position set at a position away from the display unit , an action of pressing a finger at a position corresponding to one of the buttons as a gesture operation, and executes processing according to the gesture operation. A program executed by a computer that controls a device including an imaging unit, a display unit, and an information processing unit,
the photographing unit photographs the user in the photographing booth as a subject;
the display unit displays an operation screen on which a live view image showing an image being captured by the image capture unit and a plurality of buttons used to select items, which serve as information for operation by hand gestures, are arranged;
The information processing unit recognizes, based on an image obtained by photographing the user after being guided to a standing position set at a position away from the display unit , an action of pressing a finger at a position corresponding to one of the buttons as a gesture operation, and executes processing according to the gesture operation.