JP7736989B2 - Image processing device, image processing method, and program - Google Patents

Description

This technology relates to an image processing device, an image processing method, and a program, and in particular to an image processing device, an image processing method, and a program that can adjust the appearance of bangs in a captured image.

Photo sticker creation devices are known in the art. These devices photograph a user, allow the user to edit the photographed image, and print the edited image on sticker paper. Photo sticker creation devices are installed in amusement facilities and the like.

The flow of one game provided by the photo sticker creation device typically involves taking a photo of the user in the photography space as the subject, then moving the user to the editing space, editing the image according to the operations performed in the editing space, and printing the edited image on sticker paper.

Photographed images are edited using the editing screen displayed on a display in the editing space after moving from the shooting space to the editing space. The editing screen displays the photographed image to be edited, as well as buttons used to select various editing tools.

For example, Patent Document 1 describes a technology that allows an arbitrary range to be specified using a copy range frame, and multiple composite stamp images included in the specified range to be copied to another location as a single stamp image.

Japanese Patent Application Laid-Open No. 2019-109579

In the technology described in Patent Document 1, the content that can be copied is a stamp image. Users cannot perform operations such as copying any part of a person in a captured image and pasting it in another location.

This technology was developed in light of these circumstances, and makes it possible to adjust the appearance of bangs in captured images.

An image processing device according to one aspect of the present technology includes a cropping unit that crops, as an image to be synthesized, an image within a range corresponding to a selection operation performed by a user within a selectable range set as part of the area of a person appearing as a subject in a captured image, and a display control unit that composites and displays the image to be synthesized at any position on the captured image selected by the user.

In one aspect of this technology, an image within a selectable range set as part of the area of a person appearing as a subject in a captured image, in response to a selection operation performed by a user, is extracted as an image to be synthesized, and the image to be synthesized is synthesized at any position on the captured image selected by the user.

This technology makes it possible to improve the appearance of bangs in captured images.

1 is a perspective view showing an external appearance of a photo sticker creation device according to an embodiment of the present technology; FIG. 2 is a perspective view showing an example of the front configuration of the imaging unit. 1 is a block diagram showing an example of the configuration of a photo sticker creation device; FIG. 2 is a block diagram illustrating an example of a functional configuration of the photo sticker creating device. FIG. 10 is a diagram illustrating a bangs retouch function. FIG. 2 is a block diagram showing an example of the configuration of an edit processing unit. FIG. 10 is a block diagram illustrating an example of the configuration of a bangs retouching processing unit. 10 is a flowchart illustrating the operation of the photo sticker creating device. 9 is a flowchart illustrating the editing process performed in step S3 of FIG. 8. FIG. 10 is a diagram showing a display example of a mobile graffiti screen. FIG. 11 is an enlarged view of the palette area of FIG. 10 . 10A and 10B are diagrams illustrating an example of an operation using a clipping pen for bangs retouching. 10A and 10B are diagrams illustrating another example of an operation using the clipping pen for bangs retouching. FIG. 10 is an enlarged view of a bangs image. FIG. 10 is a diagram illustrating reduction and rotation of a bangs image. 10A and 10B are diagrams illustrating examples of selecting a position to paste a bangs image. FIG. 10 is a diagram illustrating an example of combining a bangs image. FIG. 10 is a diagram showing another example of selection of the cutout range. 10A and 10B are diagrams illustrating an example of an operation using a dedicated eraser. FIG. 10 is a diagram showing an example of a photographed image onto which a stamp image is combined. 10A and 10B are diagrams illustrating an example of an operation using a normal eraser. 10A and 10B are diagrams illustrating an example of an operation using a dedicated eraser. FIG. 10 is a diagram illustrating an example of a message.

<Configuration of the photo sticker creation device>
FIG. 1 is a perspective view showing the appearance of a photo sticker creating device according to an embodiment of the present technology.

The photo sticker creation device 1 is a game machine that provides a photo creation game. The photo sticker creation device 1 is installed in a store such as a game center.

The users of the photo sticker creation device 1 are primarily high school girls and young women. Multiple users, typically two or three people per group, can enjoy the game with the photo sticker creation device 1. Of course, a single user can also enjoy the game with the photo sticker creation device 1.

In the photo creation game using the photo sticker creation device 1, the user takes a photo of themselves as the subject. The user then edits the captured image to combine it with composite images such as handwritten text and stamp images, creating a colorful image. The user then receives a sticker sheet with the edited captured image printed on it, completing the game sequence.

A pre-operation unit 11 is provided on the side of the photo sticker creation device 1, which has a box-shaped housing. The pre-operation unit 11 is a device used for operations before taking photos, such as selecting the number of users to play with and selecting the photo course. The pre-operation unit 11 is provided with a touch panel monitor 11A. Below the touch panel monitor 11A is provided a coin slot 11B for paying the play fee.

A screen used for various selections is displayed on the touch panel monitor 11A. After making various selections according to the display on the touch panel monitor 11A, the user enters the photography space through the entrance/exit G and begins photography. The photography unit 21 is provided at the front of the photography space. The space between the photography unit 21 and the background unit 22 forms the photography space. The photography unit 21 is provided with a box-shaped camera unit 31 with a built-in camera.

An editing unit 51 is provided to the left of the pre-operation unit 11. The editing unit 51 is a device used to edit captured images obtained by taking pictures with a camera. After taking pictures in the photography space, the user moves to the editing space in front of the editing unit 51 and edits the captured images. A tablet-integrated monitor 61, which is a monitor for editing captured images, is provided on a slope formed above the editing unit 51. The screen used to edit the captured images is displayed on the tablet-integrated monitor 61.

Two editing spaces are provided, one on either side of the housing of the editing unit 51. One editing space is formed in front of the editing unit 51 shown in Figure 1, and the other is formed behind the editing unit 51. This allows a group of users who have finished shooting in the shooting space to move to the available editing space, and edit in parallel in the two editing spaces.

Figure 2 is a perspective view showing an example of the front configuration of the imaging unit 21. The imaging unit 21 is configured to be surrounded by side panels 21A, 21B, and front panel 21C.

A camera unit 31 is provided at a position slightly above the center of the front of the imaging unit 21. The camera unit 31 is equipped with a camera 41 and a touch panel monitor 42.

Camera 41 is, for example, a single-lens reflex camera, and is attached inside camera unit 31 so that the lens is exposed. Camera 41 has an imaging element such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and captures images of users in the shooting space. Camera 41 functions as a shooting unit that captures images of users as subjects.

Moving images (hereinafter also referred to as live view images) captured by the camera 41 are displayed in real time on the touch panel monitor 42. Still images captured by the camera 41 at a specified timing, such as when a shooting command is issued, are saved as captured images.

The touch panel monitor 42 is provided below the camera 41. The touch panel monitor 42 is composed of a monitor such as an LCD and a touch panel stacked on top of it. The touch panel monitor 42 functions as a live view monitor that displays moving images captured by the camera 41, and also displays various GUIs (Graphical User Interfaces) and accepts user selection operations.

Above the camera unit 31, an upper strobe unit 32 is provided, with its curved light-emitting surface facing the user. The upper strobe unit 32 emits light from above and in front of the user, primarily toward the user's face and upper body.

Below the camera unit 31 is a lower box strobe unit 33 that illuminates the user from below. Both the upper strobe unit 32 and the lower box strobe unit 33 are equipped with strobe light sources. The strobe light sources emit light in synchronization with the timing of capturing a still image, illuminating the user, who is the subject, with strobe light.

A box-shaped portion 34L is provided between the lower box strobe unit 33 and the side panel 21A. Furthermore, a box-shaped portion 34R is provided between the lower box strobe unit 33 and the side panel 21B. The top surfaces of the box-shaped portions 34L and 34R are generally horizontal and are used as a luggage storage area for users performing photography work to place their baggage, etc.

<Internal configuration of the photo sticker creation device>
Fig. 3 is a block diagram showing an example of the configuration of the photo sticker creating device 1. In Fig. 3, the same components as those described above are given the same reference numerals, and duplicated explanations will be omitted where appropriate.

As shown in FIG. 3, the photo sticker creation device 1 is configured by connecting a pre-operation unit 11, a photographing unit 21, an editing unit 51, and a printing unit 103 to a PC (Personal Computer) unit 101 via an external input/output interface 102. The PC unit 101 is housed, for example, inside the housing of the editing unit 51.

The CPU (Central Processing Unit) 111, ROM (Read Only Memory) 112, and RAM (Random Access Memory) 113 that make up the PC section 101 are interconnected by a bus 114.

The CPU 111 executes predetermined programs and controls the overall operation of the photo sticker creation device 1. The ROM 112 stores programs and data executed by the CPU 111. The RAM 112 temporarily stores data and programs processed by the CPU 111.

An input/output interface 115 is also connected to the bus 114. A memory unit 116, a communication unit 117, and a drive 118 are connected to the input/output interface 115.

The storage unit 116 is a non-volatile storage medium such as a hard disk or flash memory. The storage unit 116 stores various setting information and other data supplied by the CPU 111. The information stored in the storage unit 116 is read out by the CPU 111 as appropriate.

The communication unit 117 is an interface for a network such as the Internet. The communication unit 117 communicates with external devices under the control of the CPU 111. The communication unit 117 transmits images selected by the user to, for example, a server managed by the manufacturer of the photo sticker creation device 1. Images transmitted from the communication unit 117 are allocated a specified memory area and saved, and can be displayed on or downloaded to a mobile device that has accessed the server.

Removable media 119, such as an optical disk or semiconductor memory, is appropriately attached to the drive 118. Programs and data read from the removable media 119 by the drive 118 are supplied to the CPU 111, where they are stored in the memory unit 116 or installed.

The external input/output interface 102 is connected to the input/output interface 115. Control of each part by the PC unit 101 is performed via the external input/output interface 102.

The pre-operation unit 11 performs pre-selection processing for the user before photography. The pre-operation unit 11 is composed of a touch panel monitor 11A, a speaker 121, and a coin processing unit 122.

The touch panel monitor 11A displays various selection screens under the control of the PC unit 101 and accepts user operations on the selection screens. Input signals representing the user's operations are supplied to the PC unit 101, and various settings are made.

The coin processing unit 122 detects the insertion of coins into the coin slot 11B. When the coin processing unit 122 detects that a predetermined amount of coins has been inserted, it outputs a start signal to the PC unit 101 to instruct the start of the game.

The photography unit 21 performs photography processing targeting users in the photography space. The photography unit 21 is composed of a lighting device 131, a camera 41, a touch panel monitor 42, and a speaker 132.

The lighting devices 131 are strobe units within the shooting space, and emit light in accordance with lighting control signals supplied from the PC unit 101. The lighting devices 131 include strobe units installed in the shooting space, such as the upper strobe unit 32 and the lower box strobe unit 33.

The camera 41 takes pictures in accordance with shutter control by the PC unit 101 and outputs the image data obtained by the pictures to the PC unit 101.

Editing unit 51 performs post-shooting editing. Editing unit 51 is composed of a tablet-integrated monitor 61, touch pens 141A and 141B, and speaker 142. A set of tablet-integrated monitor 61 and touch pens 141A and 141B is provided on two sides of the housing of editing unit 51. When there is no need to distinguish between touch pen 141A and touch pen 141B, they will be collectively referred to as touch pen 141, as appropriate.

The tablet's built-in monitor 61 displays an editing screen under the control of the PC unit 101 and accepts user operations on the editing screen. An input signal representing the user's operation is supplied to the PC unit 101, which then edits the captured image to be edited.

The printing unit 103 is configured to include a printer 151. A sticker paper unit 152 is attached to the printer 151.

The printer 151 prints an image on sticker paper stored in the sticker paper unit 152 based on print data supplied from the PC section 101. The printer 151 ejects the sticker paper with the printed image from the sticker paper ejection port.

<Example of functional configuration of control unit>
Fig. 4 is a block diagram showing an example of the functional configuration of the photo sticker creating device 1. At least some of the functional units shown in Fig. 4 are realized by the CPU 111 in Fig. 3 executing a predetermined program.

The photo sticker creation device 1 implements a control unit 201. The control unit 201 is composed of a pre-selection processing unit 211, a photography processing unit 212, an editing processing unit 213, a printing processing unit 214, and a transmission processing unit 215.

The pre-selection processing unit 211 executes pre-selection processing by controlling each unit of the pre-operation unit 11. The pre-selection processing involves selecting the number of users, the photography course, and so on. Information representing the user's selection is supplied to the photography processing unit 212.

The shooting processing unit 212 performs shooting processing by controlling each unit of the shooting unit 21. During the shooting processing, multiple images, such as six images, are taken. The shooting processing unit 212 outputs data for the multiple images obtained by performing the shooting processing to the editing processing unit 213.

The editing processing unit 213 performs editing processing by controlling each part of the editing unit 51. Editing processing includes editing captured images, printing images on sticker paper, and sending images to an external image management server.

The images sent to the image management server can be viewed or downloaded by the user on their mobile device, such as a smartphone, after the user has finished playing the game on the photo sticker creation device 1.

Editing a captured image includes the process of generating an image to be printed on sticker paper based on the captured image, and the process of generating an image to be sent to an image management server based on the captured image.

The image to be printed on the sticker paper is generated by having the user select a sticker design and then performing image processing according to the sticker design selected by the user. When generating the image to be printed on the sticker paper, it is also possible to combine a design-oriented image with the sticker design.

On the other hand, portable images, which are images to be sent to the image management server, are generated by applying image processing to the photographed image to be edited in accordance with operations performed using a tool selected by the user from a selection of pre-prepared editing tools, such as a pen tool or stamp tool. By using the pen tool or stamp tool, a pen image or stamp image is composited into the photographed image to be edited. The user selects their preferred editing tool and selects the location on the photographed image to be edited where the image processing will be applied, thereby proceeding with editing the photographed image.

In this way, editing of captured images, particularly for creating portable images, is carried out by the user using the touch pen 141 in a manner similar to doodling on the captured image. Hereinafter, editing will be referred to as "doodle" where appropriate. Editing of captured images to create images to be printed on sticker paper will be referred to as "sticker doodling," and editing of captured images to create portable images will be referred to as "portable doodling."

As will be explained in more detail later, a bangs retouching function is provided as a mobile doodling tool. Bangs retouching is a function used to adjust the bangs of a person appearing in a photographed image. By using the bangs retouching function, it is possible to adjust the length of the bangs by increasing the amount of hair or lengthening them, as shown by the dashed line in the bottom row of Figure 5.

The bangs retouching function allows the bangs to be styled using a touch pen 141. The bangs retouching function may also be accessed by touching the screen with a finger.

The editing processing unit 213 in Figure 4 outputs the print data generated by the sticker graffiti processing to the print processing unit 214, and outputs the portable image data to the transmission processing unit 215.

The print processing unit 214 outputs the print data to the printer 151 and prints the image on sticker paper.

The transmission processing unit 215 controls the communication unit 117 to communicate with the image management server and transmit (upload) portable images to the image management server.

Figure 6 is a block diagram showing an example configuration of the editing processing unit 213.

The editing processing unit 213 is composed of an image analysis unit 221, a sticker image generation unit 222, a sticker design data storage unit 223, a portable image generation unit 224, and a display control unit 225. The captured image data output from the capture processing unit 212 is input to the image analysis unit 221 and the display control unit 225.

The image analysis unit 221 analyzes the features of the user that appear in each captured image and recognizes the position of each part of the user that appears in the captured image. For example, the positions of the head, eyes, nose, mouth, and ears are recognized. The position of the contour points of each part is also recognized. The shape of each part is identified based on the position of the contour points. Information representing the recognition results by the image analysis unit 221 is supplied to the sticker image generation unit 222 and portable image generation unit 224 along with the captured image data.

The sticker image generation unit 222 reads information about the sticker design selected by the user from the sticker design data storage unit 223 and performs image processing on the captured image based on the read information. For example, an image for a sticker is generated by combining an image of the design selected by the user.

Print data for the sticker image is supplied to the print processing unit 214. Information indicating the content of image processing by the sticker image generation unit 222 is supplied to the display control unit 225 as appropriate and used for screen display. The image processing by the sticker image generation unit 222 is processing for creating an image to be printed on sticker paper.

The portable image generation unit 224 applies image processing to each captured image according to the portable doodles made by the user, generating a portable image. The portable doodles are doodles used to create images to be sent to the image management server. Images doodled using the portable doodles are images for transmission, used to send to the image management server.

The portable image generated by the portable image generation unit 224 is supplied to the transmission processing unit 215. Information indicating the content of the image processing by the portable image generation unit 224 is supplied to the display control unit 225 as appropriate and used for screen display.

The portable image generation unit 224 includes a bangs retouching processing unit 231. The bangs retouching processing unit 231 performs image processing to retouch bangs in response to user operations using a bangs retouching tool.

The display control unit 225 displays various screens, including a sticker doodle screen and a mobile doodle screen, on the tablet built-in monitor 61. The display control unit 225 controls the display of the screen in response to user operations.

Figure 7 is a block diagram showing an example configuration of the bangs retouching processing unit 231.

The bangs retouching processing unit 231 is composed of a cutout unit 241 and a pasting unit 242. The data of the photographed image of the graffiti target output from the image analysis unit 221 is input to the cutout unit 241 and the pasting unit 242, respectively.

The cropping unit 241 sets a selectable range for each captured image. As will be described in detail later, the selectable range is the range within which the cropping range can be selected. The selectable range is set, for example, to a portion of the area of the person appearing in the captured image based on the characteristics of the person represented by the analysis results of the image analysis unit 221. The portion of the area is an area that includes the area showing the bangs.

If multiple people appear in a single captured image, a selectable area will be set for each person.

In addition, the cropping unit 241 crops out the bangs portion of the person appearing in the photographed image that is the subject of the graffiti as an image to be combined, in response to a user operation using the touch pen 141. Here, cropping out the image to be combined is a process of copying the image of the bangs portion selected by the user, while leaving the original bangs portion appearing in the photographed image intact.

The compositing image of the bangs portion cut out by the cutout unit 241 is supplied to the pasting unit 242. Hereinafter, the image of the bangs portion cut out as the compositing image will be referred to as the bangs image where appropriate.

The pasting unit 242 composites the bangs image cut out by the cutout unit 241 onto the captured image of the graffiti target. After performing the operation to cut out the bangs portion, the user selects the position where the bangs will be combined.

<Operation of the photo sticker creation device>
Here, the operation of the photo sticker creating device 1 that provides the photo sticker creating game will be described with reference to the flowchart in Fig. 8. The process in Fig. 8 starts when coins worth a predetermined amount, such as 400 yen, are inserted.

In step S1, the pre-selection processing unit 211 performs pre-selection processing. The pre-selection processing includes selecting the number of users and the game course.

In step S2, the photography processing unit 212 controls the photography unit 21 to perform photography processing. Specifically, the photography processing unit 212 displays the live view image captured by the camera 41 on the touch panel monitor 42. When the photography timing arrives, the photography processing unit 212 photographs the user in the photography space as a subject and generates a photographed image, for example, a close-up image. The close-up image is a still image in which the upper half of the user's body is visible.

In step S3, the editing processing unit 213 performs editing processing by controlling the configuration of the editing unit 51 corresponding to the editing space to which the user has moved after completing the shooting process.

Editing Process Next, the editing process performed in step S3 of FIG. 8 will be described with reference to the flowchart of FIG.

In step S11, the display control unit 225 of the editing processing unit 213 displays a sticker graffiti screen. The sticker graffiti screen displays the photographed image to be doodled, as well as various buttons, such as buttons to be operated when selecting a sticker design and buttons to be operated when selecting an image to be combined with the photographed image.

In step S12, the sticker image generation unit 222 performs image processing on the captured image in accordance with the user's operations to generate a sticker image.

In step S13, the sticker image generation unit 222 determines whether the sticker graffiti has ended. If it is determined in step S13 that the sticker graffiti has not ended, the process returns to step S11, and the sticker graffiti continues. For example, if a preset time limit has elapsed, the sticker graffiti ends.

In step S14, the display control unit 225 displays the mobile doodle screen on the tablet built-in monitor 61. The user uses the mobile doodle screen to doodle on the captured image.

Figure 10 shows an example of the mobile graffiti screen.

As shown in FIG. 10, the mobile doodle screen is basically configured with each display arranged symmetrically on either side of the central thumbnail display area 303. The left area is used by a user standing on the left side of the tablet-built-in monitor 61. The right area is used by a user standing on the right side of the tablet-built-in monitor 61.

Note that the dashed rectangular lines surrounding each area on the mobile doodling screen shown in Figure 10 are not actually displayed on the screen.

The thumbnail display area 303, located in the center of the mobile graffiti screen, is a display area for thumbnail images representing captured images. The user uses the thumbnail images displayed in the thumbnail display area 303 to select a captured image on which to doodle.

Thumbnail images representing all captured images are displayed in the thumbnail display area 303. In the example of Figure 10, thumbnail images of six close-up images are displayed. In the example of Figure 10, the user on the left has selected the thumbnail image in the top left, and the user on the right has selected the thumbnail image in the bottom right.

A doodle input area 301L is provided approximately in the center of the left-hand area. The doodle input area 301L is a display area for the captured image selected using the thumbnail image on which to doodle. The user can select an editing tool using the touch pen 141 and edit the captured image displayed in the doodle input area 301L.

To the right of the doodle input area 301L is a tool area 302L containing buttons such as a button to be pressed to select the eraser tool, a button to be pressed to reset the doodle, and a button to be pressed to return to the previous operation.

Tool area 302L displays buttons including selection button 302-1 and selection button 302-2. Selection button 302-1 is pressed to select the eraser tool, and selection button 302-2 is pressed to select the selection tool.

The eraser tool (normal eraser) is a tool used, for example, to erase a composite image that has already been combined. The selection tool is a tool used, for example, to select and move a composite image that has already been combined. Moving a composite image using the selection tool is performed by pressing the tip of the stylus 141 against the composite image and then moving it while holding it down, a so-called drag-and-drop operation.

Selection buttons 305 are displayed below the thumbnail display area 303 and are pressed to select various editing tools. The user can use the selection buttons 305 to switch between the editing tools displayed in the palette areas 304L and 304R. In the example of Figure 10, selection buttons are provided for the editing tools retouching, makeup, headbands, stamps, and pens. Stamps have been selected as the editing tool to be displayed in the palette area 304L, and retouching has been selected as the editing tool to be displayed in the palette area 304R.

A palette area 304L is provided below the doodle input area 301L. The palette area 304L is an area where buttons used to select various editing tools to be combined with the captured image displayed in the doodle input area 301L are displayed. In the example of Figure 10, the button used to select the stamp tool is displayed in the palette area 304L.

Above the palette area 304L, tabs are provided for selecting stamp tool categories. The user can use the tabs to switch between stamp tool categories displayed in the palette area 304L.

Displays similar to those described above are also provided on the right side of the mobile doodling screen. In the example of Figure 10, buttons used to select retouching tools are displayed in palette area 304R. When a retouching tool is selected, the bangs retouching function described above becomes available.

Figure 11 is an enlarged view of the palette area 304R in Figure 10.

Above the palette area 304R, tabs are provided for selecting categories. The user can use the tabs to switch between tools displayed in the palette area 304R. In the example of Figure 11, tabs for bangs, small face, eye size, and beautiful skin are provided, and the bangs tab is selected.

A cutout pen and a dedicated eraser are provided as tools for bangs retouching. Selection button 311 is a button used to select the thickness of the cutout pen. Selection button 312 is a button used to select the thickness of the dedicated eraser.

In the example of Figure 11, three different thicknesses are provided for the cut-out pen and the dedicated eraser. The cut-out pen is a tool used to select the bangs portion to be cut out as a bangs image. The dedicated eraser is a tool used to erase the bangs image pasted onto the captured image.

Below selection buttons 311 and 312, instructions on how to use bangs retouch are displayed. The instructions show how to use bangs retouch by tracing the area you want to cut out with a pen and placing the cut-out image in the desired position. After viewing the instructions on how to use bangs retouch, the user can retouch their bangs using the cut-out pen and dedicated eraser.

When bangs retouching is performed, image processing is applied to the captured image selected as the target for doodling, and the results are reflected in the display in doodling input area 301R. The captured image showing the user with the bangs modified is displayed in doodling input area 301R.

Returning to the explanation of Figure 9, in step S15, the portable image generation unit 224 determines whether bangs retouching has been selected.

If it is determined in step S15 that bangs retouching has been selected, in step S16 the bangs retouching processor 231 performs image processing in response to the user's operation using the bangs retouching tool. Details of the image processing performed in response to the operation using the bangs retouching tool will be described later.

In step S17, the portable image generation unit 224 performs image processing on the photographed image of the graffiti target in response to user operations using tools other than the bangs retouching tool, and generates a portable image. Further graffiti is applied using other tools to the photographed image of the graffiti target that has undergone the bangs retouching process.

If it is determined in step S15 that bangs retouching has not been selected, step S16 is skipped, and in step S17, image processing is performed in accordance with operations using tools other than bangs retouching.

In step S18, the portable image generation unit 224 determines whether the portable doodle has ended. If it is determined in step S18 that the portable doodle has not ended, the process returns to step S14, and the portable doodle continues. For example, if a preset time limit has elapsed, the portable doodle ends.

If it is determined in step S18 that the mobile graffiti has been completed, in step S19, the print processing unit 214 begins printing the sticker image generated by the sticker graffiti. The timing for starting printing can be any time after the sticker graffiti has been completed.

In step S20, the transmission processing unit 215 transmits the portable image to the image management server. When transmission of the portable image is complete, the process returns to step S3 in Figure 8, and the series of processes for the photo sticker creation game comes to an end.

<Basic bangs retouching functions>
FIG. 12 is a diagram showing an example of an operation using a clipping pen for bangs retouching.

Figure 12 shows only the head and surrounding areas of a person appearing in a photographed image on which graffiti is to be added. Bangs retouching will be explained using an image different from the image on the graffiti screen in Figure 10. This also applies to the other figures described below. The bangs retouching functions are operated using the touch pen 141, just like the other functions available on the graffiti screen.

As shown in the upper part of Figure 12, the operation of using the cut-out pen for bangs retouching is performed, for example, by pressing the tip of the stylus 141 at position p1 and then, while maintaining that position, moving it to position p2 as indicated by arrow #1 in the lower part. In other words, the cut-out operation using the cut-out pen is performed by tracing the bangs portion of the person appearing in the photographed image to be doodled with the stylus 141.

When an operation of tracing the bangs is performed, the range along the trajectory from position p1 to position p2, which corresponds to the thickness of the selected clipping pen, is selected as the clipping range. For example, of the three thicknesses (thick, medium, and thin), the middle thickness is selected as the default thickness.

The selectable range, which is the range that can be selected as the cut-out range, is limited to, for example, the range from the forehead to the top of the head. In the example of Figure 12, the range above the dashed dotted line L in the person area is set as the selectable range.

When the tip of the stylus 141 is removed from the screen at position p2, the image of the bangs portion that appears in the range selected as the crop area is cropped as bangs image P1 and displayed superimposed on the crop area, as shown in Figure 13.

In the example of Figure 13, bangs image P1 is displayed superimposed on the trajectory from position p1 to position p2, which is the cut-out range, but it may also be displayed at a position offset from the cut-out range. Note that the outline of bangs image P1 is shown with a dashed line, but the line representing this outline may or may not be displayed.

Figure 14 is an enlarged view of bangs image P1.

As shown in Figure 14, bangs image P1 is an image of the bangs portion captured within the cropping range selected by the user. Bangs image P1 is used as the compositing image.

A cancel button 321 is displayed in the upper left of the bangs image P1, and a transform button 322 is displayed in the lower right. For example, the cancel button 321 and the transform button 322 are displayed at the top left and bottom right vertices of a rectangular area surrounding the bangs image P1. The cancel button 321 is a button that is operated to stop compositing using the bangs image P1. The transform button 322 is a button used to enlarge, reduce, rotate, etc. the bangs image P1.

As the user moves the stylus 141 while the transformation button 322 is pressed, the bangs image P1 is displayed as enlarged, reduced, or rotated, as shown in FIG. 15. FIG. 15A shows the bangs image P1 in a reduced state, and FIG. 15B shows the bangs image P1 in a rotated state.

The user can adjust the size and orientation of the bangs image P1 by scaling and rotating the cropped bangs image. In addition to scaling and rotation, other adjustments such as left-right flipping may also be possible.

Figure 16 shows an example of selecting the pasting position for bangs image P1.

After selecting the cut-out range as described above, the user can select a position on the bangs image P1 with the touch pen 141 and then move it to any position in that state to select the paste position. In the example of Figure 16, position p11 on the bangs image P1 is selected as shown in the top row, and the touch pen 141 is moved to position p12 to the right as shown in the bottom row. The bangs image P1 moves and is displayed in accordance with the movement of the touch pen 141.

When the user removes the tip of the stylus 141 from the screen at position p12, bangs image P1 is superimposed (pasted) at position p12. The user can also shift the superimposed position of bangs image P1 by pressing the stylus 141 again on bangs image P1 and then moving it to another position while still holding it down.

The bangs image P1 may be composited at any position on the photographed image of the graffiti target, or the composite position may be limited to positions within the same range as the selectable range.

If the photographed image on which the graffiti is to be created contains multiple people, the composition position may be limited to a position within the range of the person from whom the bangs image was extracted.

In this case, for example, if two people, user A and user B, appear in the captured image to be graffitied, and user A's bangs are cut out, the bangs image can only be composited onto the position of user A, and cannot be composited onto the position of user B. The areas containing user A and user B are identified by analyzing the captured image.

Fangs look different from person to person due to differences in hair color and texture, as well as the way the light hits them during photography. If it were possible to composite bangs from different people, the bangs could end up looking unnatural, but this can be prevented.

Figure 17 shows an example of compositing bangs image P1.

In the example of Figure 17, bangs image P1 is composited at position p21, which is approximately the center of the forehead. By composited bangs image P1 at position p21, the person's bangs appear to have more hair in the captured image.

In this way, the user can select the cutout area by tracing the bangs of a person in the captured image in the direction of the bangs' flow. Because the selection can be made by tracing, the user can cut out the fine details of the bangs.

By using the bangs retouching function, users can fine-tune the appearance of their own bangs and those of their friends in photos.

In addition, if a captured image contains multiple people, a selectable range is set for each person's area, allowing each user to adjust the appearance of their own bangs using a bangs image cut out from the selectable range set for the area in which they appear.

<Bangs retouching details>
Cutting out the combined bangs image FIG. 18 is a diagram showing another example of selection of the cut-out range.

By using the cut-out pen to select the area containing the bangs image P1 that has already been synthesized as the cut-out area, the user can cut out the image containing the bangs image P1 as a new bangs image.

In the example at the top of Figure 18, a path starting from position p31 to position p32, which passes over bangs image P1 and is shown as a continuous, mountain-shaped solid line, is selected with the touch pen 141. In this case, as shown in the bottom of Figure 18, bangs image P11 is cut out from the range along the path from position p31 to position p32. Bangs image P11 includes the portion of the bangs that appears in the captured image and the portion of the bangs image P1 that has already been synthesized.

In the example of Figure 18, an area including the entire bangs image P1 has been cut out as bangs image P11, but it is also possible to cut out a portion of bangs image P1 as a new bangs image. In this way, by using the cut-out pen, it is possible to cut out a new bangs image that includes part or all of the previously synthesized bangs image.

Dedicated Eraser FIG. 19 shows an example of an operation using a dedicated eraser.

As mentioned above, the dedicated eraser is an erasing tool used to erase a bangs image superimposed on a captured image. The top part of Figure 19 shows a state in which bangs image P1 has been superimposed at position p21 (Figure 17), approximately in the center of the forehead.

In this state, if the user selects the dedicated eraser and moves the stylus 141 from position p41 to position p42, the area of the synthesized bangs image P1 along the path of the stylus 141 will disappear, as shown in the lower part of Figure 19.

In the lower part of Figure 19, the range along the trajectory of the stylus 14, corresponding to the selected thickness, is shown as dashed line range A1. Even within range A1 along the trajectory of the stylus 141 input while the dedicated eraser is selected, only the portion of the synthesized bangs image P1 is erased by the dedicated eraser; the original bangs portion that appears in the captured image is not erased. In the erased portion of bangs image P1, the forehead and other parts that appear in the captured image are visible.

In this way, the dedicated eraser is a dedicated tool used to erase the composited bangs image P1. By using the dedicated eraser, the user can adjust the shape of the composited bangs portion.

The portion of the bangs image P1 included in range A1 does not disappear in the same way regardless of position, but the degree of disappearance may change as it moves away from the position on the path of the stylus 141. The position selected using the dedicated eraser is set as the center, and the blur rate for each position within the range of the dedicated eraser's thickness changes depending on the distance from the center. The further away from the center, the higher the blur rate is set.

For example, of the portion of the bangs image P1 included in range A1, the bangs image disappears completely at the position on the trajectory of the touch pen 141, and as it approaches the outside of range A1, the bangs image disappears in a faint manner, making it possible to blend the boundary between the portion erased with the dedicated eraser and the surrounding bangs portion.

Difference between dedicated eraser and normal eraser The normal eraser, which is selected when the selection button 302-2 (Fig. 10) is pressed, is an erasing tool used to erase images to be combined other than the bangs image, such as pen images and stamp images. The pen image is an image to be combined that is input using the pen tool. The pen image is combined along the trajectory of the touch pen 141. The stamp image is an image to be combined that is input using the stamp tool. The stamp image is combined at the position specified by the touch pen 141.

Figure 20 shows an example of a captured image with a stamp image superimposed on it.

The top part of Figure 20 shows a state in which a heart-shaped stamp image P21 has been superimposed on the bangs image P1. Stamp image P21 is an image for compositing selected from the multiple stamp images displayed when the stamp tool is selected, as shown in the palette area 304L in Figure 10. Stamp image P21 is composited onto a layer above the layer of bangs image P1.

In this state, as shown in the lower part of Figure 20, we will explain what happens when the user moves the stylus 141 from position p41 to position p42, performing an operation on range A1 corresponding to the eraser thickness.

If the tool selected by the user is a regular eraser, the stamp image P21 will disappear as shown in Figure 21. In the example of Figure 21, the part of stamp image P21 that is included in range A1 operated with the regular eraser will disappear, and the bangs image P1 will remain displayed (will not disappear). The same display will occur if a similar operation is performed on a compositing image other than a bangs image, such as a pen image.

On the other hand, if the tool selected by the user is the dedicated eraser, the bangs image P11 will disappear as shown in Figure 22. In the example of Figure 22, the part of the bangs image P1 that is included in the range A1 selected using the dedicated eraser will disappear, and the stamp image P21 will remain displayed (will not disappear).

In this way, when a user composites a pen image or stamp image over a bangs image, they use the regular eraser to erase the composite image, such as the pen image or stamp image, and the dedicated eraser to erase the bangs image. By providing a dedicated eraser separate from the regular eraser, users can fine-tune the appearance of each image, even when a stamp image or other image is composited over a bangs image.

Similarly, if you select the cut-out pen instead of the dedicated eraser and trace a position on an already-composite pen image or stamp image, the part that will be affected will be the part of the bangs that is included in the range selected by the tracing operation. In other words, if you select the cut-out pen and trace a position on an already-composite stamp image, the part of the stamp image that is included in the range selected by the tracing operation will not be cut out (copied), and the part of the bangs that is included in the same range will be cut out as the bangs image.

As described above, the bangs image can be cut out within a selectable range from the forehead to the top of the head. The selectable range is set based on, for example, the features of the person extracted by analyzing the captured image.

If a cutout range is selected outside the selectable range, such as near the neck of a person in the captured image, the user may be prompted to select a cutout range within the selectable range.

Figure 23 shows an example message regarding the cutout range.

As shown in Figure 23, a message is displayed instructing the user to select the cropping area within a selectable range, such as the area from the forehead to the top of the head. The message shown in Figure 23 is displayed as a pop-up by the display control unit 225, superimposed on the captured image of the graffiti target, for example.

This allows the user to confirm that they are performing the crop operation at a position outside the selectable range. When bangs retouch is selected, information indicating the selectable range may be displayed superimposed on the captured image of the doodle target.

<Modification>
Regarding the selection tool, the selection tool selected when the selection button 302-2 is pressed can also be used to move the composited bangs image (change the composite position). Moving the bangs image using the selection tool is effective when, for example, the bangs retouching function is selected.

When the bangs retouch function is not selected and buttons such as the bangs retouch function are not displayed in palette areas 304L and 304R, the images that can be moved using the selection tool are composite images such as pen images and stamp images. When moving a bangs image, the user uses the selection tool with the bangs retouch function selected, and when moving a stamp image, for example, the user uses the selection tool with the bangs retouch function not selected.

Example of cutout range If the bangs image includes skin on the forehead other than hair, image processing may be performed according to the content of the composite image when combining them. For example, the brightness of each pixel in the bangs image is compared with the brightness of the corresponding pixel in the composite image, and the combination is performed with priority given to the darker pixels (the darker pixels are displayed).

This means that if bangs are to be composited into the skin portion of the bangs image, the composite will be performed so that the bangs are visible.

If parts other than hair, such as skin on the forehead, are included in the cropping range, the parts other than hair may be removed from the bangs image. Removal of parts other than hair is performed, for example, based on a hair mask image generated based on the captured image. A hair mask image is an image used to identify and distinguish areas containing hair from other areas.

The process of cutting out the portion selected by the tracing operation and combining it in a desired position may also be performed on portions other than the bangs. For example, it may be possible to cut out the hair near the shoulders and combine it in a desired position.

In the above example, the cut-out area is selected by tracing using the cut-out pen, but a rectangular area of a specified size at the position selected by the user may also be selected as the cut-out area.

Other Although three different thicknesses are provided for the cut-out pen and the dedicated eraser, the user may be able to adjust each thickness using a GUI such as a slide bar.

You can also display a captured image after retouching using the bangs retouching function and an unretouched image side by side for comparison.

It may also be possible to change the hair color, including the bangs created using the bangs retouching function. Furthermore, image processing to enhance the shine of the hair may be applied to the entire hair, including the bangs.

It may also be possible to composite a bangs image cut out from one captured image with another captured image.

Regarding the program: The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program that constitutes the software is installed on a computer that is built into dedicated hardware, or on a general-purpose personal computer.

The program to be installed is provided by being recorded on removable media such as an optical disc (CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc), etc.) or semiconductor memory. It may also be provided via wired or wireless transmission media such as a local area network, the Internet, or digital broadcasting. The program can be pre-installed in ROM or memory.

The program executed by the computer may be a program in which processing is performed chronologically in the order described in this specification, or a program in which processing is performed in parallel or at the required timing, such as when called. Note that the effects described in this specification are merely examples and are not limiting, and other effects may also be present.

The embodiments of this technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of this technology.

1 Photo sticker creation device, 11 Pre-operation unit, 21 Photography unit, 51 Editing unit, 201 Control unit, 211 Pre-selection processing unit, 212 Photography processing unit, 213 Editing processing unit, 221 Image analysis unit, 224 Portable image generation unit, 225 Display control unit, 231 Bangs retouching processing unit, 241 Cutting unit, 242 Pasting unit

Claims (11)

In response to a selection operation by a user in an area of the bangs of a person appearing as a subject in a photographed image, an image of the bangs in a range along a trajectory from a start position to an end position of the selection operation is cut out as an image to be synthesized;
The image for composition is superimposed on the bangs appearing at the position on the photographed image selected by the user, and the image for composition is composited and displayed.
A program that causes a computer to execute a process. Displaying the extracted composite image in accordance with the user's operation of selecting a position on the screen;
2. The program according to claim 1, wherein the image to be combined is displayed at a position where the user has performed an operation to move the image away from the screen . 3. The program according to claim 1 , wherein the position of the composite image after the composite process is changed in response to an operation by the user . The cutting out of the image to be combined is performed by cutting out an image of the bangs within a predetermined width range selected by the user along a path from the start position to the end position of the selection operation.
4. The program according to claim 1. When the selection operation is performed on a range including the top of the already-combined image for synthesis , the image for synthesis is cut out by cutting out the bangs portion and the image for synthesis portion included in the range along the path of the selection operation.
The program according to claim 4. The program according to claim 4 or 5, wherein when the selection operation targeting an area including the top of the composite image that has already been composited is performed with an erase tool selected, the display of the portion of the composite image that is included in the area along the trajectory of the selection operation is erased. In response to an operation by the user, another image to be synthesized selected from a plurality of types is synthesized on the synthesized image to be synthesized and displayed;
The program according to claim 6, wherein when the selection operation targeting an area including the other composite image displayed on top of the composite image is performed with the erase tool selected, the display of the other composite image remains as is but the display of the other composite image is erased. The program according to claim 1 , wherein when the selection operation is performed at a position outside an area showing bangs , a guide is displayed to indicate that the selection operation should be performed within an area showing bangs . The program according to claim 1 , wherein the size and orientation of the image to be combined are changed in accordance with an operation by the user before selecting a combination position . The image processing device
In response to a selection operation by a user in an area of the bangs of a person appearing as a subject in a photographed image, an image of the bangs in a range along a trajectory from a start position to an end position of the selection operation is cut out as an image to be synthesized;
the image for composition is superimposed on the bangs appearing at a position on the photographed image selected by the user, and the composition image is then composited and displayed. a cutout unit that, in response to a selection operation by a user in an area of the bangs of a person appearing as a subject in a photographed image, cuts out an image of the bangs in a range along a trajectory from a start position to an end position of the selection operation as an image to be synthesized;
a display control unit that displays the image to be combined by superimposing the image to be combined on the bangs that appear at the position on the photographed image selected by the user;
An image processing device comprising :