JP7828393B2 - Program, method, and system - Google Patents

Description

The present invention relates to a program, a method, and a system.

A technology called AR (Augmented Reality) adds information such as virtual objects to real space, enhancing the real world. Specifically, this is an image processing technology that synthesizes an image of a virtual object (a non-real object such as a character) onto an image of real space captured by a camera, and displays the image on a device's display. Furthermore, with this type of augmented reality, users can operate the device to display their preferred virtual object (e.g., a character object) as if it were actually in real space (see, for example, Patent Document 1).

Japanese Patent Application Laid-Open No. 2018-200519

By the way, there are cases where the aforementioned augmented reality is used to take photos outdoors. Furthermore, when taking photos using such augmented reality, users may adjust the display mode by repeatedly rearranging virtual objects to create an arrangement image (composition) that satisfies them. However, depending on the shooting location, it may be difficult to continue taking photos due to concerns about being seen by others, and users may end up having to end the shoot without completing an image that satisfies them.

The present invention was conceived in light of this situation, and provides a program, method, and system that enables the re-editing of augmented reality images.

(1) A program according to an aspect of the present invention includes a computer (e.g., a user terminal 300) including an information acquisition unit (e.g., an information acquisition unit 350, a camera equipped with a depth sensor, etc.) that captures an image and acquires depth information (e.g., depth information) that enables the depth position of each real object that may be included in the image to be identified.
a means (e.g., step S102 in FIG. 10, an acquired information analysis unit 362, an augmented reality data generation unit 363, etc.) for identifying a real object (e.g., a person or object that actually exists, such as the chair R1 in FIG. 4) that may be included in the captured image based on depth information that may be identified from the information acquired by the information acquisition unit (e.g., a captured image, depth information, etc.) and for identifying a depth position of the real object in a virtual space (e.g., coordinates in the virtual space determined based on the depth information of the chair R1 in FIG. 4);
a means (e.g., steps S102 to S107 of FIG. 10 , an augmented reality data generating unit 363, etc.) for generating a virtual space (e.g., virtual space data in which the real object RO1 and the character object C are arranged, such as FIG. 5(D)) in which a real object corresponding to the real object identified in the virtual space is arranged at a depth position of the real object (e.g., generating a mask polygon for the real object RO1 so that it matches the real position and shape of the chair R1) and a specific object to be arranged in response to an operation from a user (e.g., arranging an unreal object such as a character object C or a sticker DO by a tap operation from the user);
a means (e.g., steps S103 to S107 of FIG. 10, an augmented reality data generating unit 363, a display control unit 364, an output unit 340, etc.) for displaying a composite image in which a portion of the specific object that overlaps with a real object located in front of the specific object is hidden and a portion that does not overlap can be superimposed on the captured image (e.g., an image in which a character object C is displayed in a state where it is occluded by a real object RO1, as shown in FIG. 6B), by combining the captured image with an object that is placed in the virtual space generated by the generating means;
a means (e.g., steps S108 to S109 in FIG. 10, the re-edited data management unit 365, etc.) for storing a captured image when a user performs a capture operation (e.g., a captured image that can be acquired by the information acquisition unit 350 when an operation is performed on the capture button S displayed on the capture screen in FIG. 4), and virtual space identification information for identifying the virtual space generated by the generating means (e.g., depth information, information on masks corresponding to real objects, etc.);
The virtual space data updating unit 363 functions as a unit (e.g., steps S201 to S209 in FIG. 11 , the augmented reality data generating unit 363, etc.) for updating specific objects (e.g., character object C, decorative object DO, etc.) to be placed in the virtual space based on the virtual space identification information (e.g., information on the depth of an object stored as re-editing data) stored by the storing unit in accordance with a change operation from the user (e.g., as exemplified in FIG. 6 , a change to the display mode in accordance with a change operation such as a change to the size, expression, motion, or type of character object C, or a change to a decorative object),
After the change operation, the displaying means displays a composite image obtained by combining the captured image stored by the storing means with the object placed in the virtual space being updated by the updating means (e.g., the augmented reality image after the re-editing operation in Figures 7 and 6 (B) to (D), steps S205 to S209 in Figure 11, etc.).

With this configuration, even after a composite image has been displayed and a photograph has been taken, it is possible to change the specific object afterward and regenerate an image composited with the captured image. This allows the user to change the specific object in a relaxed state, improving user convenience.

(2) In (1) above, the behavior in response to the change operation includes at least one of a change in the placement position of the specific object in the virtual space (e.g., a change in coordinates in the virtual space based on zooming in/out or moving the position left/right), a change to a different type of specific object (e.g., a change to a different character or a different decorative object), and a change in the display behavior of the specific object (e.g., the motion, facial expression, pose, or costume of character object C).

With this configuration, even after the composite image has been displayed and the photographing operation has been performed, the specific object can be changed after the fact in response to a change operation from the user. This allows the user to change the specific object in a relaxed state, improving user convenience.

(3) In the above (1), the specific object includes a character object with a predetermined motion (for example, a character object C that performs an animation action such as dancing or jumping),
When the character object is placed as the specific object, the display means can display the character object by moving it in a predetermined motion (for example, character object C is displayed in a manner involving dancing movements).

With this configuration, even after the composite image is displayed and a photo is taken, the character object can be moved later and the character object's movements at a specified timing can be saved, thereby expanding the range of expression and increasing the user's interest.

(4) In the above (1), the specific object includes a character object and an animation-related decorative object (for example, an object composed of multiple moving particles, a particle that moves a fluid, or the like, particle P in FIG. 9, etc.),
When the character object and the decorative object are arranged as the specific object, the displaying means is capable of displaying the character object and also of displaying the decorative object by moving it (for example, character object C is displayed and particles are displayed with movement, etc.).

With this configuration, even after the composite image is displayed and a photo is taken, the decorative object can be activated later and the appearance of the decorative object at a specified timing can be saved, thereby expanding the range of expression and increasing the user's interest.

(5) In the above (3) or (4), the computer is
It functions as a means for changing the movement speed of a specific object that is operated and displayed by the display means in response to an operation from the user (for example, changing the movement of a character object to slow motion by operating the speed change icon T4 in Figure 6 (D)).

This configuration makes it easier for users to specify the timing of object movements at their desired timing, broadening the scope of expression and increasing user interest.

A method according to an aspect of the present invention is a method for controlling a computer (e.g., a user terminal 300, etc.) including an information acquisition unit (e.g., an information acquisition unit 350, a camera equipped with a depth sensor, etc.) that captures an image and acquires depth information (e.g., depth information, etc.) that enables identification of a depth position of each real object that may be included in the image,
A step of identifying a real object (e.g., a person/object that actually exists, such as the chair R1 in FIG. 4 ) that may be included in the captured image based on depth information that may be identified from the information acquired by the information acquisition unit (e.g., the captured image, depth information, etc.) and identifying the depth position of the real object in the virtual space (e.g., coordinates in the virtual space determined based on the depth information of the chair R1 in FIG. 4 ) (e.g., step S102 in FIG. 10 , the acquired information analysis unit 362, the augmented reality data generation unit 363, etc.);
a step of generating a virtual space (e.g., virtual space data in which the real object RO1 and the character object C are arranged, such as in FIG. 5(D)) in which a real object corresponding to the real object identified in the virtual space is arranged at a depth position of the real object (e.g., generating a mask polygon for the real object RO1 so that it matches the real position and shape of the chair R1) and a specific object to be arranged in response to an operation from the user (e.g., arranging unreal objects such as the character object C and the sticker DO by a tap operation from the user); and
a step of displaying a composite image (e.g., steps S103 to S107 in FIG. 10, the augmented reality data generation unit 363, the display control unit 364, the output unit 340, etc.) in which a portion of the specific object that overlaps with a real object located in front of the specific object is hidden and a non-overlapping portion of the specific object can be superimposed on the captured image; and
a step of storing a captured image when a user performs a shooting operation (for example, a captured image that can be acquired by the information acquisition unit 350 when the shooting button S displayed on the shooting screen in FIG. 4 is operated), and virtual space identification information for identifying the virtual space generated in the generating step (for example, depth information, information on masks corresponding to real objects, etc.) (for example, steps S108 to S109 in FIG. 10, the re-edited data management unit 365, etc.);
a step of updating the virtual space (e.g., steps S201 to S209 in FIG. 11 , the augmented reality data generation unit 363, etc.) in which specific objects (e.g., character object C, decorative object DO, etc.) to be placed in the virtual space based on the virtual space identification information stored in the storing step (e.g., depth information of an object stored as re-editing data, etc.) are changed in accordance with a change operation from the user (e.g., as exemplified in FIG. 6 , a change to a display mode in accordance with a change operation such as a change to the size, expression, motion, or type of character object C, or a change to a decorative object, etc.);
The displaying step displays a composite image obtained by combining the captured image stored in the storing step with the object placed in the virtual space updated in the updating step after the change operation (e.g., the augmented reality image after the re-editing operation in Figures 7 and 6 (B) to (D), steps S205 to S209 in Figure 11, etc.).

With this configuration, even after a composite image has been displayed and a photograph has been taken, it is possible to change the specific object afterward and regenerate an image composited with the captured image. This allows the user to change the specific object in a relaxed state, improving user convenience.

(7) A system according to an aspect of the present invention includes a computer (e.g., a user terminal 300) that captures an image and acquires depth information (e.g., depth information) that enables identification of a depth position of each real object that may be included in the image, and includes an information acquisition unit (e.g., an information acquisition unit 350, a camera equipped with a depth sensor, etc.),
a means (e.g., step S102 in FIG. 10, an acquired information analysis unit 362, an augmented reality data generation unit 363, etc.) for identifying a real object (e.g., a person or object that actually exists, such as the chair R1 in FIG. 4) that may be included in the captured image based on depth information that may be identified from the information acquired by the information acquisition unit (e.g., a captured image, depth information, etc.) and for identifying a depth position of the real object in a virtual space (e.g., coordinates in the virtual space determined based on the depth information of the chair R1 in FIG. 4);
a means for generating a virtual space (e.g., virtual space data in which the real object RO1 and the character object C are arranged, such as in FIG. 5(D)) in which a real object corresponding to the real object identified in the virtual space is arranged at a depth position of the real object (e.g., generating a mask polygon for the real object RO1 so as to match the real position and shape of the chair R1) and a specific object to be arranged in response to a user operation (e.g., arranging an unreal object such as a character object C or a sticker DO by a tap operation from the user) is arranged (e.g., steps S102 to S107 of FIG. 10, an augmented reality data generating unit 363, etc.);
a means (e.g., steps S103 to S107 of FIG. 10, an augmented reality data generating unit 363, a display control unit 364, an output unit 340, etc.) for displaying a composite image in which a portion of the specific object that overlaps with a real object located in front of the specific object is hidden and a portion that does not overlap can be superimposed on the captured image (e.g., an image in which a character object C is displayed in a state where it is occluded by a real object RO1, as shown in FIG. 6B), by combining the captured image with an object that is placed in the virtual space generated by the generating means;
a means (e.g., steps S108 to S109 in FIG. 10, the re-edited data management unit 365, etc.) for storing a captured image when a user performs a capture operation (e.g., a captured image that can be acquired by the information acquisition unit 350 when an operation is performed on the capture button S displayed on the capture screen in FIG. 4), and virtual space identification information for identifying the virtual space generated by the generating means (e.g., depth information, information on masks corresponding to real objects, etc.);
a means for updating the virtual space (e.g., steps S201 to S209 in FIG. 11 , an augmented reality data generation unit 363, etc.) so that specific objects (e.g., character object C, decorative object DO, etc.) placed in the virtual space based on the virtual space identification information (e.g., information on the depth of an object stored as re-editing data) stored by the storing means are displayed in a manner corresponding to a change operation from the user (e.g., as illustrated in FIG. 6 , a change to the display manner corresponding to a change operation such as a change to the size, expression, motion, or type of character object C, or a change to a decorative object, etc.);
After the change operation, the displaying means displays a composite image obtained by combining the captured image stored by the storing means with the object placed in the virtual space being updated by the updating means (e.g., the augmented reality image after the re-editing operation in Figures 7 and 6 (B) to (D), steps S205 to S209 in Figure 11, etc.).

With this configuration, even after a composite image has been displayed and a photograph has been taken, it is possible to change the specific object afterward and regenerate an image composited with the captured image. This allows the user to change the specific object in a relaxed state, improving user convenience.

FIG. 1 illustrates an example of a hardware configuration of a communication system. FIG. 2 is a diagram illustrating the configuration of a distribution server. FIG. 2 is a diagram illustrating the configuration of a user terminal. 10 is a diagram illustrating an example of a screen display mode on a user terminal during AR photographing processing. 10 is an example of data for generating a composite image in AR photographing processing. 10 is an example of an editing screen on a user terminal for editing a character object during AR re-photographing processing. 10 is an example of an editing screen on a user terminal during AR re-photographing processing. 10 is an example of an editing screen on a user terminal during AR re-photographing processing. 10 is an example of an editing screen on a user terminal regarding the arrangement mode of objects during AR re-capture processing. 10 is a flowchart illustrating an example of an AR photographing process. 10 is a flowchart illustrating an example of an AR re-photographing process.

Embodiments of a communication system according to the present invention will be described below with reference to the drawings. Note that the present invention is not limited to the examples below, but is defined by the claims, and all modifications within the meaning and scope of the claims are intended to be included in the present invention. In the following description, identical elements in the drawings will be given the same reference numerals, and redundant explanations will not be repeated.

This embodiment relates to a program that allows a user to experience augmented reality by synthesizing and displaying virtual three-dimensional computer graphics (3DCG) on captured images of real space. For example, an image of a character object of the user's choice placed in real space can be displayed on a smartphone operated by the user. In addition, information regarding the character's height is associated with the character object in this embodiment, and the character object is placed so that the real space matches the set height based on depth information acquired on the user's device along with capturing an image of the real space. This allows the user to feel as if the character is actually present in real space.

However, when placing a character object or the like in augmented reality, the desired position, or angle of view, may not always be achieved immediately. For example, if the subject is a model that exists in the real world, the photographer (e.g., a cameraman) will repeatedly capture images until they are satisfied with the composition. In other words, multiple captured image data will be generated. When capturing images multiple times, the photographer may, for example, give instructions to the model about their position and pose, and make fine adjustments until they are satisfied with the image (composition). If the subject is a CG character, the user adjusts the character's position and angle. At this time, the subject is displayed on the user's screen, but is not visible to people around the user who is taking the photo. Therefore, if the user continues capturing images using augmented reality for a long period of time, there is a risk that those around them will become suspicious, and the user may end up having no choice but to end the capture even if they are not satisfied with the result.

Therefore, in this embodiment, the program that executes the process of allowing the user to experience augmented reality allows for subsequent re-editing, even after an image has been captured using augmented reality. Furthermore, it is equipped with a decoration function for image data captured using augmented reality (for example, an image before it is rasterized). While conventional applications capable of executing decoration functions (decoration functions/image editing applications) allowed for the size of decorations such as stickers (stamps) to be changed based on two-dimensional (x, y axis) information, this embodiment enables image editing using three-dimensional (x, y, z axis) concepts. In other words, it is now possible to expand the range of expression when decorating images compared to the past by using depth information included in information acquired along with capturing an image of real space and depth information of virtual objects used in image editing.

Figure 1 is a diagram showing an example hardware configuration of communication system 1. Communication system 1 includes a distribution server 100 and multiple user terminals 300a, 300b, 300c, etc. The multiple user terminals 300a, 300b, 300c, etc. are terminals owned by multiple users, and are hereinafter collectively referred to as user terminals 300.

The distribution server 100 and the user terminal 300 can each communicate via network 2, allowing them to send and receive information (data) in both directions. Network 2 is, for example, the Internet, and is composed of access networks such as LANs (Local Area Networks), WANs (Wide Area Networks), mobile communication networks (e.g., 5G, wireless networks, etc.), wired telephone networks, FTTH (Fiber To The Home), and CATV (Cable Television) networks.

The distribution server 100 is, for example, a computer such as a workstation or personal computer with communication capabilities. The distribution server 100 manages programs and content data for providing (allowing users to experience) XR (Extended Reality), which blends the real world and the virtual world. For example, the distribution server 100 manages programs that enable the user terminal to execute the augmented reality image display processing of this embodiment, and virtual spaces, which are virtual worlds constructed on a computer (including, for example, virtual spaces for executing augmented reality image display processing, and metaverse spaces), and provides these to the user terminal 300 via the network 2.

Virtual spaces are managed and configured for each type of content provided. In addition to augmented reality, content that can be provided includes, but is not limited to, content that allows users to view or experience games, live performances, events, and programs, as well as content that allows users to interact and communicate with each other using chat and emotes. For example, if the virtual space is a so-called metaverse space, users can access the distribution server 100 using the user terminal 300 and select the desired content, thereby seamlessly participating in (navigating) that content and viewing or experiencing the virtual space corresponding to that content.

Depending on the type of content, virtual space can include three-dimensional space (space constructed based on three-dimensional data) generated by CG (Computer Graphics) or two-dimensional space (space constructed based on two-dimensional data). Virtual spaces also contain virtual characters (avatar objects, character objects), objects representing backgrounds and virtual objects according to the type of content, and menu objects that can be selected by the user.

Content data includes, for example, information for identifying information about objects placed within the virtual space (e.g., object type, placement position, orientation, posture, appearance, etc.), information about the user characters of each user participating in the virtual space (e.g., user character type, placement position, orientation, posture, appearance, motion data, audio data, etc.), objects representing backgrounds and virtual objects according to the type of content, and information for identifying menu objects selected by the user.

Note that the distribution server 100 in this embodiment may be realized by a single computer, or may be realized by multiple computers (e.g., multiple servers).

The user terminal 300 is a computer capable of displaying augmented reality images, such as a personal computer, tablet terminal, smartphone, or other computer with operation input and communication functions.

The user terminal 300 can communicate with the distribution server 100 and execute programs received in response to operations on the terminal. The user terminal 300 receives content data for content selected by the user from the distribution server 100. Based on the received content data, the user terminal 300 constructs a virtual space for the content selected by the user from the virtual spaces constructed on the distribution server 100 in its own storage area, and displays images and outputs sounds from the virtual space. This allows the user to view and experience the content's virtual space via the user terminal 300. For example, the user terminal 300 can provide the user with augmented reality by combining captured images (image data) acquired by an information acquisition unit (such as a camera) with CG, etc., based on a program for performing augmented reality image display processing received from the distribution server 100. Hereinafter, an application that provides augmented reality to the user using a program for performing augmented reality image display processing is also referred to as AR content. An application that provides augmented reality to the user is, for example, an application that can be installed on the user terminal 300.

<Configuration of distribution server>
Next, the configuration of distribution server 100 will be described. As shown in Fig. 2, distribution server 100 includes a communication unit 110 that communicates with other computers, a storage unit 120 that stores various data, and a control unit 130 that controls the entire computer. Communication unit 110, storage unit 120, and control unit 130 are interconnected by a bus line.

The communication unit 110 is a communication interface equipped with a NIC (Network Interface Card controller) for wired or wireless communication. The communication unit 110 communicates with other computers via the network 2.

The storage unit 120 is composed of RAM (Random Access Memory), ROM (Read Only Memory), flash memory, HDD (Hard Disk Drive), etc. The storage unit 120 stores programs for executing various control processes (e.g., AR content that enables augmented reality photography and image display, and programs for managing and providing content using virtual spaces such as the Metaverse), various data, etc. The various data stored in the storage unit 120 includes object data 121 that stores object data that can be displayed in AR content provided to the user. Also included, although not shown, are game programs that include information for identifying images of virtual spaces provided for each type of content (e.g., game content), and user information related to the user. User information related to the user includes, for example, an ID for identifying the user and a username.

Object data 121 includes and manages various objects that are placed in the virtual space of the AR content provided to the user. The object data table stored in the storage unit 120 stores information such as the shape and pattern of various objects, and the amount of cash or virtual currency required to generate the various objects.

Objects include, for example, two-dimensional and three-dimensional CG data such as character objects and decorative objects. This CG data can be displayed in an augmented reality image through user manipulation. Hereinafter, in this embodiment, virtual objects that differ from real-space objects, such as character objects and decorative objects, are also referred to as unreal objects. Unreal objects are objects that differ from real-space objects in the captured image, and include various virtual objects such as objects that do not exist in real space and objects that mimic real-space objects. Objects that correspond to real-space objects in the captured image (e.g., objects and polygon masks that correspond to objects or surfaces contained in a captured image of real space) are also referred to as real objects. Furthermore, the display mode of a character object can be changed in response to user manipulation. For example, the character's facial expression, pose, costume, and motion can be changed. A height corresponding to the character is predetermined for each character object.

Decorative objects include, for example, decorations such as letters and icons (stickers, stamps, etc.), frames, and particles. The display mode of decorative objects can be changed in response to user operations. For example, the size, color, and pattern of decorative objects can be changed. Particles are visual effects that accompany animation, and are objects that move objects made up of multiple particles such as snow, flowers, rain, and light, or fluids such as water and fire, making it possible to create effects such as snow falling in a specified area or a specified area bursting into flames.

The control unit 130 is composed of a CPU (Central Processing Unit) and other components. The control unit 130 controls the overall operation of the distribution server 100 by executing programs stored in the storage unit 120.

The functional configuration of the control unit 130 will be described below. The control unit 130 functions as at least a data transmission/reception unit 131, a content management unit, and a user management unit (not shown).

The data transmission/reception unit 131 receives various information transmitted from the user terminal 300 and transmits various information to the user terminal 300. The data transmission/reception unit 131 references various management tables stored in the memory unit 120. The various information transmitted and received by the data transmission/reception unit 131 includes, for example, information about objects placed in the virtual space managed by the content management unit (for example, operations to place, delete, or move an object associated with the user in the virtual space), as well as game information and various notifications. Information about objects placed in the virtual space includes, for example, information for displaying character objects and decorative objects stored as object data 121 on the user terminal 300.

<Configuration of user terminal>
Next, the configuration of user terminal 300 will be described in detail. As shown in Fig. 3, the user terminal includes a communication unit 310 that communicates with other computers including distribution server 100, a storage unit 320 that stores various data, an operation unit 330 for inputting operations, an output unit 340 for outputting images, an information acquisition unit 350, and a control unit 360 that controls the entire computer. Communication unit 310, storage unit 320, operation unit 330, output unit 340, information acquisition unit 350, and control unit 360 are interconnected by a bus line.

The communication unit 310 is a communication interface for wired or wireless communication. The communication unit 310 communicates with other computers, including the distribution server 100, via the network 2.

The storage unit 320 is a memory composed of RAM, ROM, etc. The storage unit 320 stores programs for executing various control processes (for example, a program for executing an AR content application that enables augmented reality photography and image display, a program for viewing content that uses virtual spaces such as the Metaverse, etc.), various data, etc. Examples of the various data include character objects and decorative objects stored as object data 121 on the distribution server 100, and data that allows users to re-edit synthesized images created by augmented reality (for example, captured images and virtual space data that specifies the coordinates of various objects).

The operation unit 330 includes an input device (e.g., a touch panel, touch pad, pointing device such as a mouse, keyboard, microphone, etc.) for receiving input operations and voice from the user. In this embodiment, user operations refer to operations from the user on the operation unit 330. Examples include touch operations on the touch panel, slide operations, flick operations, button operations, drag (swipe) operations, pinch-in/out operations, operations on icons displayed on the display unit of the user terminal 300, operations on a pointing device or keyboard, and voice input to a microphone. User operations on the operation unit 330 enable the selection, movement, rotation, scaling, etc. of unreal objects that may be displayed by the AR content.

The output unit 340 includes an output device (such as a display unit or speaker) for presenting and outputting information (text, images, audio, etc.) to the user.

The information acquisition unit 350 is an input device for acquiring information about real space, and includes various sensors such as a camera and an acceleration sensor and a gyro sensor. The information acquisition unit 350 performs imaging processing of real space and identifies the distance from the information acquisition unit 350 to an object in real space (e.g., depth, depth position) and the shape of the object. The information acquisition unit 350 is, for example, a camera that combines a lens (optical system) and an image sensor (image capture element), and includes a depth-sensing camera equipped with a depth sensor that measures the distance from the terminal to an object or the distance between objects. For example, it includes a camera equipped with a depth sensor (e.g., a ToF (Time of Flight) sensor). The camera may be a stereo camera capable of estimating depth, or it is possible to use a camera equipped with various sensors that enable depth estimation within a captured image.

The control unit 360 is composed of a CPU and other components. The control unit 360 controls the overall operation of the user terminal 300 and performs calculations by executing programs stored in the storage unit 320.

The functional configuration of the control unit 360 will be described below. The control unit 360 includes an image processing unit 361 for displaying AR content. The image processing unit 361 functions as an acquired information analysis unit 362, an augmented reality data generation unit 363, a display control unit 364, and a re-edited data management unit 365, among others.

The acquired information analysis unit 362 detects planes (horizontal or vertical) in real space and identifies the coordinates of objects, etc., based on the captured image (general camera image) acquired by the information acquisition unit 350 and the depth information (depth information in image format generated based on sensor measurements or depth estimation by machine learning) identified (estimated) by the depth sensor of the information acquisition unit 350. For example, it is possible to identify the shape of an object shown in the image acquired by the information acquisition unit 350 and identify the position (coordinates) of the object. The acquired information analysis unit 362 also identifies information about the camera, such as the size, focal length, and brightness of the camera sensor, associated with the captured image captured by the information acquisition unit 350. The acquired information analysis unit 362 determines a coordinate system (e.g., world coordinates) based on the information acquired by the information acquisition unit 350.

The augmented reality data generation unit 363 generates a virtual space (a virtual space for realizing augmented reality) in which unreal objects (such as character objects and decorative objects) that can be displayed by AR content are placed. The virtual space for realizing augmented reality, described below, is information for identifying coordinates using augmented reality and includes at least information that enables identification of the anteroposterior relationship between unreal objects and objects in real space (real objects). In this embodiment, a position within the virtual space is identified based on coordinates determined by the acquired information analysis unit 362 based on depth information, etc. The augmented reality data generation unit 363 places character objects, decorative objects, etc. stored in the memory unit 320 in the virtual space in response to user operation. Based on information acquired by the information acquisition unit 350, the augmented reality data generation unit 363 places a virtual camera in the virtual space at a position corresponding to (equivalent to) the real-world captured image and controls the position, orientation, tilt, etc. of the virtual camera. In addition, the augmented reality data generation unit 363 generates a composite image by combining an image of an object in the virtual space acquired from the virtual camera with the captured image acquired by the information acquisition unit 350.

The augmented reality data generation unit 363 also generates a transparent polygon mesh (hereinafter simply referred to as a mask or mask polygon) as a real object corresponding to the object detected by the acquired information analysis unit 362, as shown in Figures 5(C) and (D) described below. The mask is placed at coordinates (placement position) specified by depth information, etc., of the imaging information. For example, the augmented reality data generation unit 363 creates a mask polygon using depth information, etc., acquired by the information acquisition unit 350, making it possible to create a front-to-back relationship between the character object and the mask polygon in virtual space. Note that although the mask is transparent and not displayed, it is also referred to as a real object because it is an object corresponding to a real object.

The display control unit 364 displays on the display unit (such as a display) an image of the virtual space stored in the storage unit 320 based on the content data acquired by the communication unit 310, which corresponds to the field of view of the virtual camera. In other words, the user can change the position, orientation, tilt, etc. of the virtual camera placed in the virtual space in response to operations on the input unit 330, thereby displaying on the user terminal 300 an image of the virtual space within the field of view of the virtual camera. The display control unit 364 also displays on the display unit (such as a display) a composite image of the real space image and the virtual space image generated by the augmented reality data generation unit 363. The display control unit 364, for example, renders and draws two-dimensional or three-dimensional image data.

The re-editing data management unit 365 stores in the storage unit 320 a composite image (augmented reality image) of a real-space image and a virtual-space image displayed on the display unit of the user terminal 300 by the display control unit 364 in response to a user's shooting operation via the operation unit 330. The re-editing data management unit 365 also stores re-editing data in the storage unit 320 so that augmented reality images captured using AR content can be re-edited. The re-editing data includes information for recreating the world as seen from the camera at the time of capture in a state that allows for additions and corrections to mask information (e.g., depth overwriting). For example, the re-editing data includes a background captured image (general camera image) and virtual-space identification information for identifying the virtual space. The virtual-space identification information includes, for example, depth information, transparent polygon mesh information corresponding to real objects, information on the position and orientation (direction) of the camera in the virtual-space coordinate system associated with the background captured image, and information about the camera that captured the captured image included in the information acquisition unit 350 (e.g., camera sensor size, focal length, brightness, etc.). This makes it possible to re-edit the augmented reality image using the synthesized 3DCG after the image has been captured using the AR content in this embodiment.

<Example of screen display during AR shooting>
(Example of a shooting screen)
4 to 9, the display mode of the AR content on the display unit of the user terminal 300 in this embodiment will be described. Fig. 4 shows an example of a capture screen when the AR content program in this embodiment is executed and augmented reality capture (hereinafter also referred to as AR capture) is performed. Fig. 4(A) shows an example of a screen displayed in which a captured image of real space captured and acquired by the information acquisition unit 350 of the user terminal 300 is combined with 3DCG (three-dimensional computer graphics) of a character object C.

In this embodiment, when augmented reality shooting begins with AR content, floor coordinates are determined to correspond to the plane detected by a plane scan to determine the horizontal plane corresponding to the floor of the virtual space. While the information acquisition unit 350 captures a real-world image that could potentially serve as a background image, such as that shown in FIG. 5B, the acquired information analysis unit 362 detects the plane as the user operates the device. The first plane detected is a horizontal plane that could potentially serve as the floor, such as the floor or ground. After the plane is detected, the user can confirm the start of shooting (e.g., by pressing the start button), and a preview of the character object C placed in real space is displayed on the display unit of the user terminal 300, as shown in FIG. 4A or 5A. The character object C can be moved by tapping on the plane and the detected area. The orientation of the displayed character object C can also be rotated in response to an operation (e.g., a slide operation) performed on the character object C. The initial placement of character object C in the virtual space may be automatically placed at default coordinates or the like based on the user's operation to confirm the start of shooting, or may be placed by accepting a tap operation or the like from the user.

The shooting screen shown in Figure 4 (A) displays multiple operation icons and the like for the user to operate when taking AR photos. For example, the character change icon T1 is an icon for changing the character object C to be placed to a different character. The reset icon T2 is an icon for re-detecting the horizontal plane that serves as the reference when placing the character object C. For example, if you want to re-detect the plane because the location detected as a plane is misaligned with the actual floor, for example, operating the reset icon T2 will re-detect the plane and re-position the character object C on the re-detected plane. The UI display icon T3 is an icon for switching between displaying and hiding the UI (User Interface), such as icons and operation menus. Operating the UI display icon T3 makes it easier for the user to grasp the overall picture of the image being taken.

Based on the depth information and other information identified by the acquired information analysis unit 362, character object C is positioned to fit into the real space (for example, to achieve a state where there is no inconsistency (geometric consistency) in the three-dimensional alignment between the real space and the virtual object according to the user's viewpoint), as shown in Figure 4(B). Therefore, for example, if the user moves backward away from character object C (see the movement of user U), as shown in Figure 4(B), the position at which character object C is positioned will become farther away, and character object C will also be displayed smaller in size (wide angle of view). Note that even if the character is positioned at the back of the screen by tapping the back side of the shooting screen, character object C will be displayed in a wide angle of view.

Furthermore, because character object C is 3DCG, if the user moves their shooting position so as to go around behind character object C displayed on the display unit, as illustrated in FIG. 4(C) (see the movement of user U), they can photograph character object C from behind at the position (coordinates) of character object C in real space shown in FIG. 4(A). In this way, the user can photograph an image from a preferred angle of view by adjusting their own standing position and the orientation of the user terminal when photographing. Also, as will be described later with reference to FIG. 5, a mask has been generated for chair R, which corresponds to an object in real space, based on depth information acquired by information acquisition unit 350. As a result, on the screen of FIG. 4(C), part of character object C, which is 3DCG, is obscured by chair R and is not displayed on the display unit of user terminal 300.

At the bottom of the shooting screen in Figure 4, an operation menu T10 with multiple tabs is displayed. Of the multiple tabs displayed in the operation menu T10, for example, "Expression," "Pose," and "Motion" are menus for changing the display mode of the character object C. Motion refers to making the character object C perform an animated action such as dancing or jumping. For example, as illustrated in Figure 6 (D) described below, selecting the motion tab and one of the actions will cause the character object C to perform the action. In addition, by operating the speed adjustment icon T4, the user can adjust (control) the motion speed to a speed of their choice. Note that the motion speed control operation cannot be performed (selected) during initial shooting, but may be an operation that can be performed under calm conditions during re-editing.

"Special" is a tab that allows you to change the display mode of character object C to a special mode that is different from the normal mode. For example, time-limited motions can be selected. "Decoration" is a tab that displays a menu for decorating images, such as placing decorations like stickers and decorative objects like frames, and adjusting colors using filters. For example, decoration operations using stickers are possible, as shown in Figure 7 below. Furthermore, a tab that allows you to change the costume of character object C may be displayed within the operation menu T10.

Note that operating any of the menus displayed in the operation menu T10 or changing the character by operating the character change T1 may not be possible (selectable) during the initial shooting, but may be possible during the re-editing operation described below. For example, the decoration operation performed by operating "Decorate" may not be possible when initially saving the composite image created by augmented reality that combines the captured image with character object C, but may be possible during the re-editing operation after the captured image has been taken. Hereinafter, decoration operations will be described as operations that can be performed during re-editing.

The capture button S is an operation button for saving the image displayed on the display unit. For example, by operating the capture button S in the display mode shown in FIG. 4A , in which the real space and the character object C are combined, a composite image combining the CG of the previewed character object C with the captured image of the background is saved in the storage unit 320 of the user terminal 300. For example, the previewed image is rasterized (image data is converted into pixel data to reduce the size) at high resolution and saved. The saved image can be viewed, for example, in an image data folder in the storage of the user terminal 300 (for example, the same folder as the folder in which images captured with the camera app installed on the user terminal 300 are saved, or a folder dedicated to AR content). Furthermore, the AR content in this embodiment can capture not only still images but also videos. When capturing videos, the capture button S can be operated to start and stop video capture. Note that a still image/video switching icon may be displayed on the capture screen shown in FIG. 4A or the like.

Furthermore, in the AR content of this embodiment, even after the shooting button S is operated and an image is generated by combining a captured image of real space with the character object C, the composite image can be re-edited and re-combined. Therefore, when the shooting button S is operated and AR photography is performed, a rasterized composite image is generated and re-editing data is stored. The re-editing data includes information for identifying the captured image and virtual space in which the character object C can be re-combined. The virtual space identification information includes information for identifying a mask (real object) corresponding to a real object (for example, information such as coordinates and shape).

(Example of information for generating a composite image)
With reference to FIG. 5, information for generating a composite image in this embodiment will be described. FIG. 5(A) is a composite image displayed on the capture screen of FIG. 4(A) described above. FIG. 5(B) is a captured image of real space captured by the information acquisition unit 350, which serves as the background image of FIG. 5(A). FIG. 5(C) is an example of virtual space data in which real objects corresponding to real objects in the captured image of FIG. 5(B) are placed. FIG. 5(D) is an example of virtual space data to be composited with the captured image of FIG. 5(B), in which a character object C, which is a non-real object, is placed in the virtual space data of FIG. 5(C). FIG. 5(E) is an example of virtual space data when the virtual space of FIG. 5(D) is viewed from the right side of FIG. 5(D) (the left side of the character in FIG. 5(D)).

At the start of AR imaging, the user moves the user terminal 300 as if scanning the road shown in Figure 5 (B), and the road is detected as a flat surface. Also, a chair R1 and a tree R2 exist in the real space and are displayed in the captured image acquired by the information acquisition unit 350. The acquired information analysis unit 362 identifies the shape and coordinates of real objects based on depth information and other information from the imaging information acquired and accumulated by the information acquisition unit 350, and renders them in real time. For example, the acquired information analysis unit 362 detects objects and other objects appearing in the captured image based on the depth information and other information of the acquired imaging information. The augmented reality data generation unit 363 places real objects corresponding to the real objects and other objects in the virtual space based on the depth information and other information analyzed by the acquired information analysis unit 362. The real objects are mask polygons (transparent polygon meshes that are not drawn on the display unit) generated in positions and ranges corresponding to the real objects appearing in the captured image. For example, a mask process is performed on the range corresponding to the shape identified based on the depth information and other information in the imaging information. For example, a real object RO1 with a shape corresponding to a chair R1 and a real object RO2 with a shape corresponding to a tree R2 are placed in the virtual space. In this way, mask polygons corresponding to real objects are generated in real time.

Figure 5(C) is an example of a virtual space in which real objects corresponding to the objects in the captured image of Figure 5(B) are placed. Figure 5(D) is an example of a virtual space in which a character object C is placed in addition to the real objects in the virtual space of Figure 5(C). The plane of the virtual space corresponds to the plane detected at the start of AR shooting using the imaging information. Furthermore, the virtual camera is positioned so that the real space acquired by the information acquisition unit 350 and displayed on the display unit of the user terminal 300 corresponds to the virtual space (e.g., achieving geometric consistency). The real objects RO1, RO2, and character object C are positioned so that they correspond to the coordinates of the real space. The relationship between the character objects and decorative objects described below and the coordinates (horizontal, vertical, and depth information, x, y, and z axes) of the real objects in the virtual space is determined based on depth information, etc. For example, the character object is positioned at a size that fits the real space in the captured image based on depth information, etc. in the information acquired by the information acquisition unit 350. For example, if a character object with a height set to 160 cm is placed, it will be composited into the captured image and displayed in a manner that makes it appear to be 160 cm tall in the real space within the captured image.

The real object is a stereoscopic, three-dimensional mask, as exemplified in Figures 5(C) to 5(E). The mask is rendered so that a corresponding mask is generated each time the information acquired by the information acquisition unit 350 is updated during image capture. Furthermore, by generating a mask, when a character object, for example, is positioned further back than the real object, as shown in Figure 4(C) above or Figure 6(B) below, occlusion processing can be performed to prevent a portion of the character object from being drawn by the mask corresponding to chair R1. This allows the character object to appear as if it is occluded by the real chair R1. For example, in Figure 5(D), character object C is positioned in front of real object RO1, so in Figure 5(A), character object C is drawn in front of chair R1, and a portion of chair R1 is occluded by character object C. On the other hand, when the user taps or performs other operations to position character object C further back than chair R1, character object C is positioned at coordinates further back than real object RO1 in the virtual space of Figure 5(D). As a result, as shown in Figures 4(C) and 6(B), part of the character object C is displayed as being obscured by the chair R1. In other words, the character object C is not rendered on the display unit in the area that overlaps with the masked area as seen from the virtual camera.

As described above, an image (e.g., a high-resolution rasterized image) combining the captured image and virtual space data is saved in the storage unit 320 by operating the capture button S in FIG. 4. Furthermore, while the composite image of FIG. 5(A) is saved, information for recreating the world at the time the composite image was captured, allowing for additions and modifications to the mask information, is stored in the storage unit 320 as re-editing data, so that the composite image can be re-edited. Adding and modifying the mask information includes overwriting the depth of real objects in conjunction with changes to the coordinates and display mode of character object C. The re-editing data includes, for example, virtual space identification information (e.g., depth information, information on the shape and coordinates of real objects, information on the camera's position and orientation, information about the camera at the time of capture, etc.) that enables the captured image of FIG. 5(B) and the corresponding virtual space data of FIG. 5(C) to be reproduced. Furthermore, the virtual space identification information reproduced by the re-editing data may be data excluding information about character object C, as in FIG. 5(C).

(Overview of re-editing operations)
In the AR content of this embodiment, even after AR shooting, a composite image of a captured image of real space and 3DCG can be re-edited based on re-editing data. Editing functions enabled by the re-editing operation include decoration operations using depth information for the composite image of FIG. 5A . It is also possible to rearrange the character object C and change the display mode of the character object C. Conventionally, after AR shooting, when the 3DCG of the character object and the captured image serving as the background are combined, the only way to edit the character, and even to add decoration, was to add a two-dimensional decoration layer in front of the character (toward the display unit). However, in this embodiment, the re-editing data can be used to re-combine the character object and perform decoration operations so that it is geometrically consistent with the captured image. Therefore, a three-dimensional front-to-back relationship can be created between the real object, the 3DCG of the character, and the decoration object. This allows for greater freedom in the expression obtained by editing than with conventional editing using 2D layers, increasing the user's interest.

The re-editing screen may be accessible from a menu of an application or the like that can provide the AR content of this embodiment that has been downloaded to the user terminal 300. For example, by selecting a re-editing operation from a menu on the home screen, the composite image of Figure 5 (A) may be drawn on the display unit based on the re-editing data.

The re-edited images and videos can be output and saved to the user terminal 300's storage by operating the capture button S, just as when capturing the first image. In other words, the user can re-capture the augmented reality composite image after editing it to their liking.

The re-editing screen displayed based on the re-editing data may, for example, display operation icons and menus similar to those shown in FIG. 4(A), or some operation icons may be hidden. One difference from the initial shooting is that the captured image is not acquired in real time by the information acquisition unit 350, and the captured image that will be used as the composite background image is an image that has already been captured. Therefore, the captured image itself cannot be different, as in FIG. 4(B) or FIG. 4(C). However, because the re-editing is based on an image that has already been captured using AR photography, the user can easily estimate the ideal placement position for the character object. Hereinafter, operation icons and the like displayed on the re-editing screen will be described as being the same as the icons and the like described in FIG. 4. The decoration screen of FIG. 7 is displayed, for example, by selecting "Decoration" from the operation menu T10 of FIG. 4(A) displayed as the re-editing screen.

(Example of character object re-editing screen)
FIG. 6 shows an example of an editing screen for a character object C during re-editing. Note that (A) to (C) of FIG. 6 show an example of a screen in which the operation icons are hidden, while FIG. 6 (D) shows an example of a screen in which the operation icons are displayed. Assuming that the re-editing data is the data captured in FIG. 4 (A), the re-editing data is previewed as a composite image into which the character object C is superimposed, as shown in FIG. 6 (A). The virtual space generated based on the re-editing data is a space in which the real objects are positioned when the image in FIG. 4 (A) is captured, as shown in FIG. 5 (C). Note that if the re-editing data does not include information about the character object C, the initial position of the character object C in the composite image displayed during re-editing may be automatically positioned when the re-editing screen is displayed, or may be positioned by a tap operation or the like by the user. Furthermore, the type of character object C positioned during re-editing may be a type that is preset as a default by user operation or the like, or may be a type selected by the user during re-editing. During re-editing, it is possible to change the position of the character object C, change its pose, motion, costume, and facial expression, and even change the type of the character itself. For example, the character object C may be moved by tapping on the desired location or by using the operation icons (D1 to D4) illustrated in FIG. 7, etc. As illustrated in FIG. 6(B), if the character object C is placed further back than a chair R1 in the captured image, the area of the character object C that overlaps with the chair R1 as viewed from the virtual camera is obscured by a real object RO1 that has been masked to correspond to the shape of the chair R1, and is not drawn when output to the display unit.

Furthermore, even if the position of character object C is changed during re-editing, the display size is adjusted to correspond to the height of character object C based on coordinate data corresponding to the captured image, which is included in the re-editing data. Therefore, for example, as shown in Figure 6(C), if character object C is placed further back than the foreground in the captured image by a user operation, the aspect ratio of character object C is changed to correspond to the position, and it is displayed smaller.

Figure 6 (D) shows an example of a motion being performed by character object C of the re-edited data in Figure 6 (A). During re-editing, the motion speed can be controlled by operating the speed adjustment icon T4. Therefore, for example, by making the character move in slow motion, it becomes easier for the user to capture the desired image.

In addition, the AR photography using the AR content in this embodiment may be capable of continuous photography (for example, taking 10 consecutive photos every second). The control unit 310 may also be able to determine one image to recommend to the user from multiple composite images generated by continuous photography and provide it to the user. For example, from multiple images, the image processing unit 361 may identify the image in which the face of character object C is facing most forward, or the image in which the face is least obscured (for example, the image in which the face is not obscured by a motion such as the character object's hand movements). For example, the image determined to include the largest area of the frontal portion of character object C's face may be extracted and presented to the user.

Also, because character object C is 3DCG, it can be rotated in the same way as when it was first photographed. In other words, it is not limited to drawing the front of character object C; it is also possible to rotate character object C so that it is drawn diagonally, rearward, etc.

Furthermore, when re-editing, not only can the display mode of the originally placed character object C itself be changed, or a different character object be created (for example, changing from character A to a different character B), but multiple types of character objects can also be added and placed.

(Example of re-editing and decoration operation screen)
7 shows an example of a screen for performing operations to decorate an AR image using a decoration tool during re-editing. The decoration tools include photo trimming (reducing/enlarging, etc.), inset frame selection, decoration (sticker, stamp, etc.) placement, particle placement, post-effect functions, etc. Post-effect functions include depth of field (focus) processing using depth estimation, image stabilization, color filters, person extraction, etc. Hereinafter, objects that can be placed in an image, such as inset frames, stickers, particles, and text, are also referred to as decoration objects.

Figure 7 shows an example screen for placing a sticker. The sticker is a decorative object that can be placed in front of or behind a character object C in virtual space based on depth information included in the re-edited data. For example, when a decorative tool menu is displayed at the bottom of the screen and the sticker is selected by a user operation, a sticker DO is placed in the composite image, as shown in Figure 7(A). The display mode of the sticker DO can be edited using operation icons (D1-D4). For example, in Figure 7(A), operation icons are displayed around the sticker DO, including a discard icon D1 for discarding the sticker (cancelling placement), a rotate icon D2 for rotating the sticker, a scale icon D3 for scaling the sticker, and a depth icon D4 for adjusting the sticker's depth. For example, the scale icon D3 allows the sticker to be scaled by touching the icon and operating diagonally, and the depth icon D4 allows the sticker's depth to be adjusted by touching the icon and operating vertically. Furthermore, editing operations to move the stamp vertically, horizontally, or in the depth direction may be possible by sliding the stamp itself, or by operating on another icon. Note that depth operations are not limited to operations on the operation icon, and similar editing operations may also be possible by directly operating on the sticker itself (for example, touching or swiping).

The composite image is previewed on the display of the user terminal 300 in a composited state even during re-editing. However, because the re-editing operation processes data before final rasterization, which is the stage prior to final output (also called printing), editing operations using 3D CG data are possible. Furthermore, decorative objects that can be placed on the re-edited data are also associated with three-dimensional information, including depth. For example, as shown in FIG. 7(B), when a user operates the depth icon D4 to place sticker DO at the back, the size of the sticker also changes. This is because the sticker is placed at coordinates farther from the virtual camera in FIG. 5(C). Furthermore, when sticker DO is placed behind chair R1, the chair R1 obscures the area where sticker DO overlaps with chair R1 when viewed from the front, and sticker DO is not rendered. Placeable stickers may include both 3D and flat stickers.

Figure 8 shows an example screen when trimming a photo or adjusting the position of an inset frame. For example, the user can pinch the touch panel to enlarge or reduce the image, and drag to move it, resulting in image editing as shown in Figure 8 (A) to (B). When trimming, it may be possible to select whether or not to display the inset frame (frame F). Furthermore, in this embodiment, because character object C is positioned as 3DCG data, even if trimming is performed, character object C is in a state before final rasterization (before final output together with the captured image and editing content created by decorative operations), so the edges are rendered neatly when final output.

Furthermore, the decorative object of the inset frame (frame F) is positioned in front of other objects in terms of coordinates and is always drawn at the same scale, but the user may be able to adjust its anteroposterior relationship with other decorative objects. The decorative objects may include sticker objects of a type that are placed relative to the frame coordinates. For example, stickers of a type that are placed relative to the frame coordinates may be placed and drawn as 2D layers overlaid on frame F. For stickers of a type that are placed relative to the frame coordinates, a button (selectable button) may be displayed to switch the display position between in front of and behind the frame. For example, this may be displayed together with an operation tool. As shown in FIG. 8(C), stickers selected to be displayed in front of frame F are displayed in front of the frame coordinates, like sticker DO2. In other words, they are positioned in a position closer to the virtual camera capturing FIG. 5(C). On the other hand, sticker DO3 has been selected to be displayed behind the frame, and is therefore positioned behind the frame coordinates.

Decorative objects in the inset frame and stickers of a type placed relative to the frame coordinates are positioned so that they are drawn in front of other objects. However, while displayed at full size on the display unit of the user terminal 300, the coordinates may be moved further back in the virtual space by user operation. For example, if frame F needs to be brought closer to character object C, frame F can be moved closer to character object C by user operation of the operation tool (e.g., by enlarging the composite image from FIG. 8(A) to FIG. 8(B)). In this case, if a real object is placed in front of frame F, the depth of the real object can be overwritten with the position of frame F, so that frame F is positioned in front as seen from the camera. Furthermore, with regard to character object C, all or part of the character object may be drawn in front of frame F. This makes it possible to express the hands or body of character object C as if they are protruding from frame F, as exemplified in FIG. 8(C).

Figure 9 is an example of a screen where particle decorative objects have been placed. Particles are decorative objects that involve animation. Figure 9 illustrates an example of particles P that can drop flower objects. Flower particles P are made up of multiple flower objects. Coordinate information, including depth, is associated with each particle object as well. Therefore, if flower particles P are placed so that they overlap with the placement position of character object C, some of the flower objects will be drawn in front of character object C, as their coordinates are in front of character object C. On the other hand, flower objects associated with coordinates behind character object C will be drawn behind character object C, and so the parts that overlap with character object C will not be displayed.

In addition, based on the depth information acquired during image capture and particle coordinate information included in the re-editing data, it is also possible to have particles drawn only in the back, as shown in the example of Figure 9(B). In this case, the size is changed according to the estimated distance, as explained for the stickers in Figure 6(B). This means that, for example, if you want particles such as snow to fall in an area 30 meters away from a subject in the foreground in the captured image based on depth information, it is possible to draw the particles so that they appear as small particles in the back. This increases realism. Note that editing operations may be performed by displaying an operation icon to move the location or size, or by directly operating on the particles to change the display mode, as shown in the example using stickers in Figure 7.

Furthermore, when it is determined that there is a wall in the captured image and particles are placed outside the wall, as shown in Figure 9(C), occlusion processing can be used to mask the area where the wall exists (placement of real objects), preventing particles from being drawn within the area of the wall. On the other hand, particles are drawn in the area outside the wall (in the case of Figure 9(C), which corresponds to the window, which is further back than the wall).

The particle animation may continue to play until the capture button S is operated. When the capture button S is operated, the animation is fixed at the position at the timing of the operation and output. The particle animation speed may also be controlled by user operation. For example, a speed adjustment icon may be displayed in the same way as the character object motion, and the speed may be made variable by operating the icon. This allows still images and videos to be taken with the user's preferred animation timing, thereby expanding the scope of creative expression using AR content.

Post-effect functions that can be edited with the decoration tool may utilize the re-editing data (3D scene storage using virtual space identification information) to adjust depth of field (focus) using depth estimation, apply image stabilization, color filters, or extract people from captured images. For example, if a person is not identified during the initial capture, the person may be extracted during a later re-editing process, and image recognition processing may be used to apply privacy processing (such as mosaic processing) to the portion of the captured image corresponding to the person. Furthermore, during re-editing, it may be possible to extract not only people but also product logos and other items, and apply mosaic processing. In this way, since the re-editing data includes both the captured image and information about the real objects corresponding to the captured image (such as transparent polygon mesh information corresponding to real objects), it is possible to apply post-processing to the objects in the background captured image itself according to the purpose, while recombining unreal objects such as characters into the real world while maintaining a 3D relationship, unlike conventional methods. Therefore, with the AR content in this embodiment, the user can easily operate the composite photograph, without having to operate an application that requires specialized knowledge, thereby expanding the range of expressions possible with the composite photograph, and also expanding the range of uses for the captured composite photograph.

Note that decorative objects may be either 2DCG or 3DCG, as long as they are associated with coordinates. It may also be possible to rotate the 3DCG of decorative objects. There may also be a template decoration function that combines multiple sets of decorative tool functions. For example, a template that combines a set of effects such as frames, multiple stickers, visual effects such as particles, and post-effects may be selectable for each theme. This allows even users who are unfamiliar with image editing to easily decorate. As described above, making it easy to edit 3DCG based on re-edited data broadens the scope of creative expression for images that combine real space with favorite characters, enhancing user enjoyment.

<AR content shooting process flow>
(AR photography processing)
10 is a flowchart for explaining the AR image capturing process when capturing the AR content of this embodiment for the first time in the user terminal 300. The control unit 360 repeatedly performs the AR image capturing process at predetermined time intervals in response to an operation by the user to launch an application for executing the AR content, etc.

In step S101, the AR photography mode is activated. For example, the AR photography mode is activated when it is determined that a request to start AR photography has been made, such as when an application capable of executing AR content installed on the user terminal 300 is launched in response to a user operation. In addition, by activating the AR photography mode, the acquired information analysis unit 362 detects a plane from the captured image acquired by the information acquisition unit 350.

In step S102, the augmented reality data generation unit 363 executes a process of placing an unreal object at a predetermined position in the virtual space where the real object is placed, based on the depth information and color information identified by the acquired information analysis unit 362 from the information acquired by the information acquisition unit 350. For example, after a plane in the captured image is detected in step S101, the augmented reality data generation unit 363 places a character object C (unreal object) in the virtual space (FIG. 5C) where the detected plane (the plane in FIG. 5B) corresponds (e.g., angle) to the plane that serves as the floor, as illustrated in FIG. 5. The predetermined position where the unreal object character object C is placed can be moved from its initial coordinates to different coordinates by user operation. For example, the location where the character object C is displayed can be changed by performing a tap operation or the like on the desired location. Furthermore, if an object, surface, etc. is detected based on the depth information, etc. identified by the acquired information analysis unit 362, the object, surface, etc. are masked. For example, as shown in Figure 5(B), if a chair R1 or a tree R2 is detected as an object, the transparent polygon meshes of the real objects RO1 and RO2 are placed in the corresponding virtual space.

In step S103, a process is executed to display a composite image obtained by combining the captured image and virtual space data on the display unit of the user terminal 300. For example, the display control unit 364 renders image data obtained by combining the captured image of FIG. 5(B) with the corresponding virtual space data of FIG. 5(C) on the display unit. At this time, if character object C is placed at coordinates deeper than real object RO, as shown in FIG. 6(B), the area that is blocked from the perspective of the virtual camera of character object C will not be drawn when rendering.

In step S104, a real object is generated based on depth information, etc., identified from the information acquired by the information acquisition unit 350, and the virtual space is updated. For example, if the image of real space acquired from the information acquisition unit 350 changes because the user changes the angle of the user terminal 300, the augmented reality data generation unit 363 generates a mask at predetermined time intervals according to the object, surface, etc. identified based on the depth information, etc., acquired from the information acquisition unit 350 (mask processing). In other words, the real object in the virtual space is updated.

In step S105, it is determined whether a display mode change condition for the unreal object has been met. If it is determined that the display mode change condition has been met, in step S106, a process is executed to change and update the display mode of the unreal object in accordance with the met display mode change condition for the unreal object. Meeting a display mode change condition includes, for example, an operation to change the facial expression, pose, or costume of character object C by selecting it from the operation menu T10 of FIG. 4; selecting a motion to cause character object C to move; or an operation to change character object C to a different type of character object C by operating character change T1. For example, when the augmented reality data generation unit 363 determines that an operation to change the facial expression has been performed, it updates character object C so that the facial expression of character object C is changed to the changed facial expression. Furthermore, when a motion is selected, the display mode is switched and updated so that character object C performs the motion action.

If it is not determined in step S105 that the display mode change condition for the unreal object has been met, or after the unreal object has been updated in accordance with the display mode change condition in step S106, processing is performed in step S107 to display a composite image, which is a composite of the captured image and the virtual space data, on the display unit of the user terminal 300. For example, the display control unit 364 uses real-time rendering to render the virtual space data updated in step S104 and the captured image that has been acquired, on the display unit, so that a preview is displayed. Furthermore, if the display mode of the character object C in the virtual space has been changed in step S106, rendering is performed in the changed display mode.

In step S108, it is determined whether a shooting operation has been performed. If it is determined that a shooting operation has been performed, in step S109, the captured image, virtual space identification information, and composite image at the time the shooting operation was received are stored in the storage unit 320. For example, when the shooting button S on the shooting screen illustrated in FIG. 4 is tapped, the re-editing data management unit 365 stores the captured image that is rendered at the time the shooting button S is operated and serves as the background image of the composite image previewed on the display unit, virtual space identification information for identifying the virtual space of FIG. 5(C) that includes information such as the position and shape of real objects, and a composite image obtained by rasterizing the previewed composite image at high resolution. The captured image and the virtual space identification information including coordinate information are stored so that they can be read as re-editing data for re-editing the composite image captured in step S108. Furthermore, since AR shooting can continue after step S109, the process returns to step S104 and repeats.

On the other hand, if it is not determined in step S108 that a shooting operation has been performed, it is determined in step S110 whether an end operation has been performed. For example, the process ends when an operation to end AR shooting has been performed, such as an operation to close the application that executes the AR shooting process, or an operation to execute the AR re-shooting process described below with reference to Figure 11. If it is not determined in step S110 that an end operation has been performed, the process returns to step S104 and repeats the process.

(AR re-shooting process)
11 is a flowchart for explaining the AR re-capture processing based on the re-editing data stored in step S109 of the AR capture processing described with reference to Fig. 10 in the user terminal 300. The control unit 360 repeatedly performs the AR re-capture processing at predetermined time intervals in response to an operation from the user requesting re-editing of the AR image to an application for executing AR content, etc.

In step S201, it is determined whether a re-shooting operation has been performed. If it is determined that a re-shooting operation has not been performed, the process ends. On the other hand, if it is determined that a re-shooting operation has been performed, in step S202, the stored captured image and virtual space identification information is read out. For example, the re-editing data management unit 365 reads out the re-editing data stored in step S109 of FIG. 10. The re-editing data is data stored when the shooting operation was performed in step S108 of FIG. 10. For example, the captured image of FIG. 5(B) that serves as the background image and virtual space identification information for identifying the virtual space in which the real object of FIG. 5(C) is placed are read out from the storage unit 320. The virtual space identification information in step S202 includes, for example, information such as the coordinates and shape of the real object (transparent polygon mesh), and information such as the orientation of the virtual camera.

In step S203, a process is performed to place a non-real object at a predetermined position in the virtual space based on the read virtual space identification information. For example, the augmented reality data generation unit 363 re-positions the character object C at a position in the virtual space as shown in FIG. 5(D) based on the virtual space data of FIG. 5(C) in which the real object RO generated based on the virtual space identification information included in the re-editing data is positioned. This re-positioning may be automatically displayed when the re-editing screen is displayed, as with the initial positioning at the time of initial shooting, or may be performed by a positioning operation by the user.

In step S204, processing is performed to display a composite image obtained by combining the captured image with the virtual space data. For example, the display control unit 364 combines the captured image with the virtual space in which the object is placed, and renders it so that it is displayed as a preview on the display unit as shown in Figure 4(A). At this time, unlike the initial capture described in Figure 10, the captured image is not newly acquired but is fixed.

In step S205, it is determined whether an operation to add and place an unreal object has been performed. For example, if an operation to place a decorative object as described with reference to Figures 7 to 9 or an operation to add and place a character object C is performed, it is determined that an operation to add and place an unreal object has been performed.

If it is determined in step S205 that an additional placement operation has been performed, then in step S206, the added unreal object is placed in the virtual space, and the virtual space is updated. For example, as shown in Figure 7, when a selection is made to place a type of sticker that can be placed in a posterior-to-posterior relationship with character object C, the coordinate information associated with the sticker is associated with the virtual space data, and the decorative object sticker DO is placed, updating the virtual space.

If it is determined in step S205 that no operation to add or place an unreal object has been performed, or after an unreal object has been added or placed, it is determined in step S207 whether or not a condition for changing the display mode of the unreal object has been met. For example, for character object C, it is determined that a condition for changing the display mode has been met when an operation to change the facial expression, pose, costume, motion, etc. by operating the operation menu T10, an operation to move the placement position by tapping, an operation to rotate the object three-dimensionally, or an operation to change the character type by operating the character change icon T1 has been performed. Furthermore, for decorative objects, it is determined that a condition for changing the display mode has been met when an operation to enlarge or reduce the object or an operation to change its placement position has been performed. It should be noted that for character motion and particle animation, a display mode change operation is also determined to have been performed when a speed control operation is performed.

In step S208, the display mode of the unreal object is changed and updated in accordance with the display mode change condition determined to be met in step S207. For example, when it is determined that an operation to change the pose of character object C has been performed, the augmented reality data generation unit 363 updates the virtual space by changing the pose of character object C in the virtual space to the pose selected by the change operation.

After the display mode of the unreal object is changed in step S208, processing is performed in step S209 to display a composite image obtained by combining the captured image and virtual space data. For example, the display control unit 364 performs processing to render on the display unit a composite image of the captured image, the captured image of the re-editing data, the updated virtual space displaying the unreal object added in step S206 or changed in step S208, and the captured image.

After the updated virtual space is displayed in step S209, or if it is not determined in step S207 that an operation to change the display mode of the unreal object has been performed, it is determined in step S210 whether a shooting operation has been performed. If it is determined that a shooting operation has been performed, the virtual space identification information and composite image at the time the shooting operation was received are stored in the storage unit 320 in step S211. For example, as in step S108 of FIG. 10 when shooting for the first time, when the shooting button S is operated, the composite image currently being previewed on the display unit is rasterized and stored in the storage unit 320 of the user terminal 300. Note that in step S211, the captured image may also be stored in the storage unit 320, as in step S109 of FIG. 10. The shooting operation in step S210 may enable video shooting; for example, even if the captured image of the re-edited data is a still image, a video of the motion of the character object may be stored.

After the composite image is stored in step S211, or if it is determined in step S210 that a shooting operation has not been performed, it is determined in step S212 whether or not an end operation has been performed. For example, similar to S110 in FIG. 10, if an operation to end the app or an operation to perform a re-editing operation based on different re-edited data is performed, the process ends. On the other hand, if it is not determined that an end operation has been performed, the process returns to step S205 and repeats the process. This makes it possible to generate multiple composite images based on the re-edited data. For example, it is possible to generate multiple patterns of display modes for character objects using the same captured image as the background.

<Examples of specific configurations and effects>

(1) In the above-described embodiment, in steps S102 to S107 of the AR imaging process of FIG. 10, based on depth information acquired from the information acquisition unit 350 (e.g., a camera equipped with a depth sensor), a mask polygon (real object RO1) corresponding to the shape of an object such as a chair R1 that may be included in a captured image of real space, such as that of FIG. 5(B) acquired by the information acquisition unit 350, is positioned at coordinates consistent with the real space. Furthermore, an augmented reality composite image, which combines virtual space data in which an unreal object, such as a character object C, is placed based on user operations, with the captured image of real space, can be displayed on the display unit of the user terminal 300. In other words, a composite image combining a captured image and an object in virtual space can be displayed. Furthermore, in the augmented reality composite image, as in FIG. 4(C) or FIG. 6(B), an unreal object, such as character object C, is positioned further back than a real object, such as chair R1, so that the portion of character object C that overlaps with chair R1 is obscured by chair R1 and not rendered. 10, when the user performs a photographing operation using the photographing button S displayed on the photographing screen of FIG. 4, the captured image acquired by the information acquisition unit 350 and virtual space identification information that specifies the coordinates of the real object RO1, which is the mask polygon of the real object, and the character object C are stored as re-editing data in the storage unit 320. Also, the rasterized composite image generated when the photographing button S is operated is stored in the storage unit 320, and is saved in an image data folder or the like of the user terminal 300. 11 , in steps S202 to S204, a composite image obtained by combining the captured image with virtual space data in which real objects such as those shown in FIG. 5C are arranged, which are identified based on the virtual space identification information stored as re-editing data, is displayed on the display unit of the user terminal 300. In steps S205 to S209, the virtual space data is updated based on user operations, such as adding and arranging unreal objects such as character object C and sticker DO, and changing the display mode of character object C, such as changing the size, expression, motion, or type of character object C, to obtain an augmented reality composite image obtained by combining the updated virtual space data with the captured image of real space, and the updated composite image is displayed on the display unit of the user terminal 300. In addition, when the user operates the capture button S, the composite image in step S209 is subjected to a 3DCG editing operation, and then newly rasterized and can be stored in the storage unit 320. This makes it possible to change unreal objects such as character objects after the initial augmented reality capture operation has been performed, and regenerate a new augmented reality image by combining the captured image with the unreal object. This allows users to calmly edit augmented reality images that display unreal objects such as character objects, making it easier to create images that they are satisfied with and improving user convenience.

(2) In the above-described embodiment, in step S207 of the AR re-photographing process in Fig. 11, as illustrated with reference to Figs. 6 to 9, it is possible to change the display mode of an unreal object, such as changing the coordinates of an unreal object in the virtual space based on operations such as enlarging/reducing the unreal object or moving the unreal object to the left or right, changing the character object C to a different character, changing a decorative object to a different decorative object, or changing the motion, expression, pose, costume, etc. of the character object C. This allows a composite image to be displayed, and the initial augmented reality experience can be improved.

Even after a photographing operation has been performed, unreal objects such as character objects can be changed after the fact in response to a change operation from the user. This allows the user to calmly edit an augmented reality image that displays unreal objects such as character objects, making it easier to create an image that satisfies the user, improving user convenience.

(3) In the above-described embodiment, as illustrated in FIG. 6(D), character object C can be displayed with animation movements based on a predetermined motion, such as dancing or jumping. In the AR re-capture process, in steps S207 to S208 of FIG. 11, character object C can be displayed on the display unit of the user terminal 300 in a state where it is moving with a predetermined motion. This allows the character object to be moved after the fact, even after a composite image has already been displayed and a capture operation has been performed, and the state of the character object moving at a predetermined timing that satisfies the user can be saved, thereby expanding the range of expression and increasing the user's interest.

(4) In the above-described embodiment, as illustrated in FIG. 9, it is possible to display, together with the character object C, an object made up of multiple moving particles, or particles that move a fluid, etc. This allows the decorative object to be moved (animated) after the composite image has been displayed and a photographing operation has been performed, and the appearance of the decorative object at a predetermined timing can be saved, thereby expanding the range of expression and increasing the user's interest.

(5) In the above-described embodiment, in steps S207 to S208 of FIG. 11, the motion of the character object exemplified in FIG. 6(D) or the particle animation exemplified in FIG. 9 can be displayed at a different speed by the user operating the speed change icon T4, for example. This allows the movement of the character object to be changed to slow motion, making it easier for the user to specify the timing of the object's movement at the desired timing, thereby expanding the range of expression and increasing the user's interest.

<Modification>
Modifications to the above-described embodiment are listed below.

(Regarding detection of real objects, etc.)
In the above-described embodiment, an example has been described in which planes and objects can be detected based on depth information acquired by the depth sensor of the information acquisition unit 350. However, instead of or in addition to this, detection of objects and the like may be performed by depth estimation by the acquired information analysis unit 362 using other sensors or image recognition processing using machine learning (e.g., using a depth estimation algorithm). For example, it may be possible to identify the type of object appearing in the image acquired by the information acquisition unit 350 or color-code the objects and the like in the image by object region (segmentation/labeling), thereby performing segmentation processing, such as into floors, walls, people, chairs, desks, animals, etc. Based on the segmentation processing, it may be possible to identify the position of each segment or identify the range within the image.

For example, in the case of plane detection at the start of AR shooting, operating the reset icon T2 analyzes the captured image, and if an area that can be labeled as a floor, such as the ground, is identified through segmentation, the area of the ground may be treated as a plane, and the virtual space of FIG. 5(C) may be reconstructed. Furthermore, with regard to the particle display mode exemplified in FIG. 9, the area in which particles can be placed may be determined, for example, based on the tagging determined through segmentation. For example, whether the area is indoors or outdoors may be determined through segmentation, and particles may be drawn outdoors, but not in areas determined to be indoors.

(Regarding the object to be shielded)
In the above-described embodiment, an example has been described in which occlusion processing is possible, in which masking (generating a transparent polygon mesh that becomes a real object) is performed on an object, etc., in real space that is detected by the acquired information analysis unit 362 as a plane, an object, etc., so that a portion of an unreal object, such as the character object C, is not rendered on the display unit of the user terminal 300. The target of the unreal object that is occluded by this occlusion processing may be selectable in advance in accordance with a setting operation by the user. For example, whether or not to perform occlusion processing on a character object by a person (e.g., an entire person or a hand that is part of a person) may be set in advance by a user operation. Alternatively, whether or not to perform occlusion processing on a character object by a plane may be set in advance by a user operation.

(Regarding re-edited data)
In the above-described embodiment, the virtual space identification information stored as the re-editing data in step S109 is information that identifies and enables reproduction of the virtual space in which the real object of FIG. 5C is placed, and does not include information about the character object C at the time of the photographing operation in step S108. However, this is not limiting, and the re-editing data may also include virtual space identification information that includes information about the character object C that can reproduce the composite image of FIG. 5A. For example, when the photographing button S for the first photographing is operated in step S108 of FIG. 10, the captured image of FIG. 5B that serves as the background image and virtual space identification information that identifies and enables reproduction of the virtual space in which the character object C and the real object of FIG. 5C are placed may be stored. The information about the character object C stored as the re-editing data may include information about the type of character object C, its display mode (e.g., pose and facial expression), and the coordinates at which it is placed, so that, for example, the character object C may be re-displayed in the same mode as when it was initially photographed in the re-editing screen of FIG. 6 or the like. Alternatively, a part of the appearance of the character object C at the time of the first photograph may be reproduced by storing any of information such as the type of character object C, the display appearance such as pose and facial expression, the coordinates where the character object C is located, etc. This makes it easy for the user to make fine adjustments based on the previous photograph.

Furthermore, in step S212 of FIG. 11, an example was described in which the data saved by the shooting operation during re-editing is a composite image. However, this is not limited to this. Similarly to step S109 of FIG. 10, during re-editing, the captured image and virtual space identification information, or either one of the pieces of information, may be stored each time a shooting operation is performed. For example, in step S212, in addition to storing the composite image, virtual space identification information including information on unreal objects that identify character objects and decorative objects may be stored as information for reproducing the virtual space after the re-editing operation. As a result, when re-editing is performed again based on the virtual space identification information, in step S204, the character objects and decorative objects are placed in the virtual space in the previous composition mode, and editing can be continued from where it left off last time.

In addition, the re-editing data is not limited to including mask information for real objects. 3D representations that are combined with real objects may be possible by, for example, creating new transparent polygon meshes based on depth information associated with the captured image acquired by the information acquisition unit 350 at the time of the initial shooting, which is included in the virtual space identification information in the re-editing data.

In the above-described embodiment, an example has been described in which AR re-shooting processing is possible by storing re-editing data in the storage unit 320 when AR shooting is performed. This re-shooting (re-editing) re-editing data may be temporarily editable only for the previously shot image, or may be stored so that the user can select one of multiple composite images shot using the AR content of this embodiment as the data to be re-edited. For example, the AR content application may display thumbnails of re-editable data so that the user can select it. Furthermore, an upper limit may be set on the number of re-editable re-editing data that can be stored as options, and some of the data may be deleted due to old data or a deletion operation by the user. Furthermore, the upper limit on the number of options that can be stored may be variable by user operation.

Furthermore, the re-editing screen is not limited to a transition from a menu screen or the like, but may be configured to transition to the re-editing screen immediately after capturing an image by operating the capture button S. For example, for a composite image, options for re-editing operations may be displayed along with an operation menu for posting to SNS or the like.

(Processing that can also be performed by the distribution server 100)
In the above-described embodiment, an example has been described in which the image capture processing shown in FIGS. 10 and 11 is performed in the user terminal 300. However, this is not limiting, and some of the processing (e.g., some of the functions of the image processing unit 361) may be executed in the distribution server 100. For example, information obtainable by the information acquisition unit 350 of the user terminal 300 may be transmitted to the distribution server 100, and shape and coordinate information (information for generating a transparent polygon mesh) corresponding to a real object may be identified based on depth information obtained from the information obtained from the information acquisition unit 350 in the distribution server 100. Based on the information about the identified object, virtual space data in which a polygon mask corresponding to the real object is generated may be generated in the distribution server 100, and virtual space data in which a non-real object is arranged may be distributed to the user terminal 300. Furthermore, during the AR re-shooting process of FIG. 11 , for example, the re-editing data may be stored in the storage unit 120 of the distribution server 100, and in response to a request from the user terminal 300 (e.g., corresponding to step S201), the read re-editing data may be distributed to the user terminal 300 (e.g., corresponding to step S202), and virtual space data updated in response to a re-editing operation on the user terminal 300 (e.g., corresponding to steps S206 and S208 performed in response to operations from the user in steps S205 and S207) may be distributed to the user terminal 300.

(Regarding captured images used in AR photography)
In the above-described embodiment, an example of executing the AR re-capture process based on a captured image captured using AR content has been described. However, this is not limited to this. AR capture may also be possible, in which the image processing unit 361 reads a captured image not captured using AR content and positions a character object C. For example, a process may be performed in which, using image data previously stored in the user terminal 300, objects that may appear in the captured image are identified by image recognition processing using a depth estimation algorithm, and the character object is drawn at a position where geometric consistency with the captured image can be achieved. Note that, in the AR capture process of FIG. 10 and the AR re-capture process of FIG. 11, the acquisition of real-space information by the information acquisition unit 350 (e.g., a camera capable of determining depth with a depth sensor) and the process of combining the real-space image and the unreal object using augmented reality may be performed by different devices. In other words, the terminal performing the process of combining the captured image and 3DCG using augmented reality may not be equipped with the information acquisition unit 350.

(Image editing operations available when re-editing)
In the above-described embodiment, an example has been described in which a reorganization operation can be performed in the AR re-capture process described with reference to FIG. 11 and the like, enabling geometric consistency between the real space and the 3DCG. That is, an example has been described in which editing can be performed so that the 3DCG of a character object or the like is rendered at a position that achieves three-dimensional consistency with the real space in the captured image. In addition, a process for achieving optical consistency may be performed during re-editing. For example, during re-editing, an editing operation may be performed to shade a non-real object so as to correspond to the lighting environment in the captured image of real space based on the re-editing data. For example, the shadow of the character object C may be rendered on the ground.

[Software implementation example]
The various control blocks of the control unit of a computer such as a server or terminal in the above-described embodiments may be implemented by logic circuits (hardware) formed on an integrated circuit (IC chip) or by software using a CPU (Central Processing Unit). When implemented by software using a CPU, the computer equipped with the control unit includes a CPU that executes instructions from a program, which is software that implements each function; a ROM (Read Only Memory) or storage device (collectively referred to as a "recording medium") in which the program and various data are recorded so as to be readable by the computer (or CPU); and a RAM (Random Access Memory) in which the program is deployed. The object of the present invention is achieved when the computer (or CPU) reads and executes the program from the recording medium. The recording medium may be a "non-transitory tangible medium," such as a tape, disk, card, semiconductor memory, or programmable logic circuit. The program may also be supplied to the computer via any transmission medium capable of transmitting the program (such as a communication network or broadcast waves). Note that one aspect of the present invention may also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission.

The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of the present invention is indicated by the claims, not the above description, and is intended to include all modifications that are equivalent in meaning to and within the scope of the claims.

REFERENCE SIGNS LIST 1 Communication system, 2 Network, 100 Distribution server, 300 User terminal, 110 Communication unit, 120 Storage unit, 310 Communication unit, 320 Storage unit, 330 Operation unit, 340 Output unit, 350 Information acquisition unit, 360 Control unit

Claims (6)

a computer including an information acquisition unit that captures an image according to a shooting state of an imaging unit at the time of shooting and acquires depth information that enables identification of a depth position of each real object that may be included in the image;
a means for identifying a real object that may be included in the captured image and identifying a depth position of the real object in the virtual space based on depth information that may be identified from the information acquired by the information acquisition unit;
a means for placing real objects corresponding to the real objects identified in the virtual space at depth positions of the real objects, and generating a virtual space in which specific objects are placed in response to an operation by a user;
a means for combining the captured image with an object placed in the virtual space generated by the generating means, and displaying a first composite image in which a portion of the specific object that overlaps with a real object located in front of the specific object is not displayed and a non-overlapping portion can be superimposed on the captured image;
a means for storing a captured image when a photographing operation is received from a user in a first composite image display state in which a first composite image is displayed by the display means , and virtual space identification information for identifying a virtual space generated by the generating means when the photographing operation is received, the virtual space also identifying a state of a specific object placed in the virtual space ;
regenerating the virtual space in the first composite image display state based on the virtual space identification information stored by the storing means, and updating the virtual space so that the appearance of a specific object placed in the virtual space corresponds to a change operation from the user and is different from the appearance in the first composite image display state ;
the aspect corresponding to the change operation includes at least one of an aspect in which a placement position of the specific object in the virtual space is changed, an aspect in which the specific object placed in the virtual space is changed to a specific object of a different type, and an aspect in which a display aspect of the specific object placed in the virtual space is changed;
The displaying means re-displays a composite image of the regenerated virtual space in the first composite image display state and the captured image stored by the storing means, and further, after the change operation, displays a composite image obtained by combining the captured image stored by the storing means with the object placed in the virtual space being updated by the updating means as a second composite image , regardless of the shooting state of the imaging unit at the timing of displaying the second composite image. the specific object includes a character object with a predetermined motion;
2. The program according to claim 1, wherein when the character object is arranged as the specific object, the displaying means is capable of displaying the character object by moving it with a predetermined motion. the specific objects include character objects and decorative objects with animation;
The program according to claim 1 , wherein when the character object and the decorative object are arranged as the specific object, the display means is capable of displaying the character object and also capable of moving and displaying the decorative object. The computer
4. The program according to claim 2 , further comprising a function of changing the movement speed of the specific object that is moved and displayed by said display means in response to an operation by a user. A method for controlling a computer including an information acquisition unit that captures an image according to a shooting state of an imaging unit at the time of capturing the image and acquires depth information that enables identification of a depth position of each real object that may be included in the image,
a step of identifying a real object that may be included in the captured image and identifying a depth position of the real object in the virtual space based on depth information that may be identified from the information acquired by the information acquisition unit;
a step of placing real objects corresponding to the identified real objects at depth positions of the identified real objects in the virtual space, and generating a virtual space in which specific objects are placed in response to an operation by a user;
a step of combining the captured image with an object placed in the virtual space generated by the generating step, and displaying a first composite image in which a portion of the specific object that overlaps with a real object located in front of the specific object is not displayed and a non-overlapping portion can be superimposed on the captured image;
a step of storing a captured image when a photographing operation is received from a user in a first composite image display state in which the first composite image is displayed by the displaying step , and virtual space identification information for identifying the virtual space generated by the generating step when the photographing operation is received, the virtual space identification information also identifying the state of a specific object placed in the virtual space ;
a step of regenerating the virtual space in the first composite image display state based on the virtual space identification information stored in the storing step, and updating the virtual space so that a state of a specific object to be placed in the virtual space corresponds to a change operation from a user and is different from that in the first composite image display state ,
the aspect corresponding to the change operation includes at least one of an aspect in which a placement position of the specific object in the virtual space is changed, an aspect in which the specific object placed in the virtual space is changed to a specific object of a different type, and an aspect in which a display aspect of the specific object placed in the virtual space is changed;
The displaying step re-displays a composite image of the regenerated virtual space in the first composite image display state and the captured image stored in the storing step, and further, after the change operation, displays a composite image obtained by combining the captured image stored in the storing step with the object placed in the virtual space updated in the updating step as a second composite image , regardless of the shooting state of the imaging unit at the timing of displaying the second composite image. A system having a computer including an information acquisition unit that captures an image according to a shooting state of an imaging unit at the time of capturing an image and acquires depth information that enables identification of a depth position of each real object that may be included in the image,
a means for identifying a real object that may be included in the captured image and identifying a depth position of the real object in the virtual space based on depth information that may be identified from the information acquired by the information acquisition unit;
a means for placing real objects corresponding to the real objects identified in the virtual space at depth positions of the real objects, and generating a virtual space in which specific objects are placed in response to an operation by a user;
a means for combining the captured image with an object placed in the virtual space generated by the generating means, and displaying a first composite image in which a portion of the specific object that overlaps with a real object located in front of the specific object is not displayed and a non-overlapping portion can be superimposed on the captured image;
a means for storing a captured image when a photographing operation is received from a user in a first composite image display state in which a first composite image is displayed by the display means , and virtual space identification information for identifying a virtual space generated by the generating means when the photographing operation is received, the virtual space also identifying a state of a specific object placed in the virtual space ;
a means for regenerating the virtual space in the first composite image display state based on the virtual space identification information stored by the storing means, and updating the virtual space so that a state of a specific object to be placed in the virtual space corresponds to a change operation by a user and is different from that in the first composite image display state ;
the aspect corresponding to the change operation includes at least one of an aspect in which a placement position of the specific object in the virtual space is changed, an aspect in which the specific object placed in the virtual space is changed to a specific object of a different type, and an aspect in which a display aspect of the specific object placed in the virtual space is changed;
The display means re-displays a composite image of the virtual space in the regenerated first composite image display state and the captured image stored by the storage means, and further, after the change operation, displays a composite image obtained by combining the captured image stored by the storage means with the object placed in the virtual space being updated by the update means as a second composite image , regardless of the shooting state of the imaging unit at the timing of displaying the second composite image.