JP7540905B2 - Information processing system and program - Google Patents

Description

The present invention relates to an information processing system and a program.

For example, Patent Document 1 discloses an image processing device that includes an image acquisition unit that acquires an image showing a real object having a primary recognition target and a secondary recognition target, and a control unit that sets an augmented reality space associated with the image based on image recognition of the primary recognition target, and determines an augmented reality object to be placed in the augmented reality space depending on the secondary recognition target recognized based on image recognition of the primary recognition target.

JP 2015-1875 A

A possible embodiment is one in which a beverage container is photographed using a terminal device having a photographing function, and processing associated with the image of the beverage container obtained by photographing is performed by this terminal device.
Here, the image of the beverage container is basically an image of the beverage container itself, and there is little variation in the image, so the processing performed by the terminal device is likely to become monotonous.
An object of the present invention is to provide a more versatile process that can be performed by a terminal device having a function for photographing beverage containers.

The information processing system to which the present invention is applied is an information processing system that includes an action recognition means that analyzes an image obtained by photographing an action performed by a user with respect to a beverage container containing a beverage, the image being obtained by a terminal device having a photographing function, and recognizes the action performed by the user, and a decision means that makes a decision regarding a process to be performed by the terminal device based on the action recognized by the action recognition means.

Here, the decision means may be characterized in making a decision regarding a display process to be performed on the terminal device based on the action recognized by the action recognition means.
The determination means may be characterized in making a determination regarding an image to be displayed on the terminal device together with the beverage container photographed by the terminal device and displayed on the terminal device.
The determination means may be characterized in making a determination to display the image in a state in which the image is aligned with the longitudinal direction of the beverage container displayed on the terminal device.
The determination means may determine that the image is to be displayed in a position facing the lid of the beverage container displayed on the terminal device.
The determination means may determine that the image is to be displayed in a position facing an outer circumferential surface of the beverage container displayed on the terminal device.
The determination means can also be characterized in making a determination that the image is displayed at an opposing position on the outer peripheral surface of the beverage container and that the image is displayed in a state where it is aligned in the axial direction of the beverage container.
The determining means may determine that the image is to be moved around an axis of the beverage container displayed on the terminal device.
The decision means may be characterized in that, when the action recognized by the action recognition means is a predetermined specific action, it makes a decision to display a predetermined specific image on the terminal device.
The decision means may also be characterized in making a decision to display the specific image together with a displayed beverage container, which is the beverage container photographed by the terminal device and displayed on the terminal device.
The decision means may be characterized in that, when the action recognized by the action recognition means is an action of opening the beverage container, it makes a decision to display the specific image together with the displayed beverage container.
The decision means can be characterized in that when the action recognized by the action recognition means is an action of pouring the contents of the beverage container into another container, it makes a decision to display the specific image together with the displayed beverage container.
The determination means may make a decision regarding a sound to be output from the terminal device based on the action recognized by the action recognition means.
The decision means can be characterized in that, when the action recognized by the action recognition means is the action of opening the beverage container, it makes a decision to output from the terminal device the sound made by the beverage container when the beverage container is opened.
The decision means can be characterized in that, when the action detected by the action detection means is an action of crushing the beverage container, it makes a decision to output from the terminal device the sound made by the beverage container when the beverage container is crushed.
In addition, the decision means can be characterized in that, when the action recognized by the action recognition means is an action of pouring the contents of the beverage container into another container, it makes a decision to output from the terminal device the sound produced when the contents are poured.
The decision means can be characterized in that, when the action detected by the action detection means is an action indicating that the beverage container has been finished, it makes a decision to output information from the terminal device indicating that the beverage container should be thrown away in a trash can.

In addition, when the present invention is considered as a program, the program to which the present invention is applied is a program for causing a computer to realize an action recognition function that analyzes an image obtained by photographing an action performed by a user with respect to a beverage container containing a beverage, the image being obtained by a terminal device having a photographing function, and recognizes the action performed by the user, and a decision function that makes a decision regarding a process to be performed by the terminal device based on the action recognized by the action recognition function.

The present invention makes it possible to make the processing performed by a terminal device that has the function of photographing beverage containers more versatile.

1 is an overall view of an information processing system. 1A to 1C are diagrams illustrating a can body. FIG. 1 is a diagram illustrating an example of a hardware configuration of a computer suitable for use as a terminal device and a server device. FIG. 2 is a functional block diagram of the server device. FIG. 2 is a diagram showing the state of a terminal device. FIG. 2 is a diagram showing the state of a terminal device. FIG. 2 is a diagram showing the state of a terminal device. FIG. 2 is a diagram showing the state of a terminal device. FIG. 2 is a diagram showing the state of a terminal device.

Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.
FIG. 1 is an overall view of an information processing system 1 according to the present embodiment.
The information processing system 1 includes a terminal device 10 owned by a user, and a server device 30 to which the terminal device 10 is connected. The terminal device 10 and the server device 30 are capable of transmitting and receiving information to and from each other via a network 40 (communication line).

Network 40 is not particularly limited as long as it is a communications network used for data communications between each device, and may be, for example, a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, etc. The communications lines used for data communications may be either wired or wireless, or a combination of these may be used. In addition, each device may be connected via multiple networks or communications lines using relay devices such as gateway devices or routers.

In this embodiment, a can body 50 is placed in front of the terminal device 10. Furthermore, the user performs an action on the can body 50, such as opening the can body 50.
At this time, in this embodiment, the can body 50 and the user's action are photographed by the terminal device 10, and the terminal device 10 obtains an image (video) of the can body 50 and the user's action.
More specifically, the terminal device 10 includes a camera 10C and has a photographing function, and the terminal device 10 obtains an image of the can body 50 and the user's actions.

When the terminal device 10 captures an image of the can body 50 and the user's action, for example, a user other than the user holds the terminal device 10 and captures an image of the can body 50 and the user's action.
At this time, in this embodiment, as described below, the terminal device 10 outputs an image based on the image data transmitted from the server device 30, and a sound based on the sound data transmitted from the server device 30.

In addition, for example, a user may perform an action on the can body 50 with one hand, hold the terminal device 10 with the other hand, and photograph the can body 50 and the action with the other hand.
In this case as well, in this embodiment, an image based on the image data transmitted from the server device 30 and a sound based on the sound data transmitted from the server device 30 are output by the terminal device 10 .

In this embodiment, the action of the user with respect to the can body 50 is photographed together with the can body 50, and an image of the can body 50 and the user's action is obtained by the terminal device 10.
The image obtained by the terminal device 10 is then transmitted to the server device 30. The server device 30 determines, based on the image, the processing (output of image and sound) to be performed by the terminal device 10. The terminal device 10 then outputs the image and sound based on this determination.

[Can body 50]
FIG. 2 is a diagram illustrating the can body 50. As shown in FIG.
Fig. 2(A) is a perspective view of the can body 50 when viewed from a specific direction. Fig. 2(B) is a perspective view of the can body 50 when viewed from a direction different from the specific direction of Fig. 2(A). And Fig. 2(C) is an explanatory diagram of an image printed on the can body 50.

As shown in Figures 2(A) and 2(B), the can body 50 has a tubular portion 510 formed in a cylindrical shape, a disk-shaped lid member 520 that covers an opening located at one end of the can body 50 in the axial direction, and a bottom portion 530 provided on the opposite side of the lid member 520 in the axial direction of the tubular portion 510.
The can body 50 is an example of a beverage container, and each can body 50 contains a beverage such as juice, alcohol, tea, etc. The can body 50 is made of a metal material such as aluminum.
Although the tubular portion 510 in this embodiment is a cylinder formed in a circular shape, it may be formed in a polygonal shape.

As shown in FIG. 2(C), various information is printed on the cylindrical portion 510 of the can body 50, including the beverage manufacturer's name 51m, product graphic 51g, product name 51p, alcohol mark 51a, recycle identification mark 51r, barcode 51b, can manufacturer logo 51s, product text 51t, nutritional information 51n, warning text 51c, and product description section 51e.

The beverage manufacturer name 51m is information indicating the name of the company that fills the can body 50 with a beverage and manufactures it, or sells the can body 50 filled with a beverage (hereinafter referred to as the product). The product graphic 51g is information indicating the product's logo. The product name 51p is information indicating the product name. The alcohol mark 51a is information indicating that the product is an alcoholic beverage. The recycle identification mark 51r is information that serves as a marker when the can body 50 is reused as a resource. The barcode 51b is encoded information such as the name of the company that manufactures or sells the product, and the type of product.

The can manufacturer logo 51s is information showing the logo of the company that manufactured the can body 50 itself. The product text 51t is information showing various information about the product in text form. The nutritional information 51n is information showing the nutritional content of the product. The warning text 51c is information showing precautions for handling the product.
The product description section 51e includes text 51e1 that explains the product, and explanatory figures 51e2 that graphically represent the image of the product.

In addition, in this embodiment, the can body 50 is exemplified as a beverage container, but the beverage container is not limited to the can body 50.
Another example of a beverage container is a bottle can made of metal, shaped like a bottle, and equipped with a cap.
Other examples of beverage containers include plastic bottles with caps made of synthetic resin such as polyethylene terephthalate, and glass bottles made of glass.
Even in these cases, the various types of information described above may be provided on the surface of the bottle, on the packaging material that covers the bottle, on a label affixed to the bottle, or the like.

[Terminal device 10]
Examples of the terminal device 10 (see FIG. 1) include a mobile phone such as a smartphone, and a tablet-type mobile terminal.
The terminal device 10 is provided with a display device 100 configured with a touch panel or the like, and various information is displayed on this display device 100. In addition, this display device 100 displays images such as still images and videos.
Note that, although FIG. 1 illustrates an example in which only one terminal device 10 is provided, in reality, the number of terminal devices 10 installed corresponds to the number of users, and multiple terminal devices 10 are provided.

The terminal device 10 also includes a camera 10C. In this embodiment, the camera 10C captures an image of the can body 50 and an action performed by the user on the can body 50. The terminal device 10 also includes a speaker 10S that outputs sound.
In the terminal device 10 , the user inputs information to the terminal device 10 by touching the display screen 100</b>A of the display device 100 .
Furthermore, images and information received from the server device 30 or other server devices are displayed on the display device 100 of the terminal device 10. Furthermore, when an image is captured by the camera 10C, the image captured by the camera 10C is displayed on the display device 100 of the terminal device 10.

Furthermore, in this embodiment, when a user being photographed by camera 10C performs a specific action, an image transmitted from server device 30 is displayed on display device 100 of terminal device 10 (details will be described later).
In addition, in this embodiment, when a user being photographed by camera 10C performs a specific action, a sound based on the sound data transmitted from server device 30 is output from speaker 10S of terminal device 10 (details will be described later).

FIG. 3 is a diagram showing an example of a hardware configuration of a computer suitable for applying the terminal device 10 and the server device 30. As shown in FIG.
As shown in FIG. 3, the terminal device 10 and the server device 30 each include a CPU (Central Processing Unit) 101, which is a computing means, a ROM (Read Only Memory) 102, which is a storage area for storing programs such as a BIOS (Basic Input Output System), and a RAM (Random Access Memory) 103, which is an execution area for the programs.

The terminal device 10 and the server device 30 each have a hard disk drive (HDD) 104. The HDD 104 stores various programs such as an operating system (OS) and applications, input data for the various programs, output data from the various programs, etc. The HDD 104 also stores image data that is the source of images displayed on the terminal device 10, and sound data that is the source of sounds output from a speaker 10S of the terminal device 10.
It should be noted that the terminal device 10 and the server device 30 may each be provided with a plurality of CPUs, and the plurality of CPUs may cooperate to perform arithmetic processing.

Furthermore, the terminal device 10 and the server device 30 each include a communication interface (communication I/F) 105 for communicating with the outside, a display device 100, and an input device 107 such as a keyboard, a mouse, or a touch panel.
Here, the terminal device 10 is provided with a touch panel, and in the terminal device 10, the display device 100 and the input device 107 are configured by this touch panel.
In the server device 30, the display device 100 is configured, for example, by a liquid crystal display, and the input device is configured, for example, by a mouse.
Although not shown in FIG. 3, as described above, the terminal device 10 is provided with a speaker 10S that outputs sound.

Fig. 4 is a functional block diagram of the server device 30. Note that Fig. 4 shows only the functional units related to the processes performed by the terminal device 10, among the multiple functional units included in the server device 30.
As shown in FIG. 4 , the server device 30 includes an action grasping unit 31 , a determining unit 32 , and an output information generating unit 33 .

The action grasping unit 31, which is an example of an action grasping means, grasps the action performed by the user with respect to the can body 50. More specifically, the action grasping unit 31 analyzes images (videos) obtained by the terminal device 10 and transmitted from the terminal device 10 to the server device 30, and grasps the action performed by the user with respect to the can body 50.
The decision unit 32 , which is an example of a decision means, makes a decision regarding a process to be performed by the terminal device 10 based on the user's action grasped by the action grasping unit 31 .

The output information generating unit 33 generates information to be output to the terminal device 10 based on the content determined by the determining unit 32 .
In other words, the output information generating unit 33 generates image data and sound data to be transmitted from the server device 30 to the terminal device 10 based on the content determined by the determining unit 32 .
In other words, the output information generating unit 33 generates image data and sound data that are the source of images and sounds to be outputted from the terminal device 10 based on the contents determined by the determining unit 32 .

The action grasping unit 31, the decision unit 32, and the output information generation unit 33 of the server device 30 are realized by programs stored in the ROM 102, the HDD 104, etc. provided in the server device 30 being read into the RAM 103 and executed by the CPU 101.
The program for realizing the action grasping unit 31, the decision unit 32, and the output information generating unit 33 in the server device 30 (or in a computer provided therein) may be provided to the server device 30, for example, using a communication means, or may be stored in various recording media (for example, a USB (Universal Serial Bus) memory, an EEPROM (Electrically Erasable and Programmable ROM), etc.) and provided to the server device 30.

[First Processing Example]
The decision unit 32 in this embodiment makes a decision regarding, for example, a display process to be performed in the terminal device 10 based on the action recognized by the action recognition unit 31 .
More specifically, the decision unit 32 makes a decision regarding, for example, an image to be displayed on the terminal device 10 based on the action recognized by the action recognition unit 31 .
More specifically, when the action recognized by the action recognition unit 31 is a specific action, the determination unit 32 makes a determination to display a specific image on the terminal device 10, for example.

More specifically, when the user's action recognized by the action recognition unit 31 is a specific action such as opening the can 50, pouring the beverage inside the can 50 into another container, drinking the beverage inside the can 50, or crushing the can 50, the decision unit 32 makes a decision to display a specific image on the terminal device 10.

More specifically, the determination unit 32 makes a determination to display, on the terminal device 10, a specific image such as a specific character or a specific motif, or a specific image composed of text information, for example.
Here, the "specific image" is not particularly limited, and any image that can be displayed on the terminal device 10 and whose existence can be recognized by the user is considered to be a "specific image."

When the decision unit 32 makes a decision to display a specific image, image data corresponding to this specific image is transmitted from the server device 30 to the terminal device 10, and an image based on this image data (specific image) is displayed on the display device 100 of the terminal device 10.
More specifically, in this embodiment, when the decision unit 32 decides to display a specific image on the terminal device 10, the output information generation unit 33 generates image data representing this specific image.

More specifically, in this embodiment, image data representing a specific image is registered in HDD 104 of server device 30, and output information generation unit 33 generates image data representing the specific image by reading this image data from HDD 104.
In this embodiment, the image data generated by the output information generating unit 33 is transmitted to the terminal device 10, and an image based on this image data is displayed on the display device 100 of the terminal device 10.

As another process, the decision unit 32 makes a decision regarding, for example, the can body 50 (hereinafter referred to as the "displayed can body 50A") displayed on the terminal device 10 together with the image (hereinafter referred to as the "displayed image 90") to be displayed on the terminal device 10.
In other words, the decision unit 32 makes a decision regarding the display image 90, which is an image photographed by the terminal device 10 and displayed together with the display can body 50A displayed on the terminal device 10.

More specifically, in this embodiment, a so-called augmented reality display can be performed on the terminal device 10, and an image of the can body 50 as a real image is displayed on the terminal device 10. The determination unit 32 determines a display image 90 to be displayed on the terminal device 10 together with this real image.
In other words, the determination unit 32 makes a determination regarding the display image 90 to be displayed on the terminal device 10 together with the display can body 50A displayed on the terminal device 10 .
More specifically, the decision unit 32 makes a decision regarding the display image 90 to be additionally displayed on the terminal device 10 when a can body 50 is located behind the terminal device 10 and the can body 50 is displayed on the terminal device 10.

More specifically, when the action recognized by the action recognition unit 31 is a specific action, the decision unit 32 makes a decision to display, for example, the display image 90 at a position facing the lid member 520 of the display can body 50A.
In this case, the output information generation unit 33 reads out, for example, a display image 90 representing a character from the HDD 104, and rotates (tilts) the display image 90 so that the inclination of the display image 90 matches the inclination of the display can body 50A.
Then, image data showing the display image 90 after rotation is transmitted from the server device 30 to the terminal device 10 .

In this case, the terminal device 10 displays this display image 90 together with the display can body 50A (an actual image of the can body 50) as shown in FIG. 5 (a diagram showing the state of the terminal device 10).
More specifically, the terminal device 10 places the display image 90 at a position facing the lid member 520 of the display can 50A, and displays the display image 90.

More specifically, the terminal device 10 determines the top and bottom of the display can body 50A based on information attached to the surface of the can body 50, such as the product graphic 51g (see FIG. 2), and recognizes the lid member 520 of the display can body 50A. Then, the terminal device 10 displays an image (display image 90) based on image data at a position facing the lid member 520.
As a result, in addition to the display can body 50A, an image different from the can body 50, such as a character display image 90, is displayed on the terminal device 10.

After a specific action is recognized by the action recognition unit 31, the angle, position, etc. of the can body 50 may change sequentially. In this case, the terminal device 10 changes the inclination and position of the display image 90 displayed on the display screen according to the angle and position of the can body 50.
More specifically, the terminal device 10 keeps track of the state of the display can 50A, and when the angle or position of the display can 50A changes, it changes the inclination or position of the display image 90 in response to this change.
This maintains the display image 90 positioned opposite the lid member 520 of the display can 50A.

As another example, when the action recognized by the action recognition unit 31 is a specific action, the decision unit 32 makes a decision to display the display image 90 at a position facing the outer peripheral surface 50X of the display can body 50A.
In this case, the output information generation unit 33, as described above, for example, reads out a display image 90 corresponding to the character from the HDD 104, and rotates the display image 90, for example, so that the display image 90 is aligned in a direction perpendicular to the longitudinal direction of the display can body 50A.
Then, in this embodiment, image data showing the display image 90 after rotation is transmitted from the server device 30 to the terminal device 10 .

Then, in the terminal device 10, as described above, this display image 90 is displayed together with the display can body 50A, and as shown in Figure 6 (a diagram showing the state of the terminal device 10), the display image 90 is displayed at a position opposite the outer peripheral surface 50X of the display can body 50A.
More specifically, when the terminal device 10 receives image data indicating the display image 90, the terminal device 10 displays the display image 90 based on this image data at a position facing the outer circumferential surface 50X of the display can body 50A, as shown in FIG. 6 .

More specifically, the terminal device 10 recognizes the top and bottom of the display can body 50A based on the information attached to the surface of the can body 50, and further recognizes the outer peripheral surface 50X of the display can body 50A.
The terminal device 10 then displays a display image 90 based on the received image data at a position facing the recognized outer circumferential surface 50X.

As described above, the angle and position of the display can 50A may change. In this case, the terminal device 10 changes the inclination and position of the display image 90 according to the angle and position of the display can 50A.
As a result, even if the angle or position of the display can body 50A changes, the display image 90 moves in accordance with the display can body 50A, and the display image 90 is maintained at a position opposite the outer peripheral surface 50X of the display can body 50A.

As another example, when the action recognized by the action recognition unit 31 is a specific action, the determination unit 32 makes a determination to display the display image 90 at a position facing the outer peripheral surface 50X of the display can body 50A and in a state in which the display image 90 is aligned in the longitudinal direction (axial direction) of the display can body 50A.

In this case, the output information generation unit 33, as described above, for example, reads out a display image 90 corresponding to the character from the HDD 104, and rotates the display image 90 so that the display image 90 is aligned along the longitudinal direction of the display can body 50A.
In other words, the output information generating unit 33 rotates (tilts) the display image 90 so that the inclination of the display image 90 matches the inclination of the display can body 50A.
In this embodiment, image data showing the display image 90 after rotation is transmitted from the server device 30 to the terminal device 10 .

7 (a diagram showing the state of the terminal device 10), the display image 90 is displayed together with the display can body 50A, and the display image 90 is also displayed at a position facing the outer circumferential surface 50X of the display can body 50A. The display image 90 is displayed along the longitudinal direction of the display can body 50A.
More specifically, when the terminal device 10 receives image data indicating the display image 90, the terminal device 10 displays the display image 90, which is an image based on this image data, at a position opposite the outer peripheral surface 50X of the display can body 50A, as shown in FIG. 7.

More specifically, the terminal device 10 recognizes the top and bottom of the display can body 50A based on the information attached to the surface of the can body 50, as described above, and further recognizes the outer peripheral surface 50X of the display can body 50A.
Then, the terminal device 10 displays the display image 90 at a position facing the recognized outer circumferential surface 50X.
Here, as described above, the angle and position of the display can body 50A may change, in which case the terminal device 10 changes the inclination and position of the display image 90 according to the angle and position of the display can body 50A.
In the example shown in Figure 7, an example was described in which the display image 90 is displayed to the side of the display can body 50A. However, as shown in Figure 8 (a diagram showing the state of the terminal device 10), the display image 90 may also be displayed in front of the display can body 50A.

As another example, when the action recognized by the action recognition unit 31 is a specific action, the determination unit 32 makes a determination to move the display image 90 around the axis of the display can body 50A.
In this case, the output information generating unit 33 reads the display image 90 from the HDD 104, for example, as described above, and rotates the display image 90 so that the inclination of the display image 90 matches the inclination of the display can body 50A.
Also in this case, image data showing the display image 90 after rotation is transmitted from the server device 30 to the terminal device 10 .

Then, the terminal device 10 displays the display image 90 together with the display can body 50A, as shown in Figure 9 (a diagram showing the state of the terminal device 10), and further moves the display image 90 so that the display image 90 moves around the axis of the display can body 50A.
More specifically, the terminal device 10 recognizes the top and bottom of the indicating can body 50A based on the information attached to the surface of the can body 50, as described above, and further recognizes the axis of the indicating can body 50A.
More specifically, the terminal device 10 recognizes, as the axis, a straight line passing through the radial center of the lid member 520 and the radial center of the bottom portion 530. Furthermore, the terminal device 10 recognizes the outer peripheral surface 50X of the display can body 50A.

Then, the terminal device 10 displays a display image 90 based on the received image data at a position opposite the recognized outer peripheral surface 50X, and further moves the display image 90 so that the display image 90 moves around the axis of the display can body 50A.
As described above, the angle and position of the can body 50 may change. In this case, the terminal device 10 changes the inclination and position of the display image 90 according to the angle and position of the can body 50.

Here, in this embodiment, when the action grasped by the action grasping unit 31 is a predetermined specific action, the decision unit 32 makes a decision to display the display image 90 on the terminal device 10. More specifically, the decision unit 32 makes a decision to display the predetermined specific display image 90 on the terminal device 10.

In the examples shown in Figures 5 to 9, the display image 90 is displayed together with the display can body 50A. However, the display of the display can body 50A is not required, and only the display image 90 may be displayed.
When only the display image 90 is to be displayed, for example, the display can body 50A that had been displayed up until that point is made non-displayable, in other words, the image obtained by the camera 10C is made non-displayable, and only the display image 90 transmitted from the server device 30 is displayed.

Here, an example of the above-mentioned predetermined specific action is the action of opening the can body 50 .
When the action recognized by the action recognition unit 31 is the action of opening the can body 50, the determination unit 32 makes a decision to cause a specific display image 90 to be displayed on the terminal device 10, as described above.

Here, the action recognition unit 31 analyzes the image (video) captured by the terminal device 10 to recognize the user's action of opening the can body 50 .
The action recognition unit 31 recognizes, for example, based on the movement of the user's hand, that the user has performed an action of opening the can body 50. In addition, the action recognition unit 31 determines that the user has performed an action of opening the can body 50 when, for example, a tab (not shown) provided on the can body 50 is in an upright state, and recognizes this action of opening the can body 50.
In addition, if the beverage container is not a can body 50 but a beverage container with a cap, the action recognition unit 31 determines that the user has performed the action of opening the beverage container when, for example, the cap is detached from the main body of the beverage container, and recognizes this action of opening the beverage container.

Further, other examples of predetermined specific actions include pouring the beverage in the can 50 into another container.
The action recognition unit 31 analyzes the image captured by the terminal device 10 and recognizes the action of pouring the beverage contained in the can 50 into another container.
More specifically, the action understanding unit 31 analyzes the image obtained by the terminal device 10, and, for example, when the can body 50 held by the user is tilted and the contents are discharged from the can body 50, determines that the user has poured the contents of the can body 50 into another container, and understands this pouring action.

Furthermore, another example of a predetermined specific action is drinking a beverage inside.
The action recognition unit 31 analyzes the image obtained by the terminal device 10, and, for example, when the can body 50 held by the user is tilted and the user's mouth is in contact with the can body 50, determines that the user has performed the action of drinking the beverage from the can body 50, and recognizes this drinking action.

Further, other examples of the predetermined specific actions include the action of crushing the can body 50 .
The action understanding unit 31 analyzes the image obtained by photographing with the terminal device 10, and, for example, when the side (outer surface 50X) of the can body 50 held by the user becomes dented, determines that the user has performed the action of crushing the can body 50, and understands this crushing action.

[Second Processing Example]
Additionally, the determination unit 32 may make a determination regarding the sound to be output from the terminal device 10 based on the action recognized by the action recognition unit 31 .
More specifically, for example, when the action recognized by the action recognition unit 31 is the action of opening the can body 50, the decision unit 32 makes a decision to output from the terminal device 10 the sound emitted by the can body 50 when the can body 50 is opened.

When the beverage contained in the can 50 is a carbonated beverage, the can 50 often emits a sound of gas escaping from inside the can 50 when the can 50 is opened.
When the action recognized by the action recognition unit 31 is the action of opening the can body 50 , the decision unit 32 makes a decision to output the sound of gas escaping from inside the can body 50 from the terminal device 10 .

More specifically, when performing this processing, the action recognition unit 31 determines that the user has performed the action of opening the can body 50 when, for example, the user places his/her finger on a tab provided on the can body 50 and begins to lift up the tab, and recognizes this opening action.
In this case, the decision unit 32 makes a decision to output the sound of gas escaping from inside the can body 50 from the terminal device 10.

In this case, sound data of the sound of gas escaping, which is stored in advance in HDD 104 of server device 30 , is transmitted from server device 30 to terminal device 10 .
When the terminal device 10 receives this sound data, a sound based on this sound data is output from the speaker 10S of the terminal device 10.

In this process, not only is the actual sound emitted by the can body 50 emitted from the can body 50, but the sound emitted by the can body 50 when the can body 50 is opened is also output from the terminal device 10.
More specifically, in this example, the sound of gas escaping from the can body 50 is output from the speaker 10S of the terminal device 10.
The sound output from the speaker 10S of the terminal device 10 may be a recording of the sound of gas actually blowing out of the can body 50, or may be an artificial sound that imitates the sound that the can body 50 makes when gas blows out of the can body 50. In other words, the sound output from the speaker 10S of the terminal device 10 may be a sound effect.

As another example, for example, when the action recognized by the action recognition unit 31 is an action of crushing the can body 50, the decision unit 32 makes a decision to output from the terminal device 10 the sound made by the can body 50 when the can body 50 is crushed.
Here, as described above, the action understanding unit 31 analyzes the video obtained by the terminal device 10, and if it determines that the user has performed the action of crushing the can body 50, it understands this action of crushing the can body 50.

More specifically, the action understanding unit 31 analyzes the image obtained by photographing using the terminal device 10, and when the side (outer surface 50X) of the can body 50 held by the user begins to dent, it determines that the user has performed the action of crushing the can body 50, and understands this crushing action.
In this case, the decision unit 32 makes a decision to output from the terminal device 10 the sound made by the can body 50 when the can body 50 is crushed.

In this case, sound data indicating the sound made when the can body 50 is crushed is transmitted from the server device 30 to the terminal device 10, and a sound based on this sound data is output from the speaker 10S of the terminal device 10.
In this case, not only is the actual sound emitted by the can body 50 emitted from the can body 50, but the sound emitted by the can body 50 when it is crushed is also output from the speaker 10S of the terminal device 10.
As described above, the sound output from the speaker 10S of the terminal device 10 may be a recording of an actual sound, or an artificial sound that imitates an actual sound.

As another example, when the action recognized by the action recognition unit 31 is an action of pouring the beverage in the can body 50 into another container, the decision unit 32 makes a decision to output from the terminal device 10 the sound generated when the beverage is poured.
When performing this process, similarly to the above, the action recognition unit 31 analyzes the video obtained by the terminal device 10 and determines, for example, whether or not the dispensing of beverage from the can 50 has started.
Then, when the action recognition unit 31 determines that the beverage has started to be dispensed from the can body 50, it recognizes that the user is pouring the beverage.

In this case, the decision unit 32 makes a decision to output from the terminal device 10 the sound that is generated when the beverage in the can 50 is poured.
In this case, as described above, sound data indicating the sound of the beverage being poured is transmitted from server device 30 to terminal device 10, and the sound of the beverage being poured is output from speaker 10S of terminal device 10.

As a result, in this case too, not only is the actual sound produced by pouring a beverage produced, but the sound produced when the beverage is being poured is output from the speaker 10S of the terminal device 10.
In this case as well, the sound output from the speaker 10S of the terminal device 10 may be a recording of an actual sound, or may be an artificial sound that imitates an actual sound.

[Third Processing Example]
In addition, when the action recognized by the action recognition unit 31 indicates the end of the beverage, the decision unit 32 makes a decision to output information indicating an instruction to throw away the can body 50 in a trash can from the terminal device 10.
When performing this processing, the action understanding unit 31 analyzes the image obtained by the terminal device 10 and determines, for example, whether the can body 50 held by the user is in contact with the user's mouth, the bottom 530 of the can body 50 is positioned above the lid member 520 of the can body 50, and the inclination of the can body 50 relative to the horizontal direction exceeds a predetermined angle.

When the can body 50 is in this state, the behavior recognition unit 31 determines that the user has finished drinking the beverage, and recognizes the behavior indicating that the user has finished drinking the beverage.
Next, in this case, the decision unit 32 makes a decision to cause the terminal device 10 to output information indicating an instruction to throw away the can body 50 in a trash can, as described above.
In this case, information instructing the user to throw the beverage container in a trash can is transmitted from the server device 30 to the terminal device 10 .

Specifically, for example, text information such as "Please throw the can in the trash after you finish drinking" is transmitted from the server device 30 to the terminal device 10.
In this case, this information is displayed on the display device 100 of the terminal device 10 or output from the speaker 10S of the terminal device 10.

〔others〕
In the above description, the server device 30 determines the process to be performed by the terminal device 10. However, the terminal device 10 may determine the process to be performed by the server device 30.
In other words, the action grasping unit 31 , the determination unit 32 , and the output information generating unit 33 may be provided in the terminal device 10 .

In this case, once an image of the can body 50 and the user's actions is obtained by the terminal device 10, the image is analyzed by the terminal device 10, and then a decision is made by the terminal device 10 regarding the processing to be performed by the terminal device 10.
Based on this decision, an image pre-stored in the terminal device 10 is displayed on the display device 100, and sound based on sound data pre-stored in the terminal device 10 is output from the speaker 10S of the terminal device 10.

In addition, some of the action grasping unit 31 , the decision unit 32 , and the output information generating unit 33 may be provided in the server device 30 , and the other parts may be provided in the terminal device 10 .
In other words, the processing performed in this embodiment is not limited to being executed by one specific device, but may be executed in a distributed manner using a plurality of devices.
Whether the processing performed in this embodiment is performed by a single device or distributed across multiple devices, the part where this processing is performed can be considered as an information processing system that performs the processing related to this embodiment.

Here, the information processing system 1 of this embodiment realizes a service that transmits provided information including images and sounds to the terminal device 10 in response to a user's actions on the can body 50. The information processing system 1 of this embodiment creates a new customer experience for a user who has purchased the can body 50, for example, by performing an AR (augmented reality) display, such as superimposing a predetermined image as provided information on an image showing the can body 50, or by outputting sound from the speaker 10S.

A web application for AR (Augmented Reality) (hereinafter, simply referred to as a "web application") downloaded via a network, for example, is installed in the terminal device 10. The web application of this embodiment operates resources of the terminal device 10, such as hardware mounted on the terminal device 10 and software pre-installed on the terminal device 10.
Specifically, the web application of this embodiment operates the resources of the terminal device 10 to realize various functions such as photographing a subject including the can body 50 and displaying it on the screen, transmitting information to a server device 30, etc., receiving information provided from the server device 30, etc., outputting the provided information, and accepting user operations and instructions via the screen.
Examples of the provided information include text information, still images, videos, two-dimensional images, three-dimensional images, music, voice, sound effects, and sounds obtained by recording actual events.

The terminal device 10 captures the user's actions based on a web application. The terminal device 10 transmits the captured image to the server device 30. When the server device 30 receives the captured image from the terminal device 10, it transmits provided information corresponding to the user's actions contained in the captured image to the terminal device 10. The terminal device 10 displays the received provided information by a viewer, or outputs the received provided information from the speaker 10S.

1... Information processing system, 10... Terminal device, 31... Action recognition unit, 32... Decision unit, 50... Can body, 50X... Outer surface

Claims (6)

An action recognition means for analyzing an image obtained by photographing an action performed by a user with respect to a beverage container containing a beverage, the image being obtained by a terminal device having a photographing function, and for recognizing the action performed by the user;
a determination means for making a decision regarding a sound to be output from the terminal device based on the action recognized by the action recognition means;
An information processing system comprising: The information processing system of claim 1, wherein the decision means, when the action recognized by the action recognition means is an action of opening the beverage container, makes a decision to output from the terminal device a sound made by the beverage container when the beverage container is opened. The information processing system described in claim 1, wherein the decision means, when the action recognized by the action recognition means is an action of crushing the beverage container, makes a decision to output from the terminal device a sound made by the beverage container when the beverage container is crushed. The information processing system described in claim 1, wherein the decision means makes a decision that, when the action recognized by the action recognition means is an action of pouring the contents of the beverage container into another container, the sound generated when the contents are poured is output from the terminal device. The information processing system described in claim 1, wherein the decision means makes a decision to cause the terminal device to output a sound representing information instructing the user to throw the beverage container in a trash can when the action detected by the action detection means indicates that the beverage container has been finished. An action recognition function that analyzes an image obtained by photographing an action performed by a user on a beverage container containing a beverage, the image being obtained by a terminal device having a photographing function, and recognizes the action performed by the user;
a determination function for making a decision regarding a sound to be output from the terminal device based on the action recognized by the action recognition function;
A program to make the above happen on a computer.