JP7594573B2 - Information processing device, program, system, and information processing method - Google Patents

Description

The present invention relates to an information processing device, a program, a system, a display device, and an information processing method.

Patent Literature 1 describes a matching service that uses gaze data of a wearer wearing an eyeglass-type device such as AR (Augmented Reality) glasses. Patent Literature 2 describes an AR display control device that can reduce discomfort in space sharing. Patent Literature 3 describes an image processing device that creates a highly stealthy latent image. Patent Literature 4 describes an apparatus for adapting default encoding of a digital video signal during a scene change period.
[Prior Art Literature]
[Patent Documents]
[Patent Document 1] JP 2022-085730 A [Patent Document 2] JP 2021-131741 A [Patent Document 3] JP 2021-004957 A [Patent Document 4] WO 2008/019525

According to one embodiment of the present invention, an information processing device is provided. The information processing device may include a marker insertion unit that inserts a marker for determining a display position of content related to an image displayed on a display unit of a display device while an imaging unit mounted on the display device is imaging the image displayed in a display target area, into the image for a period shorter than a period recognizable by a user of the display device.

The marker insertion unit may insert the marker into the image by performing an image processing process on the image.

In any of the information processing devices, the marker insertion unit may insert the marker into the image while the image is being displayed in the display target area.

In any of the information processing devices, the marker insertion unit may insert the marker into the image by having a marker projection device project the marker.

Any of the information processing devices further includes an information receiving unit that receives user operation information indicating an operation of the user of the display device. The marker insertion unit may insert the marker into the image based on the user operation information of the user of the display device.

Any of the information processing devices may further include an information receiving unit that receives an image captured by the imaging unit mounted on the display device, and the marker insertion unit may insert the marker into the video based on the captured image.

Any of the information processing devices may further include an information storage unit that stores attention area information indicating an attention area in the video where attention is focused, and the marker insertion unit may determine an insertion position of the marker such that the marker is inserted in an area outside the attention area indicated by the attention area information.

Any of the information processing devices may further include an information receiving unit that receives gaze information indicating the gaze of the user of the display device, and an attention area identifying unit that identifies an attention area in the video where attention is focused based on the gaze of the user indicated by the gaze information, and the marker insertion unit may determine an insertion position of the marker such that the marker is inserted in an area outside the attention area identified by the attention area identifying unit.

Any of the information processing devices may further include an information storage unit that stores color-related information related to colors in the image, and the marker insertion unit may determine an insertion position of the marker based on the color-related information so that when the marker is inserted, the color difference between the color of the marker and the color in the image becomes larger.

Any of the information processing devices may further include an information storage unit that stores color-related information related to colors in the image, and the marker insertion unit may determine the color of the marker based on the color-related information so that when the marker is inserted at the insertion position, the color difference between the color of the marker and the color in the image becomes larger.

Any of the information processing devices may further include an information receiving unit that receives user-related information related to the user of the display device, and the marker insertion unit may determine an insertion period for inserting the marker into the video based on the user-related information.

Any of the information processing devices further includes an information receiving unit that receives recognition information indicating whether the display device has been able to recognize the marker, and when the recognition information indicates that the display device has not been able to recognize the marker, the marker insertion unit may change at least one of the insertion position of the marker, the color of the marker, and the size of the marker so that the display device can recognize the marker.

Any of the information processing devices may further include a content delivery unit that delivers the content to the display device.

According to one embodiment of the present invention, a program is provided for causing a computer to function as the information processing device.

According to one embodiment of the present invention, a system is provided. The system may include the information processing device. The system may include the display device.

According to one embodiment of the present invention, a display device is provided. The display device may include a display unit. The display device may include an imaging unit. The display device may include a marker recognition unit that recognizes a marker inserted in a video displayed in a display target area from an image captured by the imaging unit for a period shorter than a period that a user of the display device can recognize. The display device may include a display position determination unit that determines a display position of content displayed on the display unit based on the marker recognized by the marker recognition unit.

The display device may further include an information storage unit that stores angle-of-view information indicating an angle of view of the display target area included in the captured image, and the display position determination unit may determine the display position of the content further based on the angle-of-view information.

In any of the display devices, the marker recognition unit may recognize a plurality of the markers inserted in the video from the captured image, and the display position determination unit may determine the display position of the content, which is 3D content, based on the plurality of markers recognized by the marker recognition unit.

According to one embodiment of the present invention, a program is provided for causing a computer to function as the display device.

According to one embodiment of the present invention, there is provided an information processing method executed by a computer. The information processing method may include a marker insertion step of inserting a marker for determining a display position of content related to an image displayed on a display unit of a display device while an imaging unit mounted on the display device is imaging the image displayed in a display target area, into the image for a period shorter than a period recognizable by a user of the display device.

According to one embodiment of the present invention, there is provided an information processing method executed by a computer. The information processing method may include a marker recognition step of recognizing a marker inserted in an image displayed in a display target area captured by an imaging unit mounted on the computer for a period shorter than the period that a user of the computer can recognize the marker. The information processing method may include a display position determination step of determining a display position of content displayed on a display unit mounted on the computer based on the marker recognized in the marker recognition step.

The above summary of the invention does not list all of the necessary features of the present invention. Also, subcombinations of these features may also be inventions.

1 illustrates a schematic diagram of an example of a system 10. 1 shows an example of a configuration of a display device 100. 10 is an explanatory diagram for explaining an example in which the display device 100 recognizes a marker. FIG. FIG. 2 is an explanatory diagram for explaining an example in which a display device 100 displays content. FIG. 11 is an explanatory diagram for explaining another example in which the display device 100 displays content. 11 is an explanatory diagram for explaining another example in which the display device 100 recognizes a marker. FIG. FIG. 11 is an explanatory diagram for explaining another example in which the display device 100 displays content. FIG. 11 is an explanatory diagram for explaining another example in which the display device 100 displays content. 1 shows an example of a captured image 80 captured by an imaging section 120 of a display device 100. 11 is an explanatory diagram for explaining another example in which the display device 100 recognizes a marker. FIG. 2 illustrates an example of a functional configuration of an information processing device 200. 2 illustrates an example of a functional configuration of a display device 100. 1 shows an example of a process flow by the system 10. An example of the hardware configuration of a computer 1200 that functions as the information processing device 200 or the display device 100 is shown in schematic form.

When a wearer of AR glasses watches an image projected on a screen through the AR glasses, the AR glasses need to perform self-location estimation in order to display content at an appropriate position on the display unit of the AR glasses. However, in a dark place, the AR glasses may not be able to grasp the feature points necessary to perform self-location estimation. The system 10 according to this embodiment employs a mechanism for inserting, for example, a marker used as a feature point for the AR glasses to estimate their own location into the image projected on the screen for a short period of time that is not noticeable to humans.

The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

FIG. 1 shows an example of a system 10. The system 10 includes a display device 100 and an information processing device 200. The system 10 may further include a communication device 300. The system 10 may further include a video display device 400. The system 10 may further include a communication terminal 150.

The system 10 is used in facilities that provide services using AR technology that virtually expands real space by adding virtual visual information to the image displayed in the display target area 50. The system 10 is used, for example, in movie theaters. The system 10 is used, for example, in digital art event venues such as projection mapping. The system 10 is used, for example, in live venues for virtual live shows. The system 10 is used, for example, in theme parks such as amusement parks. The system 10 is used, for example, in game arcades. The system 10 may also be used at home. FIG. 1 shows an example of the system 10 being used in a movie theater.

The display target area 50 is, for example, the screen area of a screen device. The display target area 50 is, for example, the display area of a display device. The display target area 50 is, for example, the wall area, ceiling area, floor area, etc. of a room or building. The display target area 50 may be any other area in which an image can be displayed. Figure 1 shows an example in which the display target area 50 is the screen area of a screen device.

The display device 100 may be any device having a wireless communication function and a display function for displaying content on a display unit. The display device 100 is, for example, a glasses-type device. The glasses-type device is, for example, AR glasses. The display device 100 may be a smartphone, a tablet terminal, or the like. FIG. 1 shows an example in which the display device 100 is AR glasses.

The content may be in any data format. For example, the content is 2D CG (Computer Graphics) data. For example, the content is 3D CG data. For example, the content is a still image. Also, for example, the content is a video. For example, the content is text. The content may include a combination of these.

The display device 100 may be a device capable of giving the user 102 the sensation that the content is located in real space by displaying the content on a transparent or semi-transparent display unit. In other words, the display device 100 may be a so-called transmissive (see-through) device. The display device 100 may also be a video pass-through device. In this case, the display device 100 may display a captured image capturing a range corresponding to the field of view of the user 102 on a non-transmissive display, and superimpose the content.

The display device 100 has a function of estimating its own position, for example. The display device 100 estimates its own position based on an image captured by an imaging unit mounted on the display device 100, for example. The display device 100 estimates its own position by, for example, extracting feature points for estimating its own position from the captured image. The display device 100 estimates its own position based on, for example, the shape and size of the extracted feature points. The display device 100 may display content on the display unit based on the estimation result of the own position.

The information processing device 200 executes information processing related to the video. For example, the information processing device 200 executes an insertion process to insert information into the video. For example, while an imaging unit mounted on the display device 100 is capturing the video displayed in the display target area 50, the information processing device 200 inserts a marker 60 for determining the display position of content related to the video displayed on the display unit of the display device 100 into the video for a period shorter than the period that the user 102 of the display device 100 can recognize. Note that the user 102 of the display device 100 may be the person wearing the display device 100.

The information processing device 200, for example, uses the visual masking effect to insert the marker 60 into the image for a period shorter than the period that the user 102 of the display device 100 can recognize. The visual masking effect is an effect in which, when two visual stimuli, a target stimulus and a mask stimulus, are displayed close to each other in time and space, the perception of the target stimulus is hindered compared to when only the target stimulus is displayed. Note that the case in which the mask stimulus is displayed temporally before the target stimulus is called forward masking, and the case in which the mask stimulus is displayed temporally after the target stimulus is called backward masking. For example, the information processing device 200 uses the visual masking effect to insert the marker 60 into the image for 70 to 80 ms or less.

The frame rate of the video is, for example, 60 fps (frames per second). In this case, the information processing device 200 may insert the marker 60 into 4 to 5 or fewer consecutive frames by utilizing the visual masking effect. The frame rate of the video may be any other value.

The marker 60 is, for example, a QR (Quick Response) code. The marker 60 is, for example, a barcode. The marker 60 may be a graphic of any shape, such as a rectangle or a circle, as long as the display device 100 can recognize it as a marker.

The marker 60 includes, for example, content-related information related to the content. The content-related information includes, for example, content identification information that identifies the content. The content-related information includes, for example, data format information that indicates the data format of the content.

The content related to the video may be anything as long as it is related to the video. The content related to the video may be, for example, content used in the direction of the video, content used as subtitles for the video, content used to explain the scenes in the video, content used to advertise products that appear in the video, etc.

The information processing device 200 provides the video to the communication device 300. The information processing device 200 provides the video to the communication device 300, for example, by communicating with the communication device 300 via the network 20.

The network 20 may include a core network provided by a telecommunications carrier. The core network may be compliant with, for example, a 5G (5th Generation) communication system. The core network may be compliant with a mobile communication system after the 6G (6th Generation) communication system. The core network may be compliant with a 3G (3rd Generation) communication system or an LTE (Long Term Evolution) communication system. The network 20 may include the Internet. The network 20 may be compatible with a Wi-Fi (registered trademark) communication method. The network 20 may be accessible via a Wi-Fi access point. The network 20 may be accessible using other wireless communication in general. The network 20 may be accessible via a wired connection.

The communication device 300 may be any device capable of communicating with the information processing device 200. The communication device 300 may be, for example, a laptop PC (Personal Computer), a desktop PC, or other PC. The communication device 300 may also be a smartphone, a tablet terminal, or the like. FIG. 1 shows an example in which the communication device 300 is a laptop PC.

The communication device 300 transmits, for example, an image provided by the information processing device 200 to the image display device 400. The communication device 300 transmits the image to the image display device 400, for example, via a wired connection. The communication device 300 may establish a wired connection with the image display device 400 using an HDMI (registered trademark) (High-Definition Multimedia Interface) cable or the like. The communication device 300 may transmit the image to the image display device 400 via a wireless connection. The communication device 300 may establish a wireless connection with the image display device 400 using a short-range wireless communication method such as Wi-Fi, Bluetooth (registered trademark), and Zigbee (registered trademark).

The image display device 400 may be any device capable of displaying an image received from the communication device 300. The image display device 400 is, for example, a projector. The image display device 400 may also be a display device. FIG. 1 shows an example in which the image display device 400 is a projector.

The information processing device 200 distributes, for example, video-related content to the communication terminal 150. The information processing device 200 distributes the content to the communication terminal 150 by, for example, communicating with the communication terminal 150 via the network 20.

The communication terminal 150 may be any device capable of communicating with the information processing device 200. The communication terminal 150 may be, for example, a smartphone or a tablet terminal. The communication terminal 150 may be a PC. The communication terminal 150 may be a wearable terminal. The communication terminal 150 may be AR glasses. FIG. 1 shows an example in which the communication terminal 150 is a smartphone.

The communication terminal 150 transmits content related to the video delivered by the information processing device 200 to the display device 100. The communication terminal 150 transmits the content to the display device 100, for example, via a wireless connection. The communication terminal 150 may establish a wireless connection with the display device 100 using a short-range wireless communication method such as Wi-Fi, Bluetooth, or Zigbee. The communication terminal 150 may transmit the content to the display device 100 via a wired connection. The communication terminal 150 and the display device 100 may be integrated.

For example, the display device 100 recognizes a marker 60 inserted in an image displayed in the display target area 50, from an image captured by an imaging unit mounted on the display device 100, for a period shorter than the period during which the user 102 of the display device 100 can recognize the image. Next, the display device 100 determines the display position of the content received from the communication terminal 150 based on the recognized marker 60. After that, the display device 100 displays the content on the display unit of the display device 100 according to the determined display position.

In addition, the information processing device 200 may function as MEC (Mobile Edge Computing or Multi-access Edge Computing). In other words, the information processing device 200 may be an MEC server.

In order to use AR technology to display content at an appropriate position on the display unit of a display device, it is necessary to have the display device recognize feature points for determining the display position of the content. However, in dark places such as movie theaters and event venues, the display device may not be able to recognize feature points and may not be able to display the content at the appropriate position on the display unit. On the other hand, if the wearer of the display device recognizes the feature points, there is a risk that the wearer of the display device will lose interest. For the above reasons, it is desirable that in dark places, feature points for determining the display position of the content be displayed in a manner that can be recognized by the display device but cannot be recognized by the wearer of the display device.

According to the system 10 of this embodiment, the information processing device 200 uses the visual masking effect to insert the marker 60, which is a feature point for determining the display position of the content, into the image for such a short period of time that the user 102 of the display device 100 cannot recognize it. Since the display target area 50 in which the image is displayed is a bright area in a dark place, the information processing device 200 can make the display device 100 recognize the marker 60 inserted into the image displayed in the display target area 50. Therefore, the information processing device 200 can display the content at an appropriate position on the display unit of the display device in a dark place. Furthermore, since the period during which the marker 60 is inserted into the image is shorter than the period during which the user 102 of the display device 100 can recognize the marker 60, the information processing device 200 does not make the user 102 of the display device 100 recognize the marker 60 inserted into the image displayed in the display target area 50. As described above, the system 10 according to the present embodiment can improve the sense of immersion of a user of a display device viewing video through the display device in a dark place by having the display device recognize the marker and displaying content in an appropriate position on the display unit of the display device without the user of the display device being aware of the marker.

Figure 2 shows a schematic example of the configuration of the display device 100. The display device 100 includes a frame 110 having a rim 112 and temples 118. The display device 100 may also include a battery.

The display device 100 includes, for example, an imaging unit 120. The imaging range of the imaging unit 120 may be a range that corresponds to the field of view of the user 102 wearing the display device 100. The arrangement of the imaging unit 120 in FIG. 2 is an example, and the imaging unit 120 may be arranged in another position.

The display device 100 includes, for example, a sensor 122. The sensor 122 may be a sensor for detecting the movement of the user 102 wearing the display device 100. The sensor 122 is, for example, an acceleration sensor or a gyro sensor. FIG. 2 illustrates an example in which the sensor 122 is disposed on the bridge 114, but the arrangement of the sensor 122 in FIG. 2 is only an example, and the sensor 122 may be disposed in another position.

The display device 100 includes, for example, a gaze detection unit 124. The gaze detection unit 124 may detect the gaze of the user 102 who is wearing the display device 100. The gaze detection unit 124 detects the gaze of the user 102, for example, by capturing an image of the eyes of the user 102 and monitoring them. The arrangement of the gaze detection unit 124 in FIG. 2 is one example, and the gaze detection unit 124 may be arranged in another position.

The display device 100 may accept various inputs. The display device 100 may accept various inputs from the user 102 via an operation unit arranged at any position on the display device 100. The display device 100 accepts various inputs from the user 102, for example, via an input unit such as a button arranged at any position on the frame 110. The display device 100 may accept various inputs from the user 102, for example, via a communication terminal 150.

The display device 100 may have a display function on the lens 116. The lens 116 may be an example of a display unit of the display device 100. The lens 116 may be a transparent or semi-transparent display, and the display device 100 may cause the lens 116 to perform various displays. That is, the display device 100 may be a transparent or semi-transparent display type device. The display device 100 may also have a function of projecting a display onto the lens 116. That is, the display device 100 may be a projection type device.

The display device 100 may have a function of performing communication via the network 20. The display device 100 may access the network 20 by at least one of mobile communication, Wi-Fi communication, and other general wireless communication. The display device 100 may communicate with the information processing device 200 via the network 20. The information processing device 200 may deliver content related to video to the display device 100 via the network 20.

Figure 3 is an explanatory diagram for explaining an example of how the display device 100 recognizes a marker. In Figures 3 to 8 and 10, the x-axis direction in the figure is the horizontal direction, and the y-axis direction in the figure is the vertical direction.

The display device 100 recognizes, for example, a marker 60 that is a marker inserted into the video by the information processing device 200 from an image captured by the imaging unit 120 of the video displayed in the display target area 50. The following explanations of Figures 4 and 5 each explain an example in which, when the display device 100 recognizes the marker 60 shown in Figure 3, it determines the display position of the content displayed on the lens 116 based on the marker 60.

Figure 4 is an explanatory diagram for explaining an example in which the display device 100 displays content. The display device 100 recognizes, for example, the marker 60 shown in Figure 3, and determines the display position of one piece of content to be displayed on the lens 116 based on the recognized marker 60. The display device 100 may also determine the display positions of multiple pieces of content to be displayed on the lens 116 based on the recognized marker 60. Here, an example in which the display device 100 determines the display position of one piece of content 70 to be displayed on the lens 116 based on the recognized marker 60 will be mainly described.

For example, the display device 100 determines the display position of the content 70 based on positional relationship information indicating the positional relationship between the reference position and the display position of the content 70, using the center position of the recognized marker 60 as a reference position. The display device 100 receives the positional relationship information from the information processing device 200, for example. The display device 100 may also receive the positional relationship information from the communication terminal 150. The display device 100 may read content-related information including the positional relationship information from the marker 60.

4, the display device 100 determines that a position that is x _c away from the reference position in the x-axis direction and y _c away from the reference position in the y-axis direction is the display position of the content 70. The display device 100 may cause the lens 116 to display the content 70 so that the content 70 is displayed at the determined display position.

FIG. 5 is an explanatory diagram for explaining another example in which the display device 100 displays content. In the example shown in FIG. 5, the display device 100 determines that the center position of the marker 60 shown in FIG. 3 is the display position of the content 70. The display device 100 may cause the lens 116 to display the content 70 so that the content 70 is displayed at the determined display position.

Figure 6 is an explanatory diagram for explaining another example of the display device 100 recognizing a marker. For example, the display device 100 recognizes three markers, marker 62, marker 64, and marker 66, inserted into the video by the information processing device 200 from an image captured by the imaging unit 120 of the video displayed in the display target area 50. Each of the markers 62, 64, and 66 may have the same characteristics as the marker 60. In the following explanations of Figures 7 and 8, an example will be explained in which the display device 100 determines the display position of the content displayed on the lens 116 based on the markers 62, 64, and 66 when it recognizes the markers 62, 64, and 66 shown in Figure 6.

Figure 7 is an explanatory diagram for explaining another example of how the display device 100 displays content. Here, an example is mainly explained in which the display device 100 recognizes the markers 62, 64, and 66 shown in Figure 6, and determines the display position of the content 70, which is one piece of 3D content displayed on the lens 116, based on the recognized markers 62, 64, and 66.

The display device 100 determines the display position of the content 70, for example, based on correspondence information indicating that the recognized markers 62, marker 64, and marker 66 are markers for determining the display position of the content 70, and the positional relationship between the markers 62, marker 64, and marker 66. The display device 100 receives the correspondence information from, for example, the information processing device 200. The display device 100 may also receive the correspondence information from the communication terminal 150. The display device 100 may read content-related information including the correspondence information from each of the markers 62, marker 64, and marker 66.

The correspondence information includes, for example, content identification information. The correspondence information includes, for example, marker identification information that identifies a marker.

The display device 100 may determine the positional relationship between the markers 62, 64, and 66 based on the respective central positions of the markers 62, 64, and 66. The display device 100 may cause the lens 116 to display the content 70 so that the content 70 is displayed at the determined display position.

Figure 8 is an explanatory diagram for explaining another example of displaying content by the display device 100. Here, an example will be mainly explained in which the display device 100 recognizes the markers 62, 64, and 66 shown in Figure 6, and determines the display positions of the three pieces of content 72, content 74, and content 76 displayed on the lens 116 based on the recognized markers 62, 64, and 66, respectively.

The display device 100 determines the display positions of the contents 72, 74, and 76, based on, for example, a plurality of pieces of correspondence information indicating that the recognized markers 62, 64, and 66 are markers for determining the display positions of the contents 72, 74, and 76, respectively. In the example shown in FIG. 8, the display device 100 determines that the center position of the marker 62 is the display position of the content 72, the center position of the marker 64 is the display position of the content 74, and the center position of the marker 66 is the display position of the content 76. The display device 100 may cause the lens 116 to display the contents 72, 74, and 76, respectively, so that the contents 72, 74, and 76 are displayed at the determined display positions.

Figure 9 shows an example of a captured image 80 captured by the imaging section 120 of the display device 100. Figure 9 shows an example of a captured image captured by the imaging section 120 when the user 102 and the user 502 of the display device 100 are playing a snowball fight video game displayed in the display target area 50. The captured image 80 includes footage of the snowball fight video game displayed in the display target area 50 and the user 502. The display device 100 may transmit the captured image 80 to the information processing device 200.

The information processing device 200 inserts the marker 60 into the video based on, for example, the captured image 80 received from the display device 100. For example, the information processing device 200 detects the movement of the user 502 by performing image analysis on the captured image 80. The information processing device 200 may insert the marker 60 into the video based on the detected movement of the user 502. For example, the information processing device 200 inserts the marker 60 into the video when the movement of the user 502 is a specific movement related to the video. When the video displayed in the display target area 50 is a video of a snowball fight video game, the information processing device 200 inserts the marker 60 into the video when, for example, the movement of the user 502 is a pitching movement.

The information processing device 200 may detect an overlapping area where the display target area 50 and the user 502 overlap by performing image analysis on the captured image 80. In this case, the information processing device 200 may insert the marker 60 into the video so that the marker 60 is displayed in an area outside the overlapping area.

Figure 10 is an explanatory diagram for explaining another example of the display device 100 recognizing a marker. The display device 100 may recognize a marker 60 inserted into the outer area of the overlapping area by the information processing device 200.

Figure 11 shows an example of the functional configuration of the information processing device 200. The information processing device 200 includes an information storage unit 202, an information receiving unit 204, a request transmitting unit 206, a marker inserting unit 208, an attention area identifying unit 210, a video providing unit 212, and a content distribution unit 214. Note that it is not essential that the information processing device 200 includes all of these components.

The information storage unit 202 stores various information. The information storage unit 202 stores, for example, video-related information related to the video into which the marker 60 is to be inserted. The video-related information includes, for example, the video into which the marker 60 is to be inserted. The video-related information includes, for example, attention area information indicating an attention area in the video where attention is focused. The attention area is, for example, an area occupied by an attention target such as a character or a person appearing in the video. The attention area may be the central area of the video. The video-related information includes, for example, color-related information related to a color in the video. The color-related information includes, for example, hue information indicating a hue in the video. The color-related information includes, for example, brightness information indicating a brightness in the video. The color-related information may include saturation information indicating a saturation in the video. The video-related information includes, for example, main target information indicating a main target of the video. The main target of the video is, for example, a child. The main target of the video is, for example, an adult. The main target of the video is, for example, a man. The main target of the video is, for example, a woman. The video-related information may include specific action information that indicates a specific action related to the video.

The information storage unit 202 stores content related to the video into which the marker 60 is to be inserted. The information storage unit 202 may store the content in association with video-related information.

The information storage unit 202 may store content-related information for the content. The information storage unit 202 may store the content-related information in association with the video-related information.

The information receiving unit 204 receives various information from an external device. For example, the information receiving unit 204 receives various information from an external device via the network 20. The information receiving unit 204 may store the received various information in the information storage unit 202.

The information receiving unit 204 receives various information from, for example, the display device 100. The information receiving unit 204 receives, for example, user-related information related to a user of the display device 100. The user-related information includes, for example, generation information indicating the generation of the user. The user-related information includes, for example, age information indicating the age of the user. The user-related information may include gender information indicating the gender of the user. The information receiving unit 204 may receive gaze information indicating the user's gaze.

The information receiving unit 204 receives various information, for example, from the communication terminal 150. The information receiving unit 204 may receive information from the communication terminal 150 similar to the information received from the display device 100.

The information receiving unit 204 receives various information, for example, while the video is being displayed in the display target area 50. The information receiving unit 204 receives user action information indicating the action of the user of the display device 100, for example. The action of the user of the display device 100 is detected, for example, by a sensor 122 provided in the display device 100. The action of the user of the display device 100 is detected, for example, by a sensor such as an acceleration sensor or a gyro sensor provided in the communication terminal 150. The action of the user of the display device 100 may be detected by a camera installed in a space including the display target area 50. In this case, the information receiving unit 204 may receive a captured image including the user of the display device 100, captured by the camera.

The information receiving unit 204 receives, for example, an image captured by the imaging unit 120 mounted on the display device 100 of the video displayed in the display target area 50. The information receiving unit 204 may receive recognition success/failure information indicating whether the display device 100 was able to recognize the marker 60.

The request sending unit 206 requests the external device to send various information. The request sending unit 206 requests the external device to send various information, for example, via the network 20. The information receiving unit 204 may receive various information sent by the external device in response to a request by the request sending unit 206.

The request sending unit 206 sends a request to, for example, the display device 100. The request sending unit 206 sends a request to, for example, the communication terminal 150. The request sending unit 206 may also send a request to the communication device 300.

The marker insertion unit 208 inserts the marker 60 into the video into which the marker 60 is to be inserted. For example, the marker insertion unit 208 inserts multiple markers 60 into the video into which the marker 60 is to be inserted.

The marker insertion unit 208, for example, inserts a marker 60 for determining the display position of content related to the image displayed on the display unit of the display device 100 while the imaging unit 120 mounted on the display device 100 is capturing the image displayed in the display target area 50, into the image for a period shorter than the period recognizable by the user of the display device 100. The marker insertion unit 208 inserts the marker 60 into the image, for example, by performing an image processing process on the image. The marker insertion unit 208 may insert the marker 60 into the image by having a marker projection device project the marker 60. The marker projection device is, for example, a projector.

The marker insertion unit 208, for example, determines the insertion position of the marker 60. The marker insertion unit 208, for example, determines the color of the marker 60. The marker insertion unit 208, for example, determines the size of the marker 60. The marker insertion unit 208, for example, determines the timing at which the marker 60 is inserted into the video. The marker insertion unit 208 may determine the insertion period for inserting the marker 60 into the video.

The marker insertion unit 208 inserts the marker 60 into the video, for example, based on various information stored in the information storage unit 202. The marker insertion unit 208 inserts the marker 60 into the video, for example, based on content-related information stored in the information storage unit 202 in association with the video-related information.

The marker insertion unit 208 inserts the marker 60 into the video, for example, based on the correspondence information included in the content-related information. The marker insertion unit 208 inserts the marker 60, which is associated with the content by the correspondence information, into the video. The marker insertion unit 208 may insert the marker 60 into the video so that the marker 60 includes the content-related information.

The marker insertion unit 208 inserts the marker 60 into the video based on, for example, the video-related information. The marker insertion unit 208 inserts the marker 60 into the video based on, for example, the area-of-attention information included in the video-related information. The marker insertion unit 208 determines the insertion position of the marker 60 based on, for example, the area-of-attention information. The marker insertion unit 208 determines the insertion position of the marker 60 such that, for example, the marker 60 is inserted into an area outside the area of attention indicated by the area-of-attention information.

When a user of the display device 100 views an image displayed in the display target area 50, the user tends to focus on the area of interest. Therefore, the information processing device 200 can further reduce the possibility that the marker 60 will be recognized by the user of the display device 100 by inserting the marker 60 into the image so that the marker 60 is inserted into an area outside the area of interest.

The marker insertion unit 208 inserts the marker 60 into the video based on, for example, color-related information included in the video-related information. The marker insertion unit 208 determines the insertion position of the marker 60 based on, for example, the color-related information. The marker insertion unit 208 determines the insertion position of the marker 60 such that, when the marker 60 is inserted, the color difference between the color of the marker 60 and the color in the video becomes larger. The marker insertion unit 208 may determine the color of the marker 60 based on, for example, the color-related information. The marker insertion unit 208 may determine the color of the marker 60 such that, when the marker 60 is inserted at the insertion position, the color difference between the color of the marker 60 and the color in the video becomes larger.

The greater the color difference between the color of the marker 60 and the color in the image, the easier it is for the display device 100 to recognize the marker 60 inserted in the image. Therefore, the information processing device 200 can make the display device 100 more reliably recognize the marker 60 by inserting the marker 60 into the image so that the color difference between the color of the marker 60 and the color in the image becomes greater.

The marker insertion unit 208 inserts the marker 60 into the image based on the user-related information of the user of the display device 100. The marker insertion unit 208 inserts the marker 60 into the image based on, for example, multiple pieces of user-related information of each user of multiple display devices 100.

The marker insertion unit 208 determines an insertion period for inserting the marker 60 into the video based on, for example, user-related information. The marker insertion unit 208 determines an insertion period for inserting the marker 60 into the video so that the marker 60 is inserted into the video for a period shorter than the period during which a user of the display device 100 of the age indicated by the age information included in the user-related information can recognize the marker 60. The marker insertion unit 208 determines an insertion period for inserting the marker 60 into the video so that the marker 60 is inserted into the video for a period shorter than the period during which a user of the display device 100 of the age indicated by the age information included in the user-related information can recognize the marker 60. The marker insertion unit 208 may determine an insertion period for inserting the marker 60 into the video so that the marker 60 is inserted into the video for a period shorter than the period during which a user of the display device 100 of the gender indicated by the gender information included in the user-related information can recognize the marker 60.

The marker insertion unit 208 may insert the marker 60 into the video based on main target information of the video into which the marker 60 is to be inserted. The marker insertion unit 208 determines an insertion period for inserting the marker 60 into the video, for example, based on the main target information of the video. The marker insertion unit 208 determines an insertion period for inserting the marker 60 into the video, for example, so that the marker 60 is inserted into the video for a period shorter than the period during which the main target of the video indicated by the main target information is recognizable.

The insertion period of the marker 60 during which the user of the display device 100 becomes able to recognize the marker 60 tends to differ depending on the age and gender of the user of the display device 100. Therefore, the marker insertion unit 208 can further reduce the possibility that the marker 60 will be recognized by the user of the display device 100 by inserting the marker 60 into the image while taking into account the age and gender of the user of the display device 100.

The marker insertion unit 208, for example, inserts the marker 60 into the video while the video is being displayed in the display target area 50. That is, the marker insertion unit 208 inserts the marker 60 in real time while the video is being played back. The marker insertion unit 208 may also insert the marker 60 into the video in advance before the video is displayed in the display target area 50.

The marker insertion unit 208 inserts the marker 60 into the video, for example, based on various information received by the information receiving unit 204 while the video is displayed in the display target area 50. The marker insertion unit 208 inserts the marker 60 into the video, for example, based on user operation information of the user of the display device 100. The marker insertion unit 208 determines the timing of inserting the marker 60 into the video, for example, based on the user operation information. The marker insertion unit 208 inserts the marker 60 into the video, for example, when the operation of the user of the display device 100 indicated by the user operation information is a specific operation related to the video indicated by the specific operation information. In this case, the marker insertion unit 208 may insert the marker 60 that is associated with the content corresponding to the specific operation by the correspondence information into the video. The marker insertion unit 208 may insert the marker 60 into the video when the user of the display device 100 moves a moving distance longer than a predetermined moving distance threshold.

The marker insertion unit 208 inserts the marker 60 into the video, for example, based on a captured image captured by the imaging unit 120 mounted on the display device 100. The marker insertion unit 208 inserts the marker 60 into the video, for example, by performing image analysis on the captured image.

The marker insertion unit 208 inserts the marker 60 into the video, for example, based on the video displayed in the display target area 50 and the captured image including other users who are not the user of the display device 100. The marker insertion unit 208 determines the timing to insert the marker 60 into the video, for example, based on the captured image. The marker insertion unit 208 detects the action of other users from the captured image, for example, and inserts the marker 60 into the video when the detected action of other users is a specific action related to the video indicated by the specific action information. In this case, the marker insertion unit 208 may insert the marker 60 that is associated with the content corresponding to the specific action by the correspondence information into the video. The marker insertion unit 208 may insert the marker 60 into the video when the other users move a distance longer than a predetermined movement distance threshold.

The marker insertion unit 208 determines the insertion position of the marker 60 based on, for example, the captured image. The marker insertion unit 208 detects, for example, an overlapping area in which the display target area 50 and another user overlap from the captured image, and determines the insertion position of the marker 60 so that the marker 60 is displayed in an area outside the overlapping area.

The marker insertion unit 208 may insert the marker 60 into the video based on the video displayed in the display target area 50 and a captured image including obstacles other than other users. The marker insertion unit 208 determines the insertion position of the marker 60 based on the captured image, for example. The marker insertion unit 208 detects an overlapping area where the display target area 50 and an obstacle overlap from the captured image, for example, and determines the insertion position of the marker 60 so that the marker 60 is displayed in an area outside the overlapping area.

The marker insertion unit 208 may insert the marker 60 into the video based on the recognition feasibility information. For example, when the recognition feasibility information indicates that the display device 100 cannot recognize the marker 60, the marker insertion unit 208 changes at least one of the insertion position of the marker 60, the color of the marker 60, and the size of the marker 60 so that the display device 100 can recognize the marker 60. When the information receiving unit 204 does not receive the recognition feasibility information from the display device 100 within a predetermined period from the time when the marker 60 is displayed in the display target area 50, the marker insertion unit 208 may change at least one of the insertion position of the marker 60, the color of the marker 60, and the size of the marker 60 so that the display device 100 can recognize the marker 60.

For example, when changing the insertion position of the marker 60, the marker insertion unit 208 changes the insertion position of the marker 60 so that the color difference between the color of the marker 60 at the changed insertion position of the marker 60 and the color in the video becomes larger. When changing the color of the marker 60, for example, the marker insertion unit 208 changes the color of the marker 60 so that the color difference between the color of the marker 60 after the change and the color in the video becomes larger. When changing the size of the marker 60, for example, the marker insertion unit 208 changes the size of the marker 60 so that the size of the marker 60 after the change is larger than the size of the marker 60 before the change.

The attention area identifying unit 210 identifies an attention area. The attention area identifying unit 210 identifies an attention area, for example, based on the gaze of a user indicated by gaze information of the user of the display device 100 stored in the information storage unit 202. The attention area identifying unit 210 identifies an attention area, for example, based on the gaze of each of a plurality of users indicated by a plurality of gaze information of each user of a plurality of display devices 100. The attention area identifying unit 210 identifies an attention area, for example, by identifying an area in the video to which the user's gaze is more directed. The attention area identifying unit 210 may identify an attention area by receiving input from the creator or director of the video by an input device provided in the information processing device 200.

The attention area identifying unit 210 identifies the attention area, for example, while the video is being displayed in the display target area 50. That is, the attention area identifying unit 210 identifies the attention area in real time while the video is being played back. The attention area identifying unit 210 may identify the attention area in advance before the video is displayed in the display target area 50.

The attention area identification unit 210 may store the identified attention area in the information storage unit 202 as attention area information. The marker insertion unit 208 may insert a marker 60 into the video based on the attention area identified by the attention area identification unit 210.

The video providing unit 212 provides the video into which the marker 60 has been inserted by the marker insertion unit 208. The video providing unit 212 provides the video to, for example, the communication device 300.

The video providing unit 212 provides the video by, for example, transmitting the video via the network 20. The video providing unit 212 transmits the video by, for example, a streaming method. The video providing unit 212 may transmit the video by a download method. The video providing unit 212 may provide the video by recording the video on a recording medium such as a USB (Universal Serial Bus).

The content distribution unit 214 distributes the content stored in the information storage unit 202. The content distribution unit 214 distributes the content, for example, via the network 20.

The content distribution unit 214 distributes content to, for example, the display device 100. The content distribution unit 214 distributes content to, for example, the communication terminal 150. The content distribution unit 214 may also distribute content to the communication device 300.

The content distribution unit 214 distributes content based on, for example, content-related information stored in the information storage unit 202 in association with the video-related information. The content distribution unit 214 distributes content that is associated with the marker 60 by the correspondence information included in the content-related information.

The content distribution unit 214 distributes content, for example, while a video is displayed in the display target area 50. The content distribution unit 214 distributes content in real time while the video is being played. The content distribution unit 214 distributes content related to the video, for example, while the video providing unit 212 is providing the video. The content distribution unit 214 may distribute content in advance before the video is displayed in the display target area 50.

Figure 12 shows an example of a schematic functional configuration of the display device 100. The display device 100 includes an information storage unit 104, an information receiving unit 106, an angle of view determination unit 107, a marker recognition unit 108, a display position determination unit 109, an imaging unit 120, a sensor 122, a gaze detection unit 124, a display unit 126, a request receiving unit 128, and an information transmitting unit 130. Note that it is not essential that the display device 100 includes all of these components.

The information storage unit 104 stores various types of information. The information storage unit 104 stores, for example, user-related information of the user of the display device 100. The information storage unit 104 stores, for example, captured images captured by the imaging unit 120. The information storage unit 104 stores, for example, user operation information indicating the operation of the user of the display device 100 detected by the sensor 122. The information storage unit 104 may store gaze information indicating the gaze of the user of the display device 100 detected by the gaze detection unit 124.

The information receiving unit 106 receives various information. The information receiving unit 106 receives various information, for example, via the network 20. The information receiving unit 106 may store the received various information in the information storage unit 104.

The information receiving unit 106 receives various information from, for example, the information processing device 200. The information receiving unit 106 receives various information from, for example, the communication terminal 150. The information receiving unit 106 may receive various information from, for example, the communication device 300.

The information receiving unit 106 receives, for example, content related to the video displayed in the display target area 50. The information receiving unit 106 may also receive content-related information related to the content.

The angle of view determination unit 107 determines the angle of view of the display target area 50 included in the captured image captured by the imaging unit 120. The angle of view determination unit 107 determines the angle of view of the display target area 50 based on, for example, a captured image of the display target area 50 captured by the imaging unit 120 when the space including the display target area 50 is bright before the image is displayed in the display target area 50. When the viewing position from which the user of the display device 100 views the image displayed in the display target area 50 can be specified in advance, such as in a movie theater or a live performance venue, the angle of view determination unit 107 may determine the angle of view of the display target area 50 based on the viewing position of the user of the display device 100. The angle of view determination unit 107 may store the determined angle of view of the display target area 50 as angle of view information in the information storage unit 202.

The marker recognition unit 108 recognizes the markers 60 inserted in the video displayed in the display target area 50. For example, the marker recognition unit 108 recognizes the markers 60 inserted in the video from a captured image captured by the imaging unit 120 of the video displayed in the display target area 50 for a period shorter than the period during which the user of the display device 100 can recognize the markers. The marker recognition unit 108 may recognize multiple markers 60 inserted in the video from the captured image. The marker recognition unit 108 may read content-related information from the recognized markers 60.

The marker recognition unit 108, for example, recognizes the shape of the marker 60. The marker recognition unit 108 may also recognize the size of the marker 60.

The display position determination unit 109 determines the display position of the content stored in the information storage unit 104 to be displayed on the display unit 126. The display position determination unit 109 determines the display position of the content, for example, based on the marker 60 recognized by the marker recognition unit 108. The display position determination unit 109 determines the display position of the content, which is 3D content, for example, based on multiple markers 60 recognized by the marker recognition unit 108. The display position determination unit 109 may determine the display positions of multiple pieces of content, based on multiple markers 60 recognized by the marker recognition unit 108.

The display position determination unit 109 determines the display position of the content based on, for example, the shape and size of the marker 60. For example, the display position determination unit 109 estimates the position of the display device 100 based on the shape and size of the marker 60. The display position determination unit 109 may determine the display position of the content so as to more closely match the estimated position of the display device 100. In this case, the marker 60 may be a feature point for estimating the position of the display device 100.

The display position determination unit 109 determines the display position of the content based on, for example, the positional relationship of the multiple markers 60. For example, the display position determination unit 109 estimates the position of the display device 100 based on the positional relationship of the multiple markers 60. The display position determination unit 109 may determine the display position of the content so as to more closely match the estimated position of the display device 100. In this case, the multiple markers 60 may be feature points for estimating the position of the display device 100. The display position determination unit 109 may determine the positional relationship of the multiple markers 60 by identifying the center position of each of the multiple markers 60.

The display position determination unit 109 determines the display position of the content further based on the content-related information. The display position determination unit 109 determines the display position of the content based on the positional relationship information included in the content-related information. The display position determination unit 109 determines the display position of the content based on the positional relationship between the reference position, which is the center position of the marker 60 recognized by the marker recognition unit 108, and the display position of the content, as indicated by the positional relationship information, for example.

The display position determination unit 109 determines the display position of the content, for example, based on the correspondence information included in the content-related information. For example, when the correspondence information indicates that one marker 60 is a marker for determining the display position of one piece of content, the display position determination unit 109 determines that the center position of the marker 60 is the display position of the content. For example, when the correspondence information indicates that multiple markers 60 recognized by the marker recognition unit 108 are markers for determining the display position of one piece of content that is 3D content, the display position determination unit 109 determines the display position of the content that is 3D content based on the positional relationship of the multiple markers 60. In this case, the multiple markers 60 are, for example, three or more markers 60.

The display position determination unit 109 determines the display position of the content, for example, further based on the angle of view information stored in the information storage unit 104. For example, the display position determination unit 109 estimates the position of the display device 100 based on the angle of view of the display target area 50 included in the captured image captured by the imaging unit 120, which is indicated by the angle of view information. The display position determination unit 109 may determine the display position of the content so as to more closely match the estimated position of the display device 100. In this case, the angle of view of the display target area 50 may be a feature point for estimating the position of the display device 100.

The display unit 126 displays the content stored in the information storage unit 104. For example, the display unit 126 displays the content at a display position determined by the display position determination unit 109.

The display unit 126 displays content, for example, based on the content-related information stored in the information storage unit 104. The display unit 126 displays content that is associated with the marker 60 recognized by the marker recognition unit 108, for example, based on the correspondence information included in the content-related information.

The request receiving unit 128 receives requests to transmit various information stored in the information storage unit 104. The request receiving unit 128 receives the requests, for example, via the network 20.

The request receiving unit 128 receives a request, for example, from the information processing device 200. The request receiving unit 128 may also receive a request from the communication terminal 150.

The information transmission unit 130 transmits various pieces of information stored in the information storage unit 104. The information transmission unit 130 transmits the various pieces of information, for example, via the network 20.

The information transmission unit 130 transmits various pieces of information, for example, in response to the request reception unit 128 receiving a request. The information transmission unit 130 may transmit various pieces of information periodically.

The information transmission unit 130 transmits, for example, various types of information to the information processing device 200. The information transmission unit 130 may also transmit various types of information to the communication terminal 150.

Figure 13 shows an example of the flow of processing by the system 10. Here, the starting state is a state in which no video is displayed in the display target area 50.

In step (sometimes abbreviated to S) 102, the marker insertion unit 208 inserts the marker 60 into the video into which the marker 60 is to be inserted. In S104, the video providing unit 212 transmits to the communication device 300 the video into which the marker insertion unit 208 inserted the marker 60 in S102.

In S106, the communication device 300 transmits the image received from the information processing device 200 in S104 to the image display device 400. In S108, the image display device 400 starts displaying the image received from the communication device 300 in S106 in the display target area 50.

In S110, the content delivery unit 214 delivers content to the display device 100. In S112, the imaging unit 120 captures the image displayed in the display target area 50. In S114, the marker recognition unit 108 recognizes, from the captured image captured by the imaging unit 120 in S112, a marker 60 inserted in the image for a period shorter than the period recognizable by the user of the display device 100. In S116, the display position determination unit 109 determines the display position of the content to be displayed on the display unit 126 based on the marker 60 recognized by the marker recognition unit 108 in S114. In S118, the display unit 126 displays the content at the display position determined by the display position determination unit 109 in S116.

While the image display device 400 is displaying an image in the display target area 50, steps S110 to S118 may be executed repeatedly. In S120, the image display device 400 ends the display of the image in the display target area 50, and processing by the system 10 ends.

14 shows an example of a hardware configuration of a computer 1200 functioning as an information processing device 200 or a display device 100. A program installed on the computer 1200 can cause the computer 1200 to function as one or more "parts" of the device according to the above embodiment, or cause the computer 1200 to execute operations or one or more "parts" associated with the device according to the above embodiment, and/or cause the computer 1200 to execute a process or steps of the process according to the above embodiment. Such a program can be executed by the CPU 1212 to cause the computer 1200 to execute specific operations associated with some or all of the blocks of the flowcharts and block diagrams described in this specification.

The computer 1200 according to this embodiment includes a CPU 1212, a RAM 1214, and a graphics controller 1216, which are connected to each other by a host controller 1210. The computer 1200 also includes input/output units such as a communication interface 1222, a storage device 1224, a DVD drive 1226, and an IC card drive, which are connected to the host controller 1210 via an input/output controller 1220. The DVD drive 1226 may be a DVD-ROM drive, a DVD-RAM drive, or the like. The storage device 1224 may be a hard disk drive, a solid state drive, or the like. The computer 1200 also includes legacy input/output units such as a ROM 1230 and a keyboard 1242, which are connected to the input/output controller 1220 via an input/output chip 1240.

The CPU 1212 operates according to the programs stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphics controller 1216 acquires image data generated by the CPU 1212 into a frame buffer or the like provided in the RAM 1214 or into itself, and causes the image data to be displayed on the display device 1218.

The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive 1226 reads programs or data from a DVD-ROM 1227 or the like and provides them to the storage device 1224. The IC card drive reads programs and data from an IC card and/or writes programs and data to an IC card.

ROM 1230 stores therein a boot program or the like executed by computer 1200 upon activation, and/or a program that depends on the hardware of computer 1200. I/O chip 1240 may also connect various I/O units to I/O controller 1220 via USB ports, parallel ports, serial ports, keyboard ports, mouse ports, etc.

The programs are provided by a computer-readable storage medium such as a DVD-ROM 1227 or an IC card. The programs are read from the computer-readable storage medium, installed in the storage device 1224, RAM 1214, or ROM 1230, which are also examples of computer-readable storage media, and executed by the CPU 1212. The information processing described in these programs is read by the computer 1200, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be constructed by realizing the operation or processing of information according to the use of the computer 1200.

For example, when communication is performed between computer 1200 and an external device, CPU 1212 may execute a communication program loaded into RAM 1214 and instruct communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of CPU 1212, communication interface 1222 reads transmission data stored in a transmission buffer area provided in RAM 1214, storage device 1224, DVD-ROM 1227, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes received data received from the network to a reception buffer area or the like provided on the recording medium.

The CPU 1212 may also cause all or a necessary portion of a file or database stored in an external recording medium such as the storage device 1224, DVD drive 1226 (DVD-ROM 1227), IC card, etc. to be read into the RAM 1214, and perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write back the processed data to the external recording medium.

Various types of information, such as various types of programs, data, tables, and databases, may be stored in the recording medium and may undergo information processing. The CPU 1212 may perform various types of processing on the data read from the RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search/replacement, etc., as described throughout this disclosure and specified by the instruction sequence of the program, and writes back the results to the RAM 1214. The CPU 1212 may also search for information in a file, database, etc. in the recording medium. For example, when multiple entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 1212 may search for an entry whose attribute value of the first attribute matches a specified condition from among the multiple entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

The above-described programs or software modules may be stored in a computer-readable storage medium on the computer 1200 or in the vicinity of the computer 1200. In addition, a recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the programs to the computer 1200 via the network.

The blocks in the flowcharts and block diagrams in this embodiment may represent stages of a process in which an operation is performed or "parts" of a device responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and/or a processor provided with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuitry may include digital and/or analog hardware circuits and may include integrated circuits (ICs) and/or discrete circuits. The programmable circuitry may include reconfigurable hardware circuits including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements, such as, for example, field programmable gate arrays (FPGAs) and programmable logic arrays (PLAs).

A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by a suitable device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture that includes instructions that can be executed to create means for performing the operations specified in the flowchart or block diagram. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), Blu-ray disks, memory sticks, integrated circuit cards, and the like.

The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk (registered trademark), JAVA (registered trademark), C++, etc., and conventional procedural programming languages such as the "C" programming language or similar programming languages.

The computer-readable instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, or a programmable circuit, either locally or over a local area network (LAN), a wide area network (WAN), such as the Internet, so that the processor of the general-purpose computer, special-purpose computer, or other programmable data processing apparatus, or the programmable circuit, executes the computer-readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

The present invention has been described above using an embodiment, but the technical scope of the present invention is not limited to the scope described in the above embodiment. It is clear to those skilled in the art that various modifications and improvements can be made to the above embodiment. It is clear from the claims that forms with such modifications or improvements can also be included in the technical scope of the present invention.

The order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before" or "prior to," and it should be noted that the processes may be performed in any order, unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is explained using "first," "next," etc. for convenience, it does not mean that it is necessary to perform the processes in that order.

10 System, 20 Network, 50 Display target area, 60 Marker, 62 Marker, 64 Marker, 66 Marker, 70 Content, 72 Content, 74 Content, 76 Content, 80 Captured image, 100 Display device, 102 User, 104 Information storage unit, 106 Information receiving unit, 107 Viewing angle determination unit, 108 Marker recognition unit, 109 Display position determination unit, 110 Frame, 112 Rim, 114 Bridge, 116 Lens, 118 Temple, 120 Capture unit, 122 Sensor, 124 Gaze detection unit, 126 Display unit, 128 Request receiving unit, 130 Information transmitting unit, 150 Communication terminal, 200 Information processing device, 202 Information storage unit, 204 Information receiving unit, 206 Request transmitting unit, 208 Marker insertion unit, 210, attention area identification unit, 212, video providing unit, 214, content distribution unit, 300, communication device, 400, video display device, 502, user, 1200, computer, 1210, host controller, 1212, CPU, 1214, RAM, 1216, graphic controller, 1218, display device, 1220, input/output controller, 1222, communication interface, 1224, storage device, 1230, ROM, 1240, input/output chip, 1242, keyboard

Claims (13)

a marker insertion unit that inserts a marker for determining a display position of content related to an image displayed on a display unit of a display device while an imaging unit mounted on the display device is imaging the image displayed in a display target area, into the image for a period shorter than a period recognizable by a user of the display device while the image is being displayed in the display target area by having a marker projection device project the marker;
an information storage unit for storing attention area information indicating an attention area in the video that is attracting attention;
The marker insertion unit determines an insertion position of the marker such that the marker is inserted in an area outside the area of interest indicated by the area of interest information.
Information processing device. a marker insertion unit that inserts a marker for determining a display position of content related to an image displayed on a display unit of a display device while an imaging unit mounted on the display device is imaging the image displayed in a display target area, into the image for a period shorter than a period recognizable by a user of the display device while the image is being displayed in the display target area by having a marker projection device project the marker;
an information receiving unit that receives line-of-sight information indicating a line of sight of the user of the display device;
an attention area specifying unit that specifies an attention area in the video where attention is focused based on the line of sight of the user indicated by the line of sight information,
the marker insertion unit determines an insertion position of the marker such that the marker is inserted in an area outside the area of interest specified by the area of interest specification unit.
Information processing device. a marker insertion unit that inserts a marker for determining a display position of content related to an image displayed on a display unit of a display device while an imaging unit mounted on the display device is imaging the image displayed in a display target area, into the image for a period shorter than a period recognizable by a user of the display device while the image is being displayed in the display target area by having a marker projection device project the marker;
an information receiving unit that receives user operation information indicating an operation of the user of the display device,
The marker insertion unit inserts the marker into the video based on the user operation information of the user of the display device.
Information processing device. a marker insertion unit that inserts a marker for determining a display position of content related to an image displayed on a display unit of a display device while an imaging unit mounted on the display device is imaging the image displayed in a display target area, into the image for a period shorter than a period recognizable by a user of the display device while the image is being displayed in the display target area by having a marker projection device project the marker;
an information receiving unit for receiving user-related information related to the user of the display device,
The marker insertion unit determines an insertion period for inserting the marker into the video based on the user-related information.
Information processing device. The information storage unit further stores color-related information related to colors in the image;
The marker insertion unit determines an insertion position of the marker based on the color-related information such that a color difference between a color of the marker and the color in the image becomes larger when the marker is inserted.
The information processing device according to claim 1 . The information storage unit further stores color-related information related to colors in the image;
the marker insertion unit determines a color of the marker based on the color-related information such that a color difference between the color of the marker and the color in the video becomes larger when the marker is inserted at an insertion position.
The information processing device according to claim 1 . The information processing apparatus according to claim 1 , further comprising: a content delivery unit that delivers the content to the display device. A program for causing a computer to function as the information processing device according to any one of claims 1 to 7 . An information processing device according to any one of claims 1 to 7 ;
and the display device. 1. A computer-implemented information processing method, comprising:
a marker insertion step of inserting a marker for determining a display position of content related to an image displayed on a display unit of the display device while an imaging unit mounted on the display device is imaging the image displayed in the display target area, into the image for a period shorter than a period recognizable by a user of the display device while the image is being displayed in the display target area by having a marker projection device project the marker;
The marker insertion step includes a determination step of determining an insertion position of the marker so that the marker is inserted in an area outside an attention area in the image where attention is focused, the attention area being indicated by attention area information stored in the computer.
Information processing methods. 1. A computer-implemented information processing method, comprising:
a marker insertion step of inserting a marker for determining a display position of content related to an image displayed on a display unit of a display device while an imaging unit mounted on the display device is imaging the image displayed in a display target area, into the image for a period shorter than a period recognizable by a user of the display device while the image is being displayed in the display target area by projecting the marker using a marker projection device;
receiving gaze information indicating a gaze of the user of the display device;
and an attention area identification step of identifying an attention area in the video where attention is focused based on the line of sight of the user indicated by the line of sight information,
The marker insertion step includes a determination step of determining an insertion position of the marker so that the marker is inserted in an area outside the area of interest identified in the area of interest identification step.
Information processing methods. 1. A computer-implemented information processing method, comprising:
a marker insertion step of inserting a marker for determining a display position of content related to an image displayed on a display unit of a display device while an imaging unit mounted on the display device is imaging the image displayed in a display target area, into the image for a period shorter than a period recognizable by a user of the display device while the image is being displayed in the display target area by projecting the marker using a marker projection device;
and receiving user action information indicative of an action of the user of the display device,
The marker insertion step includes inserting the marker into the image based on the user operation information of the user of the display device.
Information processing methods. 1. A computer-implemented information processing method, comprising:
a marker insertion step of inserting a marker for determining a display position of content related to an image displayed on a display unit of a display device while an imaging unit mounted on the display device is imaging the image displayed in a display target area, into the image for a period shorter than a period recognizable by a user of the display device while the image is being displayed in the display target area by projecting the marker using a marker projection device;
and receiving user related information relating to the user of the display device;
The marker insertion step includes a determination step of determining an insertion period for inserting the marker into the image based on the user-related information.
Information processing methods.

Images