JP7759801B2 - Event management server system and content image control method - Google Patents

Description

The present invention relates to an event management server system that provides services via mobile devices at event venues such as sporting events, and a content image control method implemented thereby.

Technology has been put into practical use to increase convenience at event venues, tourist facilities, and other venues by providing various information to visitors via their mobile devices. For example, technology has been proposed that notifies visitors' smartphones of the availability of restrooms at stadiums and the locations of food and drink vendors (see, for example, Patent Document 1). Another technology has been proposed that uses see-through displays and the like to display supplementary information about players of interest while watching a baseball game (see, for example, Patent Document 2). Meanwhile, technology has been proposed that films a game being held at a stadium from multiple viewpoints and distributes footage from viewpoints requested by a client (see, for example, Patent Document 3).

JP 2018-142080 A Special table 2019-531538 publication JP 2019-68130 A

The services mentioned above that event attendees can use on their mobile devices are often relatively simple, providing additional information in a one-way manner. As a result, there are few opportunities for attendees to look at or operate a device, particularly at events such as sporting events or music festivals where the audience's interest tends to be focused on the performance in real space, and situations in which the services can be used effectively are limited.

The present invention was made in light of these issues, and its purpose is to provide technology that enables event attendees to easily use appropriate services via their mobile devices.

One aspect of the present invention relates to an event management server system. This event management server system is characterized by comprising: a terminal information acquisition unit that acquires image data captured by a device held by a user who visits an event venue; a content image control unit that acquires user characteristics by comparing the captured image with registered data and selects a content image to be displayed on the terminal according to the characteristics; and an image data transmission unit that transmits data of the selected content image to the terminal for display.

Another aspect of the present invention relates to a content image control method. This content image control method is characterized by including the steps of acquiring image data captured by a terminal held by a user who has visited an event venue, acquiring user characteristics by comparing the captured image with registered data and selecting a content image to be displayed on the terminal in accordance with the characteristics, and transmitting data of the selected content image to the terminal for display.

In addition, any combination of the above components, or any transformation of the present invention into a method, device, system, computer program, or recording medium on which a computer program is recorded, is also valid as an aspect of the present invention.

This invention allows event attendees to easily access appropriate services via their mobile devices.

1 is a diagram showing the configuration of an event management system to which the present embodiment can be applied. 10A and 10B are diagrams illustrating examples of screens displayed by a mobile terminal in the present embodiment. 10A and 10B are diagrams illustrating examples of screens of a mobile terminal when promoting product sales using content images in the present embodiment. 10A and 10B are diagrams illustrating examples of the screen of a mobile terminal when the entertainment value of an event is improved by a content image in the present embodiment. FIG. 2 is a diagram illustrating an internal circuit configuration of an event management server according to the present embodiment. FIG. 2 is a diagram showing an internal circuit configuration of a mobile terminal according to the present embodiment. FIG. 2 is a diagram illustrating a functional block configuration of an event management server according to the present embodiment. FIG. 2 is a diagram showing a functional block configuration of a mobile terminal according to the present embodiment. 10A and 10B are diagrams for explaining an example of a matching process performed by a matching unit of a content image control unit in the event management server of the present embodiment. 10A and 10B are diagrams for explaining examples of content images that a content image control unit of the event management server causes to be displayed on a mobile terminal in the present embodiment. 10A and 10B are diagrams for explaining another example of a content image that the content image control unit of the event management server causes to be displayed on the mobile terminal in the present embodiment. 10A and 10B are diagrams illustrating an example of settings of the type of content image and the display trigger to be displayed on the mobile terminal by a content image control unit of the event management server in the present embodiment. 10 is a flowchart showing a processing procedure in which the event management server selects and transmits a content image to be displayed on the mobile terminal in the present embodiment.

This embodiment relates to a system that provides services to visitors at an event venue via mobile devices. An event here refers to any event where multiple people go to a venue to watch or experience something, and the event content, scale, and type of venue are not particularly limited, and could be, for example, a sporting event, music festival, concert, or show. Figure 1 shows the configuration of an event management system to which this embodiment can be applied. The event management system 1 includes an event management server 10, mobile devices 12a-12e, a virtual store management server 14, and a physical store management server 16.

Mobile terminals 12a-12e are terminals carried by event attendees (hereafter referred to as "users"), and display various information related to the event and accept user operations. In Figure 1, mobile terminals 12a, 12b, and 12c are high-function mobile phones (smartphones), and mobile terminals 12d and 12e are wearable terminals such as AR (Augmented Reality) glasses. As such, there are no limitations on the form of mobile terminals 12a-12e, as long as they are general terminals equipped with a display, an information processing device including communication functions, and an input device. Hereinafter, mobile terminals 12a-12e will be referred to simply as "mobile terminal 12."

The event management server 10 establishes communication with the mobile device 12, receives captured images and user operations, performs corresponding information processing, and returns necessary information. In recent years, with the introduction of 5G (fifth generation mobile communication systems), it has become possible to transmit a variety of data with low latency and large volumes, even via mobile devices. Therefore, such technology is appropriately used in communication between the mobile device 12 and the event management server 10.

Note that the event management server 10 may actually be realized by multiple information processing devices or storage devices. For example, the event management server 10 may be distributed according to function using MEC (Multi-access Edge Computing), a 5G technology. In this case, functions that require high responsiveness, such as the ability to send and receive image data with mobile devices 12, may be realized by a server installed near the event venue, while other functions may be realized by a server installed in a separate data center, etc. Therefore, the "event management server" in this embodiment can be read as an "event management server system" regardless of the number of devices.

The virtual store management server 14 establishes communication with the event management server 10 and mediates sales and purchases between virtual stores on the network and users. The physical store management server 16 establishes communication with the event management server 10 and mediates sales and purchases between users and physical stores inside and outside the event venue. The physical store management server 16 may also provide the event management server 10 with information on the real-time status of the physical store, such as congestion, waiting times, and product inventory, at any time.

In this embodiment, the event management server 10 provides various services to users via the mobile device 12. The services provided here can be broadly divided into mechanisms to make the event itself more enjoyable, and sales promotions for related products. In this embodiment, various information related to the services can be viewed along with the event situation via the mobile device 12. This allows users to use a variety of services without missing the real-time status of the event or losing interest.

Specifically, users hold up their mobile devices 12 to capture images of the event using the in-camera, and the real-time images are displayed on the display. Meanwhile, the event management server 10 collects and analyzes the images captured by the mobile devices 12 to estimate the characteristics and interests of each user using the mobile device 12. Based on the estimation results, the event management server 10 then generates an optimal image to be superimposed on the real-time images and displays them on each mobile device 12.

The images to be superimposed here may be represented as virtual objects rendered as if they existed at the event venue. Technologies such as AR and MR (Mixed Reality) can be used to blend virtual objects into images of real space. In this case, three-dimensional data representing the spatial structure of the event venue is used to define the placement of the virtual objects. In this embodiment, this three-dimensional data is also used to analyze captured images, making it possible to identify user characteristics and interests at low cost. Hereinafter, the images that the event management server 10 selects for each mobile device 12 and superimposes on the real-time video will be referred to as "content images."

Figure 2 shows an example of a screen displayed by the mobile device 12. In this example, and in Figures 3 and 4 described below, the event is a soccer match, and the user is watching the match taking place on the field from their seat. When the user holds up the mobile device 12 in this state, the mobile device 12 displays the real space as seen from that viewpoint, as shown in (a), for example. This situation is achieved, for example, by instantly displaying on the display a real-time image captured by a camera mounted on the back of the mobile device 12. If the user changes the orientation of the mobile device 12, the field of view of the image will naturally change accordingly.

(b) shows the state in which a content image is superimposed on the real-time video shown in (a). In this example, a cylindrical virtual signage image 200 is superimposed as the content image. The virtual signage is an object resembling an electronic scoreboard, with various information such as scores, player images, and advertisements displayed on its sides and interior walls, making it appear as if it is hanging from the sky. The virtual signage image 200 is rendered by mapping the necessary information as a texture onto a three-dimensional model representing the signage shape. However, this is not intended to limit the type, shape, or rendering method of the object represented by the content image.

The virtual signage image 200 may be a moving image, with the presented information rotating on the side of a cylinder. To achieve this type of AR, the event management server 10 first stores three-dimensional data of the spatial structure of the soccer stadium, which is the event venue. The location of the virtual signage in the same three-dimensional space is then determined, and the location information is notified to the mobile device 12. The mobile device 12 virtually places the signage object at the corresponding position in real space and projects it onto the camera's screen coordinate system, thereby rendering the image 200 on the real-time video being displayed.

This allows the virtual signage to be fixed in a predetermined position in real space regardless of the camera's field of view, allowing content images to be displayed while avoiding important parts of the event, such as the soccer field. By converting the three-dimensional data of the spatial structure into a highly accurate data format such as point cloud data, content images can be displayed with high precision in a variety of locations, such as virtually displaying a logo on the underside of the roof of a soccer stadium.

If the user selects any of the information displayed on the virtual signage image 200, the event management server 10 may acquire the user's operation and then cause the mobile terminal 12 to additionally display a content image showing other information related to the selected information. For example, when a sponsor's logo displayed on the virtual signage is selected, the event management server 10 may additionally display an advertising video for that sponsor. Furthermore, when a player's facial image is selected, the event management server 10 may additionally display an image showing the player's profile, track record, related merchandise, etc.

Similar displays may be triggered by a user pointing to an image of a real object shown in real-time video. For example, if a user points to a signboard image 201 installed in the audience seats opposite, the event management server 10 will display a related promotional video. The event management server 10 may also display a GUI (Graphical User Interface) that accepts the purchase of related products in response to a user's pointing operation on the virtual signage image 200 or the image of the real object.

The content image may be constantly displayed while the user is holding the mobile terminal 12 over the event venue, or may be switched between visible and hidden in response to a specific user operation as described above. Specific operations may include general instruction operations such as tapping on the screen, as well as shooting operations such as changing the zoom magnification or taking a still image. Alternatively, the content image may be switched between visible and hidden based on factors other than user operation, such as the status of the participants or the status of the event.

Figure 3 shows an example of the screen of a mobile terminal 12 when promoting product sales using content images. In (a), sales information 206 for the product, cleats, and a GUI 208 for accepting purchases are superimposed on real-time video. For example, when a user selects an image of a player displayed on the virtual signage image 200 shown in Figure 2 (b), sales information 206 and a GUI 208 are displayed for the same model of cleats worn by the player, along with information 202 about the player. These content images may be objects with three-dimensional models, or flat images.

The spike image included in the sales information 206 may be displayed using MR technology, allowing the user to change the orientation of the three-dimensional object or pull it closer to them, allowing for detailed confirmation. In the illustrated example, the GUI 208 is represented by a horizontally elongated three-dimensional object displaying the words "Buy this." When a user operates the GUI 208, the event management server 10 accepts the operation and performs order processing and payment processing on the virtual store management server 14 or physical store management server 16 that manages the seller.

Once the ordering and payment processes are complete, the event management server 10 superimposes an image 210 indicating the completion of the purchase on the mobile device 12, as shown in (b). In this example, the image 210 is represented by a horizontally elongated three-dimensional object displaying the words "Purchase Completed." Products purchased in this way may be available at the venue or delivered at a later date, depending on the type and where they were purchased. In the former case, the product may be delivered to the attendee's seat, or the attendee may pick it up at a store.

Figure 4 shows an example of the screen of a mobile device 12 when content images are used to enhance the entertainment value of an event. (a) is an example of the screen displayed when a goal is scored, with information 220 about the player who scored and an image 222 showing a virtual celebration of the goal using lights and text superimposed as content images on the real-time video. (b) shows a real-time image 224 of a player on the field superimposed as a content image at any time during the game. By enlarging and displaying the player image 224 as shown, players who are difficult to see from their seats can be viewed in detail.

In particular, in the case of content images such as those shown in Figure 4, whether a user finds the display results desirable or bothersome will largely depend on their individual preferences, such as the team they support or the player they are watching. Also, depending on the user's seat and the position of the player, there may be no need to enlarge the player's image. Furthermore, the information that users want to see or the products they want to purchase can change from moment to moment depending on factors that are difficult to predict in advance, such as the game situation, weather, temperature, the behavior of other spectators, and the user's individual physical condition.

In light of this, in this embodiment, the event management server 10 collects information such as images captured by the mobile device 12, the details of user operations such as shooting operations and instruction operations, and user profiles, and uses this information to estimate the characteristics and interests of individual users. Furthermore, taking into account the status of participants and the venue, the content images to be displayed and the timing of their display are optimized. This minimizes the effort required for advance registration and menu operations, allowing users to enjoy the event more and easily purchase desired products via the mobile device 12.

Figure 5 shows the internal circuit configuration of the event management server 10. The event management server 10 includes a CPU (Central Processing Unit) 23, a GPU (Graphics Processing Unit) 24, and a main memory 26. These components are interconnected via a bus 30. An input/output interface 28 is also connected to the bus 30. Connected to the input/output interface 28 are a communication unit 32 that establishes communication with the mobile terminal 12, virtual store management server 14, and physical store management server 16, a storage unit 34 such as a hard disk drive or non-volatile memory, an output unit 36 that outputs data to a display (not shown), an input unit 38 that inputs data from an input device (not shown), and a recording medium drive unit 40 that drives removable recording media such as a magnetic disk, optical disk, or semiconductor memory.

The CPU 23 controls the entire event management server 10 by executing an operating system stored in the storage unit 34. The CPU 23 also executes various programs read from removable recording media and loaded into the main memory 26. The GPU 24 performs drawing processing in accordance with drawing commands from the CPU 23 and outputs the results to the output unit 36. The main memory 26 is composed of RAM (Random Access Memory) and stores programs and data required for processing.

Figure 6 shows the internal circuit configuration of the mobile terminal 12. The mobile terminal 12 includes a CPU 50, main memory 52, display unit 54, input unit 56, audio output unit 59, communication unit 60, motion sensor 62, camera 64, and position acquisition unit 66. These units are interconnected via a bus 58. The CPU 50 processes information acquired from each unit of the mobile terminal 12 via the bus 58, and supplies output data to the display unit 54 and audio output unit 59. The main memory 52 stores programs and data necessary for processing by the CPU 50.

The display unit 54 is composed of a display panel such as an LCD panel or an organic EL panel and its drive mechanism, and displays images such as those shown in Figures 2 to 4. The input unit 56 is composed of a touchpad that covers the screen of the display unit 54 and hardware keys attached to the housing, and accepts instructions for the displayed image and shooting operations such as changing the zoom magnification and taking still images. The audio output unit 59 is composed of a speaker or earphones and their control mechanism, and allows the user to hear audio as needed.

The communication unit 60 establishes communication with the event management server 10 and sends and receives necessary data. In this embodiment, mobile communication for the mobile terminal 12 is achieved using, for example, 5G communication technology. The motion sensor 62 is composed of at least one of an acceleration sensor, gyro sensor, geomagnetic sensor, etc., and measures the posture and movement of the mobile terminal 12 in three-dimensional space. The camera 64 captures video of the real space in the opposite direction to the screen of the display unit 54. In this embodiment, in particular, the video is instantly displayed on the display unit 54, allowing the state of the real space to be viewed in real time. Data on the captured image is also transmitted to the event management server 10 via the communication unit 60 and used to select content images.

The position acquisition unit 66 constantly acquires the position information of the device itself using well-known technologies such as GPS (Global Positioning System) and A-GPS (Assisted-GPS). Measurement values from the motion sensor 62 and position acquisition unit 66, as well as images captured by the camera 64, can be used to determine a screen coordinate system for real space when rendering virtual objects, which are content images.

Figure 7 shows the functional block configuration of the event management server 10 in this embodiment. Each functional block shown in Figure 7 and Figure 8 described below can be realized in hardware terms by electronic components such as the CPU and main memory shown in Figure 5 or Figure 6, and in software terms by programs that perform various functions such as data input, data retention, image processing, and communication, loaded into main memory from a recording medium, etc. Therefore, those skilled in the art will understand that these functional blocks can be realized in various forms using hardware alone, software alone, or a combination of both, and are not limited to any one of these.

The event management server 10 includes a terminal information acquisition unit 70 that acquires captured image data and user operation details from the mobile terminal 12, a content image control unit 72 that controls the content images displayed on the mobile terminal 12, a sales processing unit 74 that implements product sales procedures, an information storage unit 76 that stores information necessary for content image control and product sales procedures, a participant status information acquisition unit 78 that acquires specified information related to the status of event participants, an event status information acquisition unit 80 that acquires specified information related to the event status, and an image data transmission unit 82 that transmits content image data.

The terminal information acquisition unit 70 acquires real-time image data captured by the camera 64 of the mobile terminal 12. The terminal information acquisition unit 70 also acquires the content of user operations on the mobile terminal 12 at any time. Here, user operations include, for example, shooting operations such as changing the zoom magnification of the camera 64 or shooting a still image, instruction operations on an image being displayed on the mobile terminal 12, and product purchase operations. The terminal information acquisition unit 70 may continuously acquire captured image data at predetermined time intervals while the mobile terminal 12 is shooting images of the event venue, or may acquire captured image data only when the user performs a predetermined operation, in association with the content of that operation.

The terminal information acquisition unit 70 acquires captured image data and information related to user operation details in association with the identification information of the mobile terminal 12, allowing subsequent processing to be performed individually for each mobile terminal 12. The content image control unit 72 selects and generates content images to be displayed on each mobile terminal 12 appropriately for each individual user based on the data and information acquired by the terminal information acquisition unit 70. In more detail, the content image control unit 72 includes a matching unit 90, an image selection unit 92, and an image generation unit 94.

The matching unit 90 estimates the characteristics and interests of the user using the mobile device 12 by matching the data and information acquired by the device information acquisition unit 70 with three-dimensional data showing the spatial structure of the event venue, or with pre-registered template images of participants, logos, etc. For example, the matching unit 90 identifies the location of the mobile device 12, and ultimately the user, by matching data on the spatial structure of the event venue with a captured image.

In this case, "matching" is achieved, for example, by searching for three-dimensional data of the spatial structure based on the distribution of feature points in the captured image, and deriving the position and orientation of the mobile device 12 when the most matching angle of view is obtained. However, this is not the only way to identify the user's position; output values from the motion sensor 62 or position acquisition unit 66 provided in the mobile device 12 may also be used, or these output values may be combined with the matching results of the captured image to improve accuracy.

The matching unit 90 may use the identified user's position to identify secondary information. For example, the matching unit 90 may derive the relationship between the distance and orientation between the subject, an event participant, and the user. Alternatively, the matching unit 90 may obtain the type of seat that includes the identified position, thereby determining the team the user is supporting or estimating a profile based on the seat rank.

The matching unit 90 may also infer the person or object the user is paying attention to by matching the image of the subject in the captured image with pre-prepared image or text information. For example, the matching unit 90 may identify the contestant of interest by extracting a facial area from the captured image and matching it with facial images of registered contestants, or by detecting the jersey number or name shown on the uniform and matching it with registered information. Alternatively, the matching unit 90 may detect signs or product logos appearing in the captured image and match them with registered company or product logo images to identify the company or product in which the user is currently interested. Technologies used in fields such as object detection and object recognition can be used to "match" images in this way.

The timing at which the matching unit 90 performs the above-mentioned matching is not limited. For example, the matching unit 90 may perform the matching only once at the start of the event, or may perform the matching periodically while captured image data is being transmitted from the mobile terminal 12. Alternatively, the matching unit 90 may perform the matching when the event situation, the status of the participants, the content of the user operation, etc., satisfy predetermined conditions. For example, when the user performs a zoom-in operation, the matching unit 90 may use the image of the subject at the center of the zoom as the target for matching.

The image selection unit 92 selects a content image to be displayed on the mobile terminal 12 based on the results of the comparison by the comparison unit 90. For example, when the team the user is rooting for scores, the image selection unit 92 selects a content image representing the virtual performance shown in FIG. 4(a). Also, when a contestant the user is paying attention to is located far away from the user, the image selection unit 92 selects an enlarged image such as that shown in FIG. 4(b). Alternatively, when a sign in the venue is zoomed in, the image selection unit 92 selects an advertisement for a related company or product, or a GUI for purchasing related products such as that shown in FIG. 3, as the content image.

The image generation unit 94 generates the content image selected by the image selection unit 92 in a state that can be displayed on the mobile terminal 12. When the content image is represented as a three-dimensional object that blends with real space, the image generation unit 94 associates the model data of the three-dimensional object with its position in a three-dimensional coordinate system that represents the event venue. Furthermore, the image generation unit 94 may generate texture data to be mapped to the three-dimensional object based on real-time information, or may cut out a real-time image of a specific contestant from live-action footage. Alternatively, the image generation unit 94 may render the content image as a two-dimensional image.

The participant status information acquisition unit 78 continuously acquires status information including at least one of the position and orientation of the participants in the event. For example, the participant status information acquisition unit 78 acquires real-time video footage of the participants from multiple viewpoints from an external camera (not shown), and analyzes this video to derive status information such as the participant's position, orientation, speed, and type of movement at the event venue. Such image analysis is widely used in fields such as sports broadcasting.

The contestant status information acquisition unit 78 sequentially provides contestant status information and video data to the content image control unit 72. The content image control unit 72 uses this information and data to control the content image. For example, the content image control unit 72 determines whether to display an enlarged image of the contestant depending on the distance between the user and the contestant. Alternatively, the content image control unit 72 changes at least one of the position and direction of the virtual viewpoint when generating the content image so as to show the appearance of the contestant facing away from the user.

This allows for improved visibility regardless of seating, even in large venues where it may be difficult to see the contestants. The content image control unit 72 may also control the display to display virtual effects that emphasize the quality of a contestant's performance, or commentary about the performance, depending on the quality of the performance.

The event status information acquisition unit 80 acquires event status information such as weather, time of day, season, temperature, humidity, the progress of the match, and the progress of the show. The event status information may be entered as needed by the administrator of the event management server 10 or the event organizer before the start of the event or during the event, or may be determined from the output values of various sensors (not shown), such as a thermometer, illuminometer, and hygrometer. Alternatively, the event status information acquisition unit 80 may acquire status information by analyzing captured images sent from the mobile terminal 12 or real-time video captured by an external camera (not shown).

The event status information acquisition unit 80 provides event status information to the content image control unit 72 as appropriate. The content image control unit 72 uses this information to control the content images. For example, the content image control unit 72 may display a content image accepting the purchase of rain gear in response to rainfall during the event, or a content image accepting the purchase of drinking water in response to a rise in temperature.

In response to a point being scored, the content image control unit 72 may also cause the mobile device 12 of a user who is cheering for the team that scored to display a content image showing a virtual effect such as that shown in FIG. 4(a) or information about the scorer. Furthermore, the content image control unit 72 may cause the mobile device 12 of a user who is cheering for the team that scored to display a discount coupon that can be used to purchase related products. In this case, the discount rate may be changed depending on the game situation, such as increasing the discount rate as the number of points increases.

The matching unit 90 of the content image control unit 72 may also switch the spatial structure data to be matched with the captured image based on the situation information. For example, in the case of a stadium with a retractable roof, the state of the roof changes depending on the weather. For this reason, spatial structure data is prepared in multiple patterns with different roof states, and the matching unit 90 selects the data representing the corresponding roof state based on the weather included in the situation information and uses it for matching. This makes it possible to stabilize the accuracy of the matching results. The event situation information acquisition unit 80 may also acquire the state of the spatial structure itself, such as whether the roof is open or closed, as situation information from the captured image, etc.

Furthermore, when a content image is represented using AR, the switching of spatial structure data based on situation information may be reflected even at the stage when the image selection unit 92 selects a content image. This avoids problems such as displaying a virtual logo as a content image in an attic location that does not actually exist. The above-mentioned contestant status information and event situation information may be used by the content image control unit 72 to select content, and may also be used as a trigger for displaying a content image.

The buying and selling processing unit 74 carries out the buying and selling procedure in response to a purchase operation on the GUI for purchasing a product, which is displayed as a content image on the mobile terminal 12. At this time, the buying and selling processing unit 74 acquires pre-registered personal information of the user of the mobile terminal 12, thereby eliminating the need for the user to re-enter their personal information on the spot. Here, personal information includes information necessary for payment, such as name, credit card number, and security code, as well as information such as the shipping address and telephone number if the product is to be mailed.

As mentioned above, the product to be purchased may be picked up by the user by visiting a store within the venue, or it may be delivered from the store to the user's seat. In the latter case, the buying and selling processing unit 74 may identify the delivery address by referencing the user's seat information obtained as a result of the matching by the matching unit 90. In either case, the buying and selling processing unit 74 establishes communication with the virtual store management server 14 or the physical store management server 16 in response to the user's purchase operation, and performs order processing and payment processing. Conventional technology can be applied to the electronic commerce system using payment methods such as credit cards and electronic money.

If a purchase operation is performed without specifying a store, or if the target product is sold at multiple chain stores, the buying and selling processing unit 74 may select the store closest to the user's location or seat, as determined by the matching unit 90, as the order destination. After placing the order, the buying and selling processing unit 74 may also cause a content image notifying the user that the order has been completed to be displayed on the purchasing mobile terminal 12 via the content image control unit 72. Through these procedures, the user can purchase the desired product with simple operations and with minimal obscuration of the real-time event video, as shown in Figure 3.

After placing an order with a store within the venue, the buying and selling processing unit 74 may obtain information such as the store's location, order number, and waiting time from the ordering store, and display a content image representing this information on the mobile terminal 12 from which the purchase was made via the content image control unit 72. The buying and selling processing unit 74 may also obtain store status information such as the level of congestion, product inventory, and waiting time at physical stores within the venue or nearby, as needed, and display a content image representing this information on the mobile terminal 12 via the content image control unit 72.

The information storage unit 76 stores information necessary for the content image control unit 72 and the trading processing unit 74 to perform the processing described above. In detail, the information storage unit 76 includes a user information storage unit 100, a related information storage unit 102, a spatial information storage unit 104, a matching rule storage unit 106, and an image data storage unit 108. The user information storage unit 100 stores identification information of the mobile terminal 12 in association with personal information of the user. As described above, the personal information is used by the trading processing unit 74 in the product trading procedures, and can also be used by the content image control unit 72 when selecting a content image.

For example, the user information storage unit 100 stores at least one of the user's age, gender, address, purchase history, event participation history, etc. as personal information. By referencing this information, the content image control unit 72 can optimize the products to be displayed as content images, and by integrating this information with the matching results by the matching unit 90, it can improve the accuracy of identifying the user's characteristics and target of interest. For example, if multiple contestants or products appear in the captured image, the matching unit 90 may narrow down which contestants or products are being focused on based on the personal information described above.

The related information storage unit 102 stores template data used by the matching unit 90 for matching, in association with information about the subject it represents. Examples of template data used for matching here include images of participants' faces, uniforms, and costumes, team logos, text information indicating team names, images of equipment such as balls, images of signboards, product or company logos, and marks.

Each image and text information is associated with the profile of the contestant or team it represents, the company profile, product sales information, etc. The matching unit 90 estimates the contestant, company, or product that is of interest to the user based on the results of matching the captured image with template data. This allows the image selection unit 92 to appropriately select the contestant, advertisement, sales information, etc. to be displayed as the content image.

The related information storage unit 102 may store product sales information in association with the display conditions of the related content image. The display conditions can be set, for example, for the status information of the contestants or the situation information of the event. This allows the image selection unit 92 to display product sales information at the appropriate time without relying on user operation, such as displaying a GUI for purchasing rain gear in response to rainfall or displaying discount coupons in response to points scored.

The related information storage unit 102 may also store the sales information of the product, such as the URL (Uniform Resource Locator) of the sales store and the location of the physical store. This allows the buying and selling processing unit 74 to place an order with the appropriate store via the virtual store management server 14 or the physical store management server 16.

The spatial information storage unit 104 stores data that represents the structure of the real space of the event venue as three-dimensional information. There are no particular restrictions on the data format, but one possible format is point cloud data, which represents the surface positions of objects that exist in the real space of the venue using the three-dimensional position coordinates of a group of points that are discretely distributed on that surface. By using point cloud data, the position and orientation of virtual objects can be precisely specified using the same three-dimensional coordinate system, allowing for a highly accurate representation of the fusion of real space and virtual objects.

As mentioned above, in this embodiment, the spatial structure data is also used to identify the user's position by matching it with the captured image. The spatial information storage unit 104 further stores the divided regions of the spatial structure data in association with real-world categories such as seats, fields, stages, aisles, and stores. This makes it possible to identify the characteristics of a user according to their seat and narrow down the direction in which the camera of the mobile device 12 is facing.

Furthermore, as mentioned above, if the structure of the event venue is variable, the spatial information storage unit 104 may store multiple patterns of spatial structure data, and the matching unit 90 may switch between the data used for matching in accordance with predetermined rules. For this reason, the spatial information storage unit 104 may store selection criteria set in situation information, etc., in association with each piece of spatial structure data.

The matching rule storage unit 106 stores the rules for the matching process performed by the matching unit 90. The matching rules specify the objects to be matched with the captured images, the trigger for matching, and the objects to be identified by matching, as shown in the following table, for example.

In this example, when the application is launched on the mobile device 12, the captured image is compared with the spatial structure of the event venue to identify the user's position and seat. Furthermore, when the mobile device 12 is used to zoom in or to specify the displayed image, the target image is compared with a template image to identify the contestant, related company, or product of interest.

The table above also shows the types of content images that the image selection unit 92 will select based on the identified information. Specifically, by identifying the user's location, seat, and the contestant they are focusing on, the system limits the teams and contestants that are the target of event content, i.e., those that improve visibility and provide virtual effects for their performances. In other words, content images for teams and contestants that the user is not rooting for will not be displayed, reducing the likelihood that they will be perceived as excessive or unpleasant.

Furthermore, by identifying companies and products of interest, promotional content, i.e., sales information and advertising videos, can be displayed and purchases can be limited. This avoids the annoyance of seeing information about companies and products that are not of interest, and increases sales effectiveness with less information. However, the above table is an example and is not intended to limit the scope of this embodiment. For example, content images for purchasing merchandise related to a contestant of interest can also be positioned as promotional content.

The image data storage unit 108 stores image data used by the image generation unit 94 to generate content images. The image data storage unit 108 stores, for example, three-dimensional model data for rendering virtual objects, texture data, and layout information for the event venue in three-dimensional space. The image data storage unit 108 also stores content image formats showing information about contestants and products, image data for products and advertisements, and the like. This data is associated with information about contestants, companies, and products stored in the related information storage unit 102.

The image data transmission unit 82 transmits data for the content image selected and generated by the content image control unit 72 to the mobile terminal 12. The data transmitted by the image data transmission unit 82 may be the content image itself, or partial data necessary for the mobile terminal 12 to interpret and render the image, such as the placement of virtual objects in real space and the text information represented therein.

In the latter case, the mobile terminal 12 places the object in a coordinate system representing real space based on the 3D model held internally and the transmitted position information, and then draws the image in a screen coordinate system formed by the display screen. Alternatively, the image data transmission unit 82 may generate an image by making areas other than the content image transparent, and then transmit the image to the mobile terminal 12. In this case, the mobile terminal 12 simply needs to overlay the transmitted image on the real-time video. Those skilled in the art will understand that there are various possibilities for dividing the data held by the event management server 10 and the mobile terminal 12 and the processing they handle.

Figure 8 shows the functional block configuration of the mobile terminal 12 in this embodiment. All or part of the functions shown in the figure may be configured by downloading from a distribution server that distributes applications, based on user selection or autonomous request control of the mobile terminal 12, or by installing the applications when the mobile terminal 12 is shipped. The mobile terminal 12 may also be configured as a terminal with an application installed, such as a high-function mobile phone, and a wearable display, such as AR glasses, that displays images. In this case, the two may send and receive data via short-range wireless communication, such as Bluetooth (registered trademark), or they may be connected by wire.

The mobile terminal 12 includes an operation reception unit 110 that receives user operations, a photographing unit 111 that photographs images of the real space, a terminal information transmission unit 112 that transmits data on the photographed images and the content of user operations to the event management server 10, a status information acquisition unit 114 that acquires information related to the status of the mobile terminal 12 itself, an image data reception unit 116 that receives content image data from the event management server 10, and a display unit 118 that displays real-time video and content images.

The mobile terminal 12 further includes a personal information storage unit 120 that stores the user's personal information, and an image data storage unit 122 that stores data used to display content images. The operation reception unit 110 receives user operations such as pointing operations and shooting operations for images displayed on the display unit 118. User operations may include tapping and pinching on the screen, as well as operations that use MR technology to virtually point to virtual objects in real space within the camera's field of view, and hand gestures within that field of view.

The photographing unit 111 includes the camera 64 shown in Figure 5 and photographs an image of the real space in which the event is taking place. In this embodiment, the basic principle is to photograph a video of the event while the user is holding up the mobile terminal 12. The terminal information transmission unit 112 transmits the images photographed by the photographing unit 111, i.e., the video frame data, and the content of the user operation accepted by the operation acceptance unit 110, together with the identification information of the mobile terminal 12, to the event management server 10.

The personal information storage unit 120 therefore stores at least the identification information of the mobile device 12. The personal information storage unit 120 may also store personal information typically recorded on a device, such as the user's name, phone number, login ID, and password. The status information acquisition unit 114 includes the motion sensor 62 and position acquisition unit 66 shown in FIG. 5, and constantly acquires the position, posture, and movement of the mobile device 12 as status information. This status information may be transmitted to the event management server 10 via the terminal information transmission unit 112.

In this case, the event management server 10 may more accurately determine the user's position and orientation by integrating the matching results using the captured image with the status information of the mobile device 12. The image data receiving unit 116 receives content image data from the event management server 10. The content image data may be received intermittently, such as once or multiple times as needed, or may be received continuously at predetermined time intervals.

The display unit 118 displays real-time video captured by the imaging unit 111, and also superimposes a content image based on the data received by the image data receiving unit 116. For example, the display unit 118 renders the content image by projecting a three-dimensional model of the object to be displayed as the content image onto a screen coordinate system corresponding to the position and orientation of the mobile terminal 12 acquired by the state information acquisition unit 114. The display unit 118 may also move or deform the object displayed as the content image so that it interacts with the user's finger, etc.

The image data storage unit 122 stores some of the data necessary for displaying content images. As mentioned above, the data stored in the image data storage unit 122 may vary depending on the division of processing between the mobile terminal 12 and the event management server 10. Note that the content images displayed by the display unit 118 are not limited to being fixed to a coordinate system in real space. For example, depending on the type of image and the timing of display, images such as corporate advertising videos, images for purchasing products, and discount coupons may be displayed at a predetermined position in the screen coordinate system.

Figure 9 is a diagram illustrating an example of the matching process performed by the matching unit 90 of the content image control unit 72 in the event management server 10. The upper part of (a) shows part of the spatial structure 230 of a soccer stadium, which is the event venue. This data may actually be represented in the form of point cloud data in a three-dimensional coordinate system with axes set at predetermined positions and directions, and is stored in advance in the spatial information storage unit 104.

When a captured image 232 is sent from the mobile device 12, the matching unit 90 extracts feature points from the image and identifies the range 234 of the spatial structure that provides the highest matching rate, as shown by the arrow. (b) shows a schematic bird's-eye view of a soccer stadium, with spectator seats 238 arranged around a field 236. Based on the matching results shown in (a), the matching unit 90 identifies the position of the mobile device 12 that provides a field of view within range 234, and ultimately the user's position 240.

The matching unit 90 may further identify the category of seat corresponding to the position 240 based on information associated with the spatial structure data. In this example, the user's position 240 corresponds to a seat behind the goal post 242, typically reserved for supporters of one of the teams. Which team's supporter seat it is is uniquely determined by the match settings.

As a result, by comparing the captured image with the spatial structure data, the matching unit 90 can identify the user's location and the team they are supporting. In addition, if seats are ranked, it can estimate the range of expenditures the user is expecting for the event. Furthermore, if the user is in a special seat, it can be inferred that the user is a valued customer. These user characteristics can be used to determine the team to display in event content, the price range of products to display in sales promotion content, etc.

The matching unit 90 may obtain not only the user's position 240, but also the attitude of the mobile terminal 12 and, ultimately, the camera's line of sight 244 by matching the captured image with the spatial structure data. Techniques such as Visual SLAM (Simultaneous Localization and Mapping) may be used to obtain the camera's position and attitude from the captured image. By obtaining the line of sight 244, the content image control unit 72 may estimate that a real object, such as a contestant, present in the line of sight is the user's target of attention and use this information to select content images. This may allow the displayed content images to be changed one after another by, for example, moving the camera's field of view.

Figure 10 is a diagram illustrating an example of a content image that the content image control unit 72 in the event management server 10 displays on the mobile device 12. In this example, the event venue is assumed to be a soccer stadium, and the user is seated behind one of the goal posts, as shown in Figure 9 (b). (a) and (b) are real-time images displayed on the mobile device 12 from that seat.

In soccer, the goal posts that each team attacks are switched between the first and second halves. This means that users behind the goal posts have a limited period of time to watch their favorite team attack the goal. For example, in the first half, as shown in (a), they can see the players of their favorite team shoot from relatively close up, but in the second half, as shown in (b), the shots are shot at the opposite goal post, making it difficult to see. For this reason, it is likely that the excitement of cheering will be less intense in the second half compared to the first half.

Therefore, when the distance between the user and the player is greater than a predetermined value, the image selection unit 92 decides to display an enlarged image 250a of the player's image, as shown in (c). The distance to the player may be acquired by the player status information acquisition unit 78, as described above, or may be determined by the image selection unit 92 from the size of the player's image in the captured image. Furthermore, the target to be enlarged may be limited to a player on the team the user is supporting, as determined from the user's seat, or a player the user is paying attention to, as estimated by a zoom-in operation, etc.

Two or more thresholds may be set for the distance between the user and the player, and the magnification of the enlarged image 250a may be increased in stages each time the distance exceeds a threshold. Alternatively, no threshold may be set, and the magnification of the enlarged image 250a may be increased continuously as the distance increases. When the distance decreases, the magnification of the enlarged image 250a may be reduced or the enlarged image 250a may be hidden by the reverse control.

The enlarged image 250a shown in (c) is an enlarged image of the relevant player as seen from the same or approximately the same direction as the user's viewpoint. Alternatively, an enlarged image 250b may be displayed as seen from a direction that is not visible from the user's position. In this example, by displaying an image of the player as seen from the front, it becomes possible to see facial expressions that would not normally be visible. The user may also be allowed to freely control the orientation of the player displayed in the enlarged image. Technology that uses multi-view video to allow players to be viewed from any viewpoint during play has been put into practical use, and this may be applied.

Note that images viewed from a virtual viewpoint different from the user's viewpoint may be at the same magnification. Furthermore, the trigger for display does not have to depend on the distance between the user and the contestant. In any case, at least one of the magnification of the image and the displayed orientation of the contestant is changed depending on the relative positions and orientations of the user and the contestant. This allows any performer on stage, for example at a concert, to be viewed from any direction, reducing the difficulty of viewing caused by the angle of the seats.

For ease of understanding, the illustration shows only one player, but naturally there are multiple players on the field. Therefore, multiple players may be enlarged. In this case, the magnification may be adjusted to allow the relative positions of the players, i.e., the formation, to be confirmed. Alternatively, the player's image may be replaced with a graphic or similar image, and an overhead view may be shown to more clearly show the formation. Furthermore, once the player the user is focusing on has been identified as described above, information such as that player's performance and play details may be automatically displayed, regardless of whether the player is enlarged.

Figure 11 is a diagram illustrating another example of a content image that the content image control unit 72 in the event management server 10 displays on the mobile terminal 12. (a) shows real-time video footage of a user pointing the camera at the spectator seats in a soccer stadium. Here, the user focuses on a product 260 held by a customer and zooms in on the vicinity of the product, as shown in (b).

When the event management server 10 acquires information about such shooting operations and data on the captured image at that time, the matching unit 90 of the content image control unit 72 matches the zoomed-in captured image with images of companies and products registered in the related information storage unit 102. In this way, the matching unit 90 identifies information about the company that manufactures product 260 and the product itself. In the illustrated example, it identifies product 260 as French fries called "AA Potatoes" based on the character string "AA," the color of the rectangle surrounding it, the shape and size of the package, etc.

In response, the image selection unit 92 selects a content image associated with the relevant product. In the illustrated example, an image 262 representing a free coupon for "AA Potatoes" is superimposed. For example, when the user taps the coupon image 262 on the screen of the mobile terminal 12, the mobile terminal 12 stores the image in memory in a state where it can be displayed again, thereby indicating that the coupon has been acquired. In this case, when the user goes to a store and presents the coupon, they will receive a free "AA Potatoes."

The content images displayed by photographing real objects are not limited to coupons; they can also be information about related companies, information about other products sold by those companies, or a GUI for purchasing. Similar content images can also be created by photographing signs displayed within the venue, as shown in Figure 2. On the other hand, as shown in Figure 11, by using products held by other spectators as recognition targets and as the basis for displaying content images, diversity can be expanded, leading to unexpected encounters with companies and products. Furthermore, being able to see actual items eaten or worn by people evokes images and further promotes sales.

In the illustrated example, the matching and content image display are triggered by a zoom-in operation by the user, but other shooting operations, such as zooming out or taking a still image, may also be used as the trigger. For example, during a zoom-out operation, if a real object becomes visible in its entirety as a result of the operation, the matching unit 90 may use the image of the real object as the target for matching. Furthermore, when a still image is taken, the matching unit 90 may match the still image with registered data, just as when zooming in.

Figure 12 shows an example of the type of content image and display trigger settings that the content image control unit 72 of the event management server 10 sets to display on the mobile terminal 12. This setting information is stored in advance, for example, in the internal memory of the image selection unit 92. As mentioned above, content images are broadly divided into event content and sales promotion content.

The former are images that improve the user's visibility and arouse feelings of support, and are preferably limited to images related to the team or player the user is supporting. The latter are images that promote sales of various products, and are preferably limited to images that the user is interested in or that suit the event situation and the user's characteristics. However, depending on the content being presented, there may be content images that are common to all users, such as the virtual signage image 200 shown in Figure 2.

In the example shown in the figure, the event content is set to display a virtual celebration when a goal is scored in a soccer game, etc., as well as a discount coupon for merchandise related to the scoring team. Additionally, while the distance between the user and a contestant is above a threshold, an enlarged image of the contestant is displayed, and when the user zooms in on or gives instructions to zoom in on the contestant, an image of the contestant from a different direction than the user's line of sight (another-viewpoint image) is displayed.

In addition, as promotional content, when a user zooms in on a sign or product within the venue, or when there is a change in the environment such as weather or temperature that exceeds a predetermined value, a GUI is set to be displayed that allows users to purchase the associated product. Also, when a user zooms in on a sign or product within the venue, a discount coupon for the associated product is set to be displayed. However, these settings are merely examples, and should be optimized according to the content of the event, the size and characteristics of the venue, etc.

In addition to the triggers for displaying content images, the settings are not limited to those shown in the figure; for example, it is also possible to set triggers for hiding displayed content images. In any case, the system is set up so that appropriate content images can be displayed flexibly depending on the situation of the event and the relative positions and orientations of the user and contestants, rather than just when the user intentionally gives the command. This allows users to naturally get excited and easily purchase desired products without losing focus on the event.

Next, we will explain the operations that can be achieved with the above configuration. Figure 13 is a flowchart showing the processing steps by which the event management server 10 selects and transmits content images to be displayed on the mobile terminal 12. First, when a user at the event venue launches an application that displays content images and performs authentication procedures such as logging in, the event management server 10 acquires the information entered by the user and performs authentication processing (S10). This allows the event management server 10 to reference the user's personal information stored in the user information storage unit 100, enabling smooth procedures such as the purchase and sale of products.

If authentication is successful, the event management server 10 notifies the mobile device 12, which then activates the camera on the mobile device 12 and displays real-time video on the display. The device information acquisition unit 70 of the event management server 10 then begins acquiring device information, such as data on images captured by the camera and information related to user operations (S12). In response, the content image control unit 72 begins matching the captured image with registered data (S14) and begins acquiring the user's characteristics and interests based on the matching results (S16).

Here, registered data includes three-dimensional data of the spatial structure of the event venue, and template images of real objects including participants, signs, and merchandise. Furthermore, user characteristics and interests include the user's location, seat, team they support, and signs and merchandise they are interested in. The user's characteristics and interests may be narrowed down by adding the user's personal information stored in the user information storage unit 100 to the results of matching the captured images. As mentioned above, matching of captured images may be performed at any time, such as at predetermined times or when a specified user operation is performed.

Then, the content image control unit 72 begins selecting a content image that suits the user's characteristics and interests (S18). The selection of a content image may also be performed whenever the user's characteristics and interests are acquired. However, rather than determining a specific content image each time, the content images may be gradually narrowed down depending on the acquired characteristics and interests.

For example, once the content image control unit 72 identifies the team the user is supporting based on their seat, it first extracts a set of event content for that team as selection candidates. Then, when the time comes for the user to select a specific content image, such as when the user zooms in or a player scores, the content image control unit 72 selects a specific content image from the selection candidates, such as an enlarged image of the player in question.

The content image control unit 72 waits until the timing arrives to select and display a specific content image (N at S20), and when that timing arrives (Y at S20), it generates the content image (S22) and transmits the data to the mobile terminal 12 (S24). Note that in the example shown in the figure, the image is generated and transmitted only once at the timing to display the content image, but this is not intended to limit the scope of this embodiment.

For example, generation and data transmission may be continuously repeated for one content image. In this case, these processes may be terminated by further determining the timing to hide the content image. Also, although not shown in the figure, when a purchase operation is performed on a GUI for purchasing a product displayed on the mobile terminal 12 as a content image, the event management server 10 may execute the buying and selling process as described above.

The content image control unit 72 then repeats the process of selecting content images, generating data, and transmitting it at appropriate times (N in S26, S20-S24). If it becomes necessary to stop the content image processing, for example, if the user closes the application, the event management server 10 will end the processing (Y in S26).

In this embodiment described above, the event management server controls the content images that are superimposed on real-time video of the real space displayed on devices carried by users attending the event. Here, the event management server collects images captured by the mobile devices and estimates the user's characteristics and interests based on these images, thereby optimizing the content images to be displayed. This allows users to enjoy the event and various presentations and easily use the presented information without being distracted by unnecessary content images.

In addition, by comparing 3D data of the spatial structure of the event venue with photographed images, the user's position and seat can be identified. By utilizing the 3D data originally required to express content images in AR, compatibility with AR technology is increased, making it possible to implement with low barriers to entry.

Furthermore, even without the user having to explicitly specify a menu or image, the system can select and display appropriate content images at the appropriate time when a shooting operation such as zooming in is performed, or when there is a change in the position or orientation of the user and contestants, or the event situation or environment. This allows users to access the information and images they want without missing an event, increasing opportunities to use the service via their mobile device.

The present invention has been described above based on an embodiment. The above embodiment is merely an example, and those skilled in the art will understand that various modifications are possible in the combination of each component and each treatment process, and that such modifications are also within the scope of the present invention.

1 Event management system, 10 Event management server, 12 Mobile terminal, 14 Virtual store management server, 16 Physical store management server, 23 CPU, 26 Main memory, 32 Communication unit, 50 CPU, 52 Main memory, 54 Display unit, 60 Communication unit, 62 Motion sensor, 64 Camera, 66 Position acquisition unit, 70 Terminal information acquisition unit, 72 Content image control unit, 74 Buying and selling processing unit, 76 Information storage unit, 78 Participant status information acquisition unit, 80 Event status information acquisition unit, 82 Image data transmission unit, 90 Collation unit, 92 Image selection unit, 94 Image generation unit, 110 Operation reception unit, 111 Photography unit, 112 Terminal information transmission unit, 114 Status information acquisition unit, 116 Image data reception unit, 118 Display unit, 120 Personal information storage unit, 122 image data storage unit.

Claims (18)

a terminal information acquisition unit that acquires data of images captured by a terminal held by a user who has visited the event venue;
a content image control unit that acquires the user's seat category by comparing the registered data with the captured image, and selects an image related to the user's seat category as a content image to be displayed on the terminal;
an image data transmission unit that transmits data of the selected content image to the terminal and displays it;
a spatial information storage unit that stores three-dimensional data of the spatial structure of the event venue as the registration data,
the content image control unit identifies a user's position by comparing the captured image with the spatial structure, and further acquires a seat category corresponding to the identified user's position based on information associated with the three-dimensional data of the spatial structure.
Event management server system. The event management server system of claim 1, wherein the content image control unit identifies the team the user is supporting from among the teams playing in the event based on the seat category. the content image control unit generates the content image as a virtual object arranged in the spatial structure;
The event management server system of claim 1 or 2, characterized in that the image data transmission unit transmits positional information of the virtual object relative to the three-dimensional data of the spatial structure in the content image data, thereby superimposing an image of the virtual object at a corresponding position in the captured image being displayed on the terminal. the spatial information storage unit stores data of the spatial structure in a plurality of patterns;
4. The event management server system according to claim 1, wherein the content image control unit switches the spatial structure data used for the matching in accordance with a predetermined rule. the terminal information acquisition unit further acquires information on a user operation for changing a zoom magnification of the captured image;
5. The event management server system according to claim 1, wherein the content image control unit selects the content image when a user operation to change the zoom magnification is performed. the terminal information acquisition unit further acquires information about a user operation to designate any one of the captured images being displayed on the terminal;
6. The event management server system according to claim 1, wherein the content image control unit selects the content image corresponding to a specified image. An event management server system as described in any one of claims 1 to 6, characterized in that the content image control unit identifies a subject that the user is paying attention to through the matching and selects the content image corresponding to that subject. a participant status information acquisition unit that acquires status information including the location of the participant at the event venue;
8. The event management server system according to claim 1, wherein the content image control unit changes the content image in accordance with a positional relationship between the user and the contestant. The event management server system described in claim 8, characterized in that the content image control unit selects an enlarged image showing the appearance of the contestant as the content image and changes the magnification ratio of the enlarged image depending on the positional relationship. An event management server system as described in claim 8 or 9, characterized in that the content image control unit selects, as the content image, an image showing the contestant as seen from a virtual viewpoint different from the user's viewpoint, determined based on the positional relationship. further comprising an event status information acquisition unit that acquires a status of the event;
11. The event management server system according to claim 1, wherein the content image control unit changes the content of the content image depending on the status of the event. The event management server system described in claim 11, characterized in that the content image control unit selects an image of a discount coupon for a related product as the content image and changes the discount rate indicated by the discount coupon depending on the status of the event. An event management server system as described in claim 11 or 12, characterized in that the content image control unit selects, as the content image, an image that represents a virtual performance according to the situation of the event. An event management server system as described in any one of claims 1 to 13, characterized in that the content image control unit selects, as the content image, an image representing information about a product related to the subject appearing in the captured image. The event management server system described in claim 14, characterized in that the content image control unit determines the related products by detecting signs or products corresponding to the registration data from the captured image. the content image control unit selects a content image that accepts purchase of the related product;
the terminal information acquisition unit further acquires information on a purchase operation of the related product performed on the terminal;
The event management server system includes:
a user information storage unit for storing correspondence information between the terminal and the user's personal information;
a buying and selling processing unit that, when the purchase operation is performed on the terminal, acquires the personal information associated with the terminal to carry out a buying and selling procedure;
16. The event management server system according to claim 14, further comprising: An event management server system acquires data of images captured by terminals held by users who have visited the event venue;
the event management server system collating the registered data with the captured image to obtain the user's seat category, and selecting an image related to the user's seat category as a content image to be displayed on the terminal;
a step in which the event management server system transmits data of the selected content image to the terminal and displays it;
the event management server system stores three-dimensional data of the spatial structure of the event venue as the registration data;
In the selecting step, the event management server system identifies the user's position by comparing the photographed image with the spatial structure, and further acquires a seat category corresponding to the identified user's position based on information associated with the three-dimensional data of the spatial structure.
Content image control method. A function to acquire data of images taken by devices held by users who visit the event venue;
a function of acquiring a seat category of the user by comparing the registered data with the captured image, and selecting an image related to the seat category of the user as a content image to be displayed on the terminal;
a function of transmitting data of the selected content image to the terminal and displaying it;
A function of storing three-dimensional data of the spatial structure of the event venue as the registered data;
A computer program that causes a computer to implement the following:
the selecting function identifies the user's position by comparing the captured image with the spatial structure, and further acquires a seat category corresponding to the identified user's position based on information associated with the three-dimensional data of the spatial structure.
Computer program.