JP7740685B2 - Image processing device, image processing system device, image processing method, program, and recording medium - Google Patents

Description

The present invention relates to an image processing device, an image processing system, an image processing method, a program, and a recording medium.

Technology has been reported that uses AR (Augmented Reality) technology to superimpose a virtual image onto a captured image (see, for example, Patent Document 1).

JP 2014-006582 A

However, there is a need for technology that makes it easier to set the position of a virtual image that is superimposed on a captured image.

The present invention therefore aims to provide an image processing device, image processing system, image processing method, program, and recording medium that make it easier to set the position of a virtual image to be superimposed on a captured image.

In order to achieve the above object, the image processing device of the present invention comprises:
The system includes an image acquisition unit, a 3D model generation unit, a virtual image generation unit, a presentation unit, a coordinate information management unit, a local terminal information acquisition unit, and an image information output unit,
the image acquisition unit acquires three-dimensional image information obtained by capturing an image of a specific area with a 3D camera;
the 3D model generation unit generates a 3D model showing a configuration within the specific area based on the three-dimensional image information;
the virtual image generation unit generates a virtual image relating to the specific area;
the virtual image is an image to be superimposed on a local captured image captured by a local terminal within the specific area,
the presentation unit presents the 3D model and the virtual image to an administrator;
the coordinate information management unit sets coordinate information of the reference marker and the virtual image based on a position in the 3D model specified by the administrator, and manages the virtual image and the coordinate information of the virtual image in association with each other;
The image acquisition unit further acquires a local captured image captured within the specific area by a local terminal,
the local terminal information acquisition unit acquires position information of the local terminal and orientation information of the local terminal relative to the reference marker;
The image information output unit is a device that, based on the position information and orientation information of the local terminal, refers to the 3D model to determine whether a position corresponding to the coordinate information of the virtual image exists in the local captured image, and if so, outputs a composite image to the local terminal with the virtual image superimposed at that position.

The image processing method of the present invention comprises:
The method includes an image acquisition step, a 3D model generation step, a virtual image generation step, a presentation step, a coordinate information management step, a local terminal information acquisition step, and an image information output step,
The image acquisition step acquires three-dimensional image information obtained by capturing an image of a specific area with a 3D camera,
the 3D model generation step generates a 3D model showing a configuration within the specific area based on the three-dimensional image information;
the virtual image generating step generates a virtual image relating to the specific area;
the virtual image is an image to be superimposed on a local captured image captured by a local terminal within the specific area,
the presenting step presents the 3D model and the virtual image to an administrator;
the coordinate information management step sets coordinate information of a reference marker and the virtual image based on a position in the 3D model designated by the administrator, and manages the virtual image and the coordinate information of the virtual image in association with each other;
The image acquisition step further includes acquiring a local captured image of the specific area captured by a local terminal;
the local terminal information acquisition step acquires position information of the local terminal and orientation information of the local terminal relative to the reference marker;
The image information output process is a method in which, based on the position information and orientation information of the local terminal, the 3D model is referenced to determine whether a position corresponding to the coordinate information of the virtual image exists within the local captured image, and if so, a composite image with the virtual image superimposed on that position is output to the local terminal.

This invention makes it easier to set the position of a virtual image superimposed on a captured image. This allows an administrator to set the position where the virtual image will be displayed without, for example, having to go to the specific area where the captured image is captured.

FIG. 1 is a block diagram showing an example of the configuration of an image processing apparatus according to the first embodiment. FIG. 2 is a block diagram showing an example of the hardware configuration of the image processing apparatus according to the first embodiment. FIG. 3 is a flowchart showing an example of processing in the image processing apparatus according to the first embodiment. FIG. 4 is a block diagram showing an example of the configuration of an image processing system apparatus according to the second embodiment. FIG. 5 is a block diagram illustrating an example of the configuration of a local terminal according to the second embodiment. FIG. 6 is a flowchart illustrating an example of processing in the local terminal according to the second embodiment. FIG. 7 is a block diagram illustrating an example of the configuration of the administrator terminal according to the second embodiment. FIG. 8 is a flowchart illustrating an example of processing in the administrator terminal according to the second embodiment. FIG. 9(A) is a schematic diagram showing an example of a 3D model of a specific area in which coordinate information for each virtual image is set, and FIG. 9(B) is a schematic diagram showing an example of a top view of the 3D model. FIG. 10 is a schematic diagram showing an example in which a composite image output from the image processing device to a local terminal is displayed on the display of the local terminal.

In the image processing device of the present invention, for example,
The presentation unit may present to the administrator a preview composite image in which the virtual image is superimposed at a position corresponding to coordinate information of the virtual image within an image included in the three-dimensional image information.

The image processing device of the present invention includes, for example,
It also includes a liaison department.
The communication intermediation unit may mediate communication between the administrator and the user having the local terminal.

The image processing system of the present invention comprises:
The image processing device of the present invention and a local terminal are included,
the image processing device and the on-site terminal are capable of communicating with each other via a communication network;
In the image processing device,
the image acquisition unit acquires a local captured image captured by the local terminal within the specific area;
The image information output unit is a device that outputs the composite image to the local terminal.

In the image processing method of the present invention, for example,
The presenting step may be a mode in which a preview composite image in which the virtual image is superimposed at a position corresponding to coordinate information of the virtual image within an image included in the three-dimensional image information is presented to the administrator.

The image processing method of the present invention includes, for example,
Further, it includes a liaison intermediation step,
The communication intermediation step may be configured to intermediate communication between the administrator and the user having the local terminal.

The program of the present invention is a program that causes a computer to execute each step of the method of the present invention as a procedure.

The recording medium of the present invention is a computer-readable recording medium on which the program of the present invention is recorded.

Next, embodiments of the present invention will be described using the drawings. The present invention is not limited to the following embodiments. In each of the following drawings, identical parts are designated by the same reference numerals. Furthermore, unless otherwise specified, the descriptions of each embodiment can be used interchangeably, and the configurations of each embodiment can be combined unless otherwise specified.

[Embodiment 1]
Fig. 1 is a block diagram showing an example of the configuration of an image processing device 10 according to this embodiment. As shown in Fig. 1, the device 10 includes an image acquisition unit 11, a 3D model generation unit 12, a virtual image generation unit 13, a presentation unit 14, a coordinate information management unit 15, a local terminal information acquisition unit 16, and an image information output unit 17. The device 10 may further include, for example, a communication intermediation unit 18 as an optional component.

The device 10 may be, for example, a single device including each of the above-mentioned components, or a device in which each of the above-mentioned components can be connected via a communication network. Furthermore, the device 10 can be connected to an external device (described later) via the communication network. The communication network is not particularly limited and any known network can be used, and may be wired or wireless, for example. Examples of the communication network include the Internet, WWW (World Wide Web), telephone lines, LAN (Local Area Network), SAN (Storage Area Network), DTN (Delay Tolerant Networking), LPWA (Low Power Wide Area), and L5G (Local 5G). Examples of wireless communication include Wi-Fi (registered trademark), Bluetooth (registered trademark), local 5G, and LPWA. The wireless communication may be in the form of direct communication between devices (ad hoc communication), infrastructure communication, or indirect communication via an access point. The device 10 may be incorporated into a server as a system. The device 10 may also be, for example, a personal computer (PC, e.g., desktop or laptop) on which the program of the present invention is installed, a smartphone, a tablet terminal, or the like. Furthermore, the device 10 may be in the form of cloud computing or edge computing, in which at least one of the components is located on a server and the other components are located on a terminal.

Figure 2 shows an example block diagram of the hardware configuration of device 10. Device 10 includes, for example, a central processing unit (CPU, GPU, etc.) 101, memory 102, bus 103, storage device 104, input device 105, display device 106, communication device 107, imaging device 108, and audio input/output device 109. Each component of device 10 is interconnected via bus 103 and its respective interface (I/F).

The central processing unit 101 is responsible for overall control of the device 10. In the device 10, the central processing unit 101 executes, for example, the program of the present invention and other programs, and also reads and writes various types of information. Specifically, the central processing unit 101 functions as, for example, an image acquisition unit 11, a 3D model generation unit 12, a virtual image generation unit 13, a presentation unit 14, a coordinate information management unit 15, a local terminal information acquisition unit 16, an image information output unit 17, and a communication intermediation unit 18.

The bus 103 can also be connected to, for example, external devices. Examples of such external devices include external storage devices (external databases, etc.), printers, external input devices, external display devices, and external imaging devices. The device 10 can be connected to an external network (the communication line network) via, for example, a communication device 107 connected to the bus 103, and can also be connected to other devices via the external network.

The memory 102 may be, for example, a main memory (primary storage device). When the central processing unit 101 performs processing, the memory 102 reads various operating programs, such as the program of the present invention, stored in the storage device 104 (described below), and the central processing unit 101 receives data from the memory 102 and executes the programs. The main memory may be, for example, a RAM (random access memory). The memory 102 may also be, for example, a ROM (read-only memory).

The storage device 104 is also referred to as an auxiliary storage device, in contrast to the main memory (primary storage device). As mentioned above, the storage device 104 stores operating programs, including the program of the present invention. The storage device 104 may be, for example, a combination of a recording medium and a drive that reads and writes from and to the recording medium. The recording medium is not particularly limited and may be internal or external, and examples include HD (hard disk), CD-ROM, CD-R, CD-RW, MO, DVD, flash memory, and memory card. The storage device 104 may be, for example, a hard disk drive (HDD) or solid state drive (SSD) that integrates the recording medium and drive.

In the device 10, the memory 102 and storage device 104 can also store various information such as log information, information obtained from an external database (not shown) or external device, information generated by the device 10, and information used by the device 10 when executing processing. Note that at least some of the information may be stored, for example, on an external server other than the memory 102 and storage device 104, or may be stored in a distributed manner across multiple terminals using blockchain technology, etc.

The device 10 may further include, for example, an input device 105, a display device 106, an imaging device 108, and an audio input/output device 109. The input device 105 may be, for example, a touch panel, a keyboard, a mouse, etc. The display device 106 may be, for example, an LED display, a liquid crystal display, etc. The imaging device 108 may be, for example, a 2D camera, a 3D camera, an infrared camera, etc. The audio input/output device 109 may be, for example, a microphone, a speaker, etc.

Next, an example of the image processing method of this embodiment will be described based on the flowchart in Figure 3. The image processing method of this embodiment is implemented, for example, using the image processing device 10 of Figure 1 as follows. Note that the image processing method of this embodiment is not limited to use with the image processing device 10 of Figure 1.

First, the image acquisition unit 11 acquires three-dimensional image information captured by a 3D camera of a specific area (S1). The 3D camera refers to a camera that can capture not only vertical and horizontal information but also depth (depth). The 3D camera is, for example, an external device. The specific area is not particularly limited as long as it is indoors. Examples of indoor locations include exhibition halls, event venues, department stores, universities, and other schools. The three-dimensional image information includes, for example, an image and point cloud data corresponding to the image. The image may be, for example, a still image or a video. The image may also include, for example, multiple panoramic images, which can be stitched together to create a full-surround panoramic image. The acquisition is performed, for example, via the communication network.

Next, the 3D model generation unit 12 generates a 3D model representing the configuration within the specific area based on the three-dimensional image information (S2). The 3D model includes, for example, coordinate information. The coordinate information may be, for example, two-dimensional coordinate information (x, y) consisting of width (x) and depth (y) values, or three-dimensional coordinate information (x, y, z) including a height (z) value. The configuration within the specific area includes, for example, not only the structure within the specific area but also the position and shape of objects located in the specific area. Below, a specific example of generating the 3D model from the three-dimensional image information is described, but this is not limiting and known techniques can be used. First, the 3D model generation unit 12 performs polygon mesh modeling on the point cloud data. Then, the 3D model generation unit 12 places an image corresponding to the point cloud data on the surface of the polygon mesh.

Next, the virtual image generation unit 13 generates a virtual image of the specific area (S3). The virtual image is an image to be superimposed on a local image captured by a local terminal within the specific area. The local terminal will be described later. The virtual image generation unit 13 may generate, for example, multiple virtual images. Specifically, the virtual image includes, for example, a virtual image for each object in the specific area. The objects are not particularly limited and can be set arbitrarily, and may be man-made or natural. Examples of man-made objects include facilities, exhibition booths, tenants, classrooms, etc. Examples of natural objects include mountains, oceans, rocks, etc. The virtual image for each object includes, for example, a position image of the object, and the position images are linked to each other for each object. The position image is, for example, an image related to the object. Generating the virtual image may mean, for example, reading the virtual image stored in memory 102 or storage device 104 in advance, or obtaining the virtual image from an external database via the communication network. Furthermore, the virtual information generation unit 13 may, for example, acquire information about the object (also referred to as object information) and generate the virtual image based on the object information.

Next, the presentation unit 14 presents the 3D model and the virtual image to the administrator (S4). The presentation format is not particularly limited, and may be, for example, by displaying the 3D model and the virtual image on the display device 106 of the device 10. The presentation unit 14 may also present the 3D model and the virtual image by outputting them to the administrator's terminal via the communication network and displaying them on the display of the administrator's terminal. The administrator's terminal (also referred to as the administrator terminal) is not particularly limited, and may be, for example, a terminal of an administrator outside the specific area, such as a personal computer, smartphone, mobile phone, or wearable device such as glasses or wristwatch. Alternatively, the device 10 may function as the administrator terminal, for example. Presentation by the presentation unit 14 may be continuous, for example, while the local terminal is displaying a composite image, described below.

Next, the coordinate information management unit 15 sets the coordinate information of the reference marker and the virtual image based on the position within the 3D model specified by the administrator, and manages the virtual image and the coordinate information of the virtual image in association with each other (S5). The administrator, for example, uses the input device 105 of the device 10 to specify any position within the 3D model as the position where the reference marker will be placed and the position where the virtual image is to be displayed. The operation may also be performed using the administrator terminal instead of the input device 105 of the device 10. The coordinate information may be, for example, as described above. The position within the 3D model specified by the administrator may be, for example, a position where only the width and depth are specified, or a position where the height is also specified. The reference marker is, for example, a marker that triggers recording of the trajectory of the on-site terminal, as described below. Examples of the reference marker include a QR code (registered trademark), an AR marker, a barcode, and a Chameleon Code (registered trademark). The setting of each piece of coordinate information, particularly the coordinate information of the virtual image, by the coordinate information management unit 15 may be performed repeatedly, for example, while the local terminal is displaying a composite image (described below).

Next, the image acquisition unit 11 acquires local captured images of the specific area captured by a local terminal (S6). The local terminal is not particularly limited and may be, for example, a terminal of a user located in the specific area, such as a personal computer, smartphone, mobile phone, or wearable device such as glasses or wristwatch. The local captured images may be, for example, still images or videos. The acquisition is performed, for example, via the communication network. Specifically, the local captured images are, for example, two-dimensional images captured by a 2D camera (RGB camera) of the local terminal. The acquisition of the local captured images by the image acquisition unit 11 may be performed, for example, in parallel with and continuously with other processes.

Next, the local terminal information acquisition unit 16 acquires position information and orientation information of the local terminal relative to the reference marker (S7). Hereinafter, the position information and orientation information are collectively referred to as local terminal information. The position information and orientation information of the local terminal is, for example, information indicating the position and orientation of the local terminal recorded based on the reference markers placed within the specific area and various sensors of the local terminal (such as an acceleration sensor or geomagnetic sensor). The local terminal information acquisition unit 16 may, for example, acquire the position information and orientation information of the local terminal in association with the local captured image. The orientation information can also be referred to, for example, as the imaging direction of the imaging unit of the local terminal.

The image information output unit 17 then outputs a composite image in which the virtual image is superimposed on the captured image (S8), and the process ends (END). Specifically, based on the position information and orientation information of the local terminal, the image information output unit 17 first references the 3D model in which the coordinate information of the reference marker and the virtual image is set, and determines whether a position corresponding to the coordinate information of the virtual image exists in the captured image on site. If the position exists, the image information output unit 17 outputs a composite image in which the virtual image is superimposed on the position to the local terminal. This output may be performed, for example, via the communication network. If the specified area is indoors, the position of the local terminal cannot be determined using GPS (Global Positioning System). However, according to this embodiment, by setting the reference marker, the position of the local terminal can be determined even indoors. The output of the composite image by the image information output unit 17 may be performed repeatedly, for example.

Furthermore, for example, in step (S4), after presenting the 3D model and the virtual image to the administrator, if the administrator specifies an arbitrary position within the 3D model, the presentation unit 14 may present the administrator with a preview composite image in which the virtual image is superimposed on the image included in the three-dimensional image information at a position corresponding to the coordinate information of the virtual image. The format of the presentation is not particularly limited and may be, for example, the same as described above. The "image included in the three-dimensional image information" may be an image having a position corresponding to the coordinate information of the virtual image, and is not particularly limited, for example, there may be multiple images. For example, the presentation unit 14 may generate multiple preview composite images using multiple images. The administrator may, for example, refer to the preview composite image to re-specify and fine-tune the position where the reference marker is to be placed and the desired position where the virtual image is to be displayed. Then, after the position specification is confirmed, the processing of step (S5) is executed. This allows, for example, the administrator to specify the position while checking an example of the composite image displayed on the local terminal. Additionally, by generating multiple preview composite images, you can check how the virtual image appears from multiple angles.

The device 10 may further include a communication intermediary unit 18, as described above. For example, after step (S8), the communication intermediary unit 18 mediates communication between the administrator and the user having the local terminal (S9). Note that step (S9) does not necessarily have to occur after step (S8). Furthermore, step (S9) is an optional step and may not be executed. The communication intermediary unit 18 may, for example, enable communication between the administrator terminal and the local terminal, and mediate communication between the administrator and the user. The method of communication is not particularly limited, and may include, for example, communication via text data such as chat, or communication via voice data such as phone calls. This allows communication between an administrator outside the specific area and a user within the specific area.

According to this embodiment, by using the 3D model, it is possible to more easily set the position of the virtual image to be superimposed on the on-site captured image. In other words, according to this embodiment, the position of the virtual image displayed using AR technology can be set on the 3D model. This allows an administrator to, for example, link and set the position at which the virtual image will be superimposed with any location within the specific area from outside the specific area where the on-site captured image is captured (e.g., a company, home, etc.). Furthermore, according to this embodiment, for example, the administrator can mediate communication with users within the specific area (e.g., "Please move to the location where the XX pin (virtual image) is displayed," "Please move along the arrow (virtual image)," etc.), allowing the administrator to navigate the users in real time.

[Embodiment 2]
Fig. 4 is a block diagram showing an example configuration of an image processing system apparatus 100 according to this embodiment. As shown in Fig. 4, this apparatus 100 includes the image processing apparatus 10 described in the first embodiment and a local terminal 20. The apparatus 100 may further include, for example, an administrator terminal 30. For example, there may be one local terminal 20 and one administrator terminal 30, or two or more local terminals 20 and 30. The image processing apparatus 10, the local terminal 20, and the administrator terminal 30 can communicate with each other via a communication network 40. The communication network 40 is the same as the communication network described above.

In the image processing device 10, the image acquisition unit 11 acquires the on-site captured image captured by the on-site terminal 20, for example, as described above. Furthermore, the image information output unit 17 outputs the composite image to the on-site terminal 20, for example, as described above.

The local terminal 20 is a terminal of a user located within the specific area, and may be, for example, a personal computer (PC, e.g., desktop or notebook), a smartphone, a tablet terminal, etc. The local terminal 20 is, for example, a device used in conjunction with the image processing device 10 described in embodiment 1.

The administrator terminal 30 is a terminal of an administrator who is outside the specified area, and may be, for example, a personal computer (PC, e.g., desktop or laptop), a smartphone, a tablet terminal, etc. The administrator terminal 30 is, for example, a device used in conjunction with the image processing device 10 described in embodiment 1.

There are no particular restrictions on the configuration, including the hardware configuration, of the local terminal 20 and the administrator terminal 30. Furthermore, there are no particular restrictions on the information processing methods used by the local terminal 20 and the administrator terminal 30.

Figure 5 is a block diagram showing an example configuration of a local terminal 20 according to this embodiment. As shown in Figure 5, the local terminal 20 includes, for example, an imaging unit 21 that captures the reference markers installed within the specific area and local captured images; a recording unit 22 that records the trajectory of the reference marker after imaging, starting from the point where the reference marker was imaged, as position information, and records the imaging direction in which the local captured images were captured as orientation information; an output unit 23 that outputs the local captured images, the position information, and the orientation information to the image processing device 10; and a display processing unit 24 that acquires the composite image output from the image processing device 10 and displays it on a display. The local terminal 20 may also be capable of contacting, for example, the administrator (administrator terminal 30).

The hardware configuration of the local terminal 20 is similar to the hardware configuration of the image processing device 10 shown in Figure 2, except that, for example, the central processing unit 101 functions as the recording unit 22, output unit 23, and display processing unit 24, and the imaging device 108 functions as the imaging unit 21.

Next, an example of an information processing method in the local terminal 20 will be described based on the flowchart in Figure 6. The information processing method in the local terminal 20 (also referred to as the first information processing method) is implemented, for example, as follows using the local terminal 20 in Figure 5. Note that the first information processing method is not limited to use with the local terminal 20 in Figure 5.

First, the imaging unit 21 captures the reference markers installed within the specific area and on-site captured images (S21). Next, the recording unit 22 records the trajectory of the captured images, starting from the point where the reference markers were captured as position information, and records the direction in which the on-site captured images were captured as orientation information (S22). Next, the output unit 23 outputs the on-site captured images, the position information, and the orientation information to the image processing device 10 (S23). Then, the display processing unit 24 acquires the composite image output from the image processing device 10 and displays it on the display (display device 106) (S24), and the process ends (END). Furthermore, the on-site terminal 20 may contact the administrator via the image processing device 10, for example, after step (S24). Note that the contacting step is not limited to being performed after step (S24). Furthermore, the contacting step is an optional step and may not be executed. Each step in the first information processing method may be executed repeatedly, for example.

Figure 7 is a block diagram showing an example configuration of an administrator terminal 30 of this embodiment. As shown in Figure 7, the administrator terminal 30 includes, for example, a display processing unit 31 that displays the 3D model and the virtual image presented from the image processing device 10, a designation unit 32 that designates the positions of the reference marker and the virtual image relative to the 3D model based on an administrator operation (for example, an operation using the input device 105), and an output unit 33 that outputs the designated positions to the image processing device 10. Furthermore, the display processing unit 31 may, for example, display the preview composite image presented from the image processing device 10. Furthermore, the administrator terminal 30 may, for example, be capable of contacting the user (local terminal 20).

The hardware configuration of the administrator terminal 30 is similar to the hardware configuration of the image processing device 10 shown in Figure 2, except that, for example, the central processing unit 101 functions as a display processing unit 31, a designation unit 32, and an output unit 33.

Next, an example of an information processing method in the administrator terminal 30 will be described based on the flowchart in Figure 8. The information processing method in the administrator terminal 30 (also referred to as the second information processing method) is implemented, for example, as follows using the administrator terminal 30 in Figure 7. Note that the second information processing method is not limited to use with the administrator terminal 30 in Figure 7.

First, the display processing unit 31 displays the 3D model and the virtual image presented by the image processing device 10 (S31). Next, the designation unit 32 designates the positions of the reference marker and the virtual image relative to the 3D model based on an administrator's operation (e.g., an operation using the input device 105) (S32). Then, the output unit 33 outputs the designated positions to the image processing device 10 (S33), and the process ends (END). After step (S32) and before step (S33), the display processing unit 31 may, for example, display the preview composite image presented by the image processing device 10. Then, the designation unit 32 may, for example, again designate the positions of the reference marker and the virtual image after displaying the preview composite image. Furthermore, the administrator terminal 30 may, for example, contact the user via the image processing device 10 after step (S33). Note that the contact step is not limited to being performed after step (S33). Furthermore, the contacting step is an optional step and does not need to be performed. Each step in the second information processing method may be performed repeatedly, for example.

The image processing method of this embodiment (also referred to as the second image processing method) includes the image processing method shown in Figure 3, the first information processing method shown in Figure 6, and the second information processing method shown in Figure 8.

According to this embodiment, as with embodiment 1, it is easier to set the position of the virtual image to be superimposed on the on-site captured image.

[Embodiment 3]
An example of superimposing the virtual image on the captured image will be described.

Figure 9 (A) shows an example of a 3D model of a specific area in which coordinate information for each virtual image has been set, and Figure 9 (B) shows an example of the positional relationship of each virtual image in the 3D model. As shown in Figure 9, the image information output unit 17 identifies the imaging direction X of the local terminal 20, for example, based on the position information and orientation information of the local terminal 20. In the example of Figure 9, the coordinate information for virtual images A to C exists in imaging direction X.

Figure 10 shows an example of a composite image output from the image processing device 10 to the local terminal 20, displayed on the display of the local terminal 20. As shown in Figure 10, the image information output unit 17 generates a composite image by superimposing virtual images A to C (e.g., the position images; hereinafter, also referred to as pins) at positions in the local captured image corresponding to objects A to C present in the imaging direction X. Pins A to C may be superimposed, for example, based on the distance between the position information of the local terminal 20 and the coordinate information of pins A to C, with pins farther from the local terminal 20 being smaller than the other pins. On the other hand, pins A to C may be superimposed, for example, based on the distance between the position information of the local terminal 20 and the coordinate information of pins A to C, with pins closer to the local terminal 20 being larger than the other pins. Furthermore, the outer edges of the pins may be colored, for example, depending on the distance between the position information of the local terminal 20 and the coordinate information of pins A to C.

Furthermore, as shown in FIG. 10, the detailed information image is specifically displayed in window 1. In the figure, window 1 is displayed below the on-site captured image, but this is by way of example and is not limiting. For example, when the present invention is applied to an exhibition, a detailed information image generated based on detailed information about the products and services on display, the company operating the booth, etc., is displayed in window 1.

[Embodiment 4]
The program of this embodiment is a program for causing a computer to execute each step of the method of the present invention as a procedure. In the present invention, "procedure" may be read as "processing." The program of this embodiment may be recorded on, for example, a computer-readable recording medium. The recording medium is, for example, a non-transitory computer-readable storage medium. The recording medium is not particularly limited, and examples thereof include a read-only memory (ROM), a hard disk (HD), an optical disk, etc.

The present invention has been described above with reference to embodiments, but the present invention is not limited to the above embodiments. Various modifications that would be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

<Additional Notes>
Some or all of the above embodiments can be described as, but not limited to, the following supplementary notes.
(Appendix 1)
The system includes an image acquisition unit, a 3D model generation unit, a virtual image generation unit, a presentation unit, a coordinate information management unit, a local terminal information acquisition unit, and an image information output unit,
the image acquisition unit acquires three-dimensional image information obtained by capturing an image of a specific area with a 3D camera;
the 3D model generation unit generates a 3D model showing a configuration within the specific area based on the three-dimensional image information;
the virtual image generation unit generates a virtual image relating to the specific area;
the virtual image is an image to be superimposed on a local captured image captured by a local terminal within the specific area,
the presentation unit presents the 3D model and the virtual image to an administrator;
the coordinate information management unit sets coordinate information of the reference marker and the virtual image based on a position in the 3D model specified by the administrator, and manages the virtual image and the coordinate information of the virtual image in association with each other;
The image acquisition unit further acquires a local captured image captured within the specific area by a local terminal,
the local terminal information acquisition unit acquires position information of the local terminal and orientation information of the local terminal relative to the reference marker;
The image information output unit refers to the 3D model based on the position information and orientation information of the local terminal, and determines whether a position corresponding to the coordinate information of the virtual image exists in the local captured image, and if so, outputs a composite image with the virtual image superimposed on the position to the local terminal.
(Appendix 2)
The image processing device according to claim 1, wherein the presentation unit presents to the administrator a preview composite image in which the virtual image is superimposed at a position corresponding to coordinate information of the virtual image within an image included in the three-dimensional image information.
(Appendix 3)
It also includes a liaison department.
3. The image processing device according to claim 1, wherein the communication intermediary unit mediates communication between the administrator and the user having the local terminal.
(Appendix 4)
An image processing device according to any one of Supplementary Notes 1 to 3 and a local terminal,
the image processing device and the on-site terminal are capable of communicating with each other via a communication network;
In the image processing device,
the image acquisition unit acquires a local captured image captured by the local terminal within the specific area;
The image information output unit outputs the composite image to the local terminal.
(Appendix 5)
The method includes an image acquisition step, a 3D model generation step, a virtual image generation step, a presentation step, a coordinate information management step, a local terminal information acquisition step, and an image information output step,
The image acquisition step acquires three-dimensional image information obtained by capturing an image of a specific area with a 3D camera,
the 3D model generation step generates a 3D model showing a configuration within the specific area based on the three-dimensional image information;
the virtual image generating step generates a virtual image relating to the specific area;
the virtual image is an image to be superimposed on a local captured image captured by a local terminal within the specific area,
the presenting step presents the 3D model and the virtual image to an administrator;
the coordinate information management step sets coordinate information of a reference marker and the virtual image based on a position in the 3D model designated by the administrator, and manages the virtual image and the coordinate information of the virtual image in association with each other;
The image acquisition step further includes acquiring a local captured image of the specific area captured by a local terminal;
the local terminal information acquisition step acquires position information of the local terminal and orientation information of the local terminal relative to the reference marker;
The image information output process is an image processing method in which, based on the position information and orientation information of the local terminal, the 3D model is referenced to determine whether a position corresponding to the coordinate information of the virtual image exists in the local captured image, and if so, a composite image with the virtual image superimposed on that position is output to the local terminal.
(Appendix 6)
The image processing method according to claim 5, wherein the presentation step presents to the administrator a preview composite image in which the virtual image is superimposed at a position corresponding to the coordinate information of the virtual image within the image included in the three-dimensional image information.
(Appendix 7)
Further, it includes a liaison intermediation step,
7. The image processing method according to claim 5, wherein the communication mediation step mediates communication between the administrator and the user having the local terminal.
(Appendix 8)
a program for causing a computer to execute procedures including an image acquisition procedure, a 3D model generation procedure, a virtual image generation procedure, a presentation procedure, a coordinate information management procedure, a local terminal information acquisition procedure, and an image information output procedure;
The image acquisition step acquires three-dimensional image information obtained by capturing an image of a specific area with a 3D camera,
the 3D model generation step generates a 3D model showing a configuration within the specific area based on the three-dimensional image information;
the virtual image generation step generates a virtual image relating to the specific area;
the virtual image is an image to be superimposed on a local captured image captured by a local terminal within the specific area,
the presentation step presents the 3D model and the virtual image to an administrator;
the coordinate information management step includes setting coordinate information of a reference marker and the virtual image based on a position in the 3D model designated by the administrator, and managing the virtual image and the coordinate information of the virtual image in association with each other;
The image acquisition step further includes acquiring a local captured image of the specific area captured by a local terminal;
the local terminal information acquisition step acquires position information of the local terminal and orientation information of the local terminal relative to the reference marker;
The image information output procedure refers to the 3D model based on the position information and orientation information of the local terminal, and determines whether a position corresponding to the coordinate information of the virtual image exists in the local captured image, and if so, outputs a composite image with the virtual image superimposed at that position to the local terminal.
(Appendix 9)
The program according to claim 8, wherein the presentation step presents to the administrator a preview composite image in which the virtual image is superimposed at a position corresponding to the coordinate information of the virtual image within the image included in the three-dimensional image information.
(Appendix 10)
It also includes a contact procedure,
10. The program according to claim 8, wherein the communication intermediation procedure mediates communication between the administrator and the user having the local terminal.
(Appendix 11)
A computer-readable recording medium having recorded thereon the program according to any one of appendices 8 to 10.

According to the present invention, it is possible to more easily set the position of a virtual image to be superimposed on the on-site captured image. Therefore, the present invention is particularly useful in fields that use AR technology, such as exhibitions and games.

10 Image processing device 11 Image acquisition unit 12 3D model generation unit 13 Virtual image generation unit 14 Presentation unit 15 Coordinate information management unit 16 Local terminal information acquisition unit 17 Image information output unit 18 Communication intermediation unit 20 Local terminal 21 Imaging unit 22 Recording unit 23 Output unit 24 Display processing unit 30 Administrator terminal 31 Display processing unit 32 Designation unit 33 Output unit 100 Image processing system device 101 Central processing unit 102 Memory 103 Bus 104 Storage device 105 Input device 106 Display device 107 Communication device 108 Imaging device 109 Sound input/output device

Claims (10)

The system includes an image acquisition unit, a 3D model generation unit, a virtual image generation unit, a presentation unit, a coordinate information management unit, a local terminal information acquisition unit, and an image information output unit,
the image acquisition unit acquires three-dimensional image information obtained by capturing an image of a specific area with a 3D camera;
the 3D model generation unit generates a 3D model showing a configuration within the specific area based on the three-dimensional image information;
the virtual image generation unit generates a virtual image relating to the specific area;
the virtual image is an image to be superimposed on a local captured image captured by a local terminal within the specific area,
the presentation unit presents the 3D model and the virtual image to an administrator;
the coordinate information management unit sets coordinate information of the reference marker and the virtual image based on a position in the 3D model designated by the administrator as a position to place a reference marker and a position where the virtual image is to be displayed , and manages the virtual image and the coordinate information of the virtual image in association with each other;
The image acquisition unit further acquires a local captured image captured within the specific area by a local terminal,
the local terminal information acquisition unit acquires position information of the local terminal and orientation information of the local terminal relative to the reference marker;
The image information output unit refers to the 3D model based on the position information and orientation information of the local terminal, and determines whether a position corresponding to the coordinate information of the virtual image exists in the local captured image, and if so, outputs a composite image with the virtual image superimposed on the position to the local terminal. The image processing device described in claim 1, wherein the presentation unit presents to the administrator a preview composite image in which the virtual image is superimposed at a position corresponding to the coordinate information of the virtual image within the image included in the three-dimensional image information. It also includes a liaison department.
The image processing apparatus according to claim 1 , wherein the communication intermediary unit mediates communication between the administrator and the users who have the local terminals. An image processing device according to any one of claims 1 to 3 and a local terminal,
the image processing device and the on-site terminal are capable of communicating with each other via a communication network;
In the image processing device,
the image acquisition unit acquires a local captured image captured by the local terminal within the specific area;
The image information output unit outputs the composite image to the local terminal. The method includes an image acquisition step, a 3D model generation step, a virtual image generation step, a presentation step, a coordinate information management step, a local terminal information acquisition step, and an image information output step,
The image acquisition step acquires three-dimensional image information obtained by capturing an image of a specific area with a 3D camera,
the 3D model generation step generates a 3D model showing a configuration within the specific area based on the three-dimensional image information;
the virtual image generating step generates a virtual image relating to the specific area;
the virtual image is an image to be superimposed on a local captured image captured by a local terminal within the specific area,
the presenting step presents the 3D model and the virtual image to an administrator;
the coordinate information management step sets coordinate information of the reference marker and the virtual image based on a position in the 3D model designated by the administrator as a position to place a reference marker and a position where the virtual image is to be displayed , and manages the virtual image and the coordinate information of the virtual image in association with each other;
The image acquisition step further includes acquiring a local captured image of the specific area captured by a local terminal;
the local terminal information acquisition step acquires position information of the local terminal and orientation information of the local terminal relative to the reference marker;
The image information output process refers to the 3D model based on the position information and orientation information of the local terminal, determines whether a position corresponding to the coordinate information of the virtual image exists in the local captured image, and if so, outputs a composite image with the virtual image superimposed on the position to the local terminal.This is an image processing method in which each process is executed by a computer . The image processing method described in claim 5, wherein the presentation step presents to the administrator a preview composite image in which the virtual image is superimposed at a position corresponding to the coordinate information of the virtual image within the image included in the three-dimensional image information. Further, it includes a liaison intermediation step,
7. The image processing method according to claim 5, wherein said communication intermediation step mediates communication between said administrator and said users having said local terminals. a program for causing a computer to execute procedures including an image acquisition procedure, a 3D model generation procedure, a virtual image generation procedure, a presentation procedure, a coordinate information management procedure, a local terminal information acquisition procedure, and an image information output procedure;
The image acquisition step acquires three-dimensional image information obtained by capturing an image of a specific area with a 3D camera,
the 3D model generation step generates a 3D model showing a configuration within the specific area based on the three-dimensional image information;
the virtual image generation step generates a virtual image relating to the specific area;
the virtual image is an image to be superimposed on a local captured image captured by a local terminal within the specific area,
the presentation step presents the 3D model and the virtual image to an administrator;
the coordinate information management step includes setting coordinate information of the reference marker and the virtual image based on a position in the 3D model designated by the administrator as a position to place a reference marker and a position where the virtual image is to be displayed , and managing the virtual image and the coordinate information of the virtual image in association with each other;
The image acquisition step further includes acquiring a local captured image of the specific area captured by a local terminal;
the local terminal information acquisition step acquires position information of the local terminal and orientation information of the local terminal relative to the reference marker;
The image information output procedure refers to the 3D model based on the position information and orientation information of the local terminal, and determines whether a position corresponding to the coordinate information of the virtual image exists in the local captured image, and if so, outputs a composite image with the virtual image superimposed at that position to the local terminal. The program described in claim 8, wherein the presentation step presents to the administrator a preview composite image in which the virtual image is superimposed at a position corresponding to the coordinate information of the virtual image within the image included in the three-dimensional image information. Furthermore, it includes a contact procedure,
10. The program according to claim 8, wherein the communication intermediation procedure mediates communication between the administrator and the user having the local terminal.