JP6906370B2 - Image processing device - Google Patents

Description

The present disclosure relates to an image processing apparatus, an image processing system and a control program.

As an image processing technology for obtaining images of a subject (also called an imaging object) viewed from various angles around it, an image to be imaged is used as an image obtained by imaging from many viewpoints (also referred to as a multi-viewpoint image). There is a technique for generating and displaying a reconstructed image viewed from an arbitrary angle (see, for example, Patent Document 1 below). In this technique, for example, the parameters required for image reconstruction are calculated from the viewpoint position of the observer and the type of image output device. Next, using this parameter, the corresponding pixel of the multi-viewpoint image is calculated for each pixel of the reconstructed image. Then, the corresponding pixels are extracted from the multi-viewpoint image to generate a reconstructed image.

Japanese Unexamined Patent Publication No. 8-116556

There is room for improvement in image processing that obtains images of the object to be imaged from various angles around it in terms of reducing the amount of calculation.

Image processing devices, image processing systems and control programs are disclosed.

One aspect of the image processing apparatus includes an acquisition unit, an estimation unit, a detection unit, an image generation unit, at least one of an output unit and a storage medium, and an area setting unit . The acquisition unit captures a plurality of captured images in which an object is captured under conditions where the combination of the viewpoint position and the line-of-sight direction is different from each other, and the viewpoint position information and the line-of-sight indicating the viewpoint position at the time of imaging for each of the plurality of captured images. The line-of-sight direction information indicating the direction is acquired. The estimation unit acquires the existence position information indicating the existence position by estimating the existence position of the object based on the viewpoint position information and the line-of-sight direction information related to each of the plurality of captured images. .. The detection unit refers to the object based on the viewpoint position information, the line-of-sight direction information, the existence position information, and the characteristic information of the imaging unit used at each imaging of the plurality of captured images. The first image portion in which the first region is captured from the first viewpoint position in the first line-of-sight direction, and the second region different from the first region in the object is the second viewpoint position different from the first viewpoint position. The second image portion captured in the second line-of-sight direction different from the first line-of-sight direction is detected. The image generation unit generates a composite image by synthesizing the first image portion and the second image portion. The output unit outputs the composite image. The storage medium stores the composite image. The output unit includes a display unit that visually outputs the composite image. The area setting unit changes the display target area captured in the composite image that is visually output by the display unit of the object in response to a signal input in response to the user's operation. .. The area setting unit is one of the already displayed areas of the object captured in the first composite image visually output by the display unit in response to a signal input in response to the user's operation. The changed display target area including the remaining portion excluding the portion and the non-display area of the object that is not captured in the first composite image that is visually output by the display unit is set. In response to the setting of the display target area after the change by the area setting unit, the detection unit receives the viewpoint position information, the line-of-sight direction information, the existence position information, and the said existence position information for the plurality of captured images. Based on the characteristic information of the imaging unit, the third image portion that captures the non-display area is detected. The image generation unit generates a second composite image by synthesizing the fourth image portion that captures the displayed region in the first composite image and the third image portion. The display unit visually outputs the second composite image.
One aspect of the image processing device includes an acquisition unit, an estimation unit, a detection unit, an image generation unit, a resolution calculation unit, an output unit, and a resolution setting unit. The acquisition unit captures a plurality of captured images in which an object is captured under conditions where the combination of the viewpoint position and the line-of-sight direction is different from each other, and the viewpoint position information and the line-of-sight indicating the viewpoint position at the time of imaging for each of the plurality of captured images. The line-of-sight direction information indicating the direction is acquired. The estimation unit acquires the existence position information indicating the existence position by estimating the existence position of the object based on the viewpoint position information and the line-of-sight direction information related to each of the plurality of captured images. .. The detection unit refers to the object based on the viewpoint position information, the line-of-sight direction information, the existence position information, and the characteristic information of the imaging unit used at each imaging of the plurality of captured images. The first image portion in which the first region is captured from the first viewpoint position in the first line-of-sight direction, and the second region different from the first region in the object is the second viewpoint position different from the first viewpoint position. The second image portion captured in the second line-of-sight direction different from the first line-of-sight direction is detected. The image generation unit generates a composite image by synthesizing the first image portion and the second image portion. The acquisition unit acquires distance distribution information indicating the distribution of the distance from the viewpoint position to the surface of the object at the time of imaging for each of the plurality of captured images. The resolution calculation unit calculates the imaging resolution of the image portion that captures the object in each of the captured images based on the distance distribution information and the characteristic information. The output unit visually outputs the composite image. The resolution setting unit sets the display resolution of the image portion that captures the object that is visually output by the output unit in response to the signal input in response to the user's operation. The detection unit detects the first image portion and the second image portion whose imaging resolution is equal to or higher than the display resolution in the plurality of captured images.
One aspect of the image processing device includes an acquisition unit, an estimation unit, a detection unit, an image generation unit, and a switching unit. The acquisition unit captures a plurality of captured images in which an object is captured under conditions where the combination of the viewpoint position and the line-of-sight direction is different from each other, and the viewpoint position information and the line-of-sight indicating the viewpoint position at the time of imaging for each of the plurality of captured images. The line-of-sight direction information indicating the direction is acquired. The estimation unit acquires the existence position information indicating the existence position by estimating the existence position of the object based on the viewpoint position information and the line-of-sight direction information related to each of the plurality of captured images. .. The detection unit refers to the object based on the viewpoint position information, the line-of-sight direction information, the existence position information, and the characteristic information of the imaging unit used at each imaging of the plurality of captured images. The first image portion in which the first region is captured from the first viewpoint position in the first line-of-sight direction, and the second region different from the first region in the object is the second viewpoint position different from the first viewpoint position. The second image portion captured in the second line-of-sight direction different from the first line-of-sight direction is detected. The image generation unit generates a composite image by synthesizing the first image portion and the second image portion. In a state where two or more captured images are sequentially acquired by the acquisition unit, the switching unit outputs an image for output in the image generation unit in response to a signal input in response to a user's operation. The image generation mode to be generated is selectively switched between the first image generation mode, the second image generation mode, and the third image generation mode. The first image generation mode includes a mode in which an image for output is prepared by using the captured image each time the captured image is acquired by the acquisition unit. The second image generation mode is the combination of the first image portion and the second image portion detected by the detection unit with respect to the plurality of captured images already acquired by the acquisition unit. It includes a mode for generating the composite image as an image for output. The third image generation mode responds to the acquisition of the first image portion detected by the detection unit and the captured image by the acquisition unit for the plurality of captured images already acquired by the acquisition unit. The mode includes a mode in which the composite image is generated as an image for output by synthesizing the second image portion detected by the detection unit from the captured image.

For example, the amount of calculation in image processing for obtaining an image of an imaged object viewed from various angles around it can be reduced.

FIG. 1 is a diagram showing an example of the configuration of the image processing system according to the first embodiment. FIG. 2 is a block diagram showing an example of the electrical configuration of the first device. FIG. 3 is a block diagram showing an example of the functional configuration of the first image processing apparatus. FIG. 4 is a schematic view showing an example of an aspect of imaging an image-imaging object. FIG. 5 is a block diagram showing an example of the electrical configuration of the second device. FIG. 6 is a block diagram showing an example of the functional configuration of the second image processing apparatus. FIG. 7 is a diagram showing an example in the representative line-of-sight direction. FIG. 8 is a diagram illustrating an example of a method of estimating the existence position of an imaging object. FIG. 9A is a diagram showing an example of a display image before enlargement on the display screen. FIG. 9B is a diagram showing an example of an enlarged display image on the display screen. FIG. 10A is a diagram showing an example of an image before scrolling on the display screen. FIG. 10B is a diagram showing an example of a scrolled image on the display screen. FIG. 11 is a diagram showing an example of imaging conditions of an imaging object by the first apparatus. 12 (a) to 12 (d) are diagrams illustrating the captured images acquired by the imaging under the imaging conditions of FIG. 11. 13 (a) to 13 (c) are diagrams showing an example of scrolling an image on a display screen. 14 (a) to 14 (g) are diagrams for explaining a first image generation mode for generating an image for display during imaging of a plurality of captured images. FIG. 14A is a schematic view showing how a plurality of captured images are acquired by time-sequential imaging. FIG. 14B is a diagram showing an example of an captured image obtained by imaging at time t0. FIG. 14C is a diagram showing an example of a mode in which the captured image at time t0 is displayed on the display screen at time t0. FIG. 14D is a diagram showing an example of an captured image obtained by imaging at time t1. FIG. 14E is a diagram showing an example of a mode in which the captured image at time t1 is displayed on the display screen at time t1. FIG. 14 (f) is a diagram showing a state in which a captured image is not acquired by imaging at time t2. FIG. 14 (g) is a diagram showing an example of a mode in which the captured image at time t1 is displayed on the display screen at time t2. 15 (a) to 15 (c) are diagrams for explaining a second image generation mode in which an image for display is generated during imaging of a plurality of captured images. FIG. 15A is a diagram showing an example of a mode in which the display target area is changed according to the input of the user. FIG. 15B is a diagram showing an example of an image before scrolling displayed on the display screen. FIG. 15C is a diagram showing an example of a scrolled image displayed on the display screen. 16 (a) to 16 (e) are diagrams for explaining a third image generation mode for generating an image for display during imaging of a plurality of captured images. FIG. 16A is a diagram showing an example of a mode in which the display target area is changed according to the input of the user. FIG. 16B is a diagram showing an example of an image before scrolling displayed on the display screen. FIG. 16C is a diagram showing an example of a swipe operation for changing the display target area on the display screen. FIG. 16D is a diagram showing an example of a scrolled image displayed on the display screen. FIG. 16E is a diagram showing an example of the most recently acquired captured image. FIG. 17 is a flow chart showing an example of an operation flow in which an image processing operation is performed a plurality of times in an image processing system. FIG. 18 is a flow chart showing an example of an operation flow related to each imaging operation in the image processing system. FIG. 19 is a flow chart showing an example of an operation flow for estimating the existence position of an imaging object in an image processing system. FIG. 20 is a flow chart showing an example of an operation flow for enlarging an image for display in an image processing system. FIG. 21 is a flow chart showing an example of an operation flow for replenishing an captured image that is insufficient for generating an image for display in an image processing system. FIG. 22 is a flow chart showing an example of an operation flow when scrolling an image on a display screen in an image processing system. FIG. 23 is a flow chart showing an example of an operation flow when scrolling an image on a display screen in an image processing system. FIG. 24 is a flow chart showing an example of an operation flow when scrolling an image on a display screen in an image processing system. FIG. 25 is a flow chart showing an example of an operation flow related to switching of an image generation mode according to a user's input during acquisition of a plurality of captured images in an image processing system. FIG. 26 is a block diagram showing an example of the functional configuration of the first image processing apparatus according to the second embodiment. 27 (a) to 27 (c) are diagrams showing an example of scrolling an image on the display screen according to the third embodiment.

For example, at a manufacturing site such as a factory, if an accident occurs in an operating manufacturing device, the manufacturing of a product is stopped. At this time, for example, the manufacturing cost may increase due to the decrease in the equipment utilization rate in the entire factory, the customer satisfaction may decrease due to the delay in delivery, and the supply chain may collapse, which may have a great adverse effect on the corporate activities. Therefore, there is an urgent need to recover from accidents in manufacturing equipment.

In such a situation, for example, the person in charge of manufacturing equipment (also called the person in charge of equipment), such as the person in charge of developing the manufacturing equipment, requests the immediate restoration of the manufacturing equipment from the manufacturing site where the accident occurs. Therefore, it is assumed that there is no choice but to rush from the original workplace to the manufacturing site. In this case, for example, the person in charge of the equipment temporarily suspends the original work currently in progress in the original workplace and rushes to the manufacturing site. At this time, for example, even if the accident of the manufacturing equipment is caused by a trivial cause or can be easily solved by the worker at the manufacturing site (also referred to as the field worker), the person in charge of the equipment Is bound by the restoration work of manufacturing equipment until he returns to his original workplace.

Normally, the number of manufactured equipment shipped in operation tends to increase year by year after development. For this reason, the time that the person in charge of equipment is restrained from the restoration work of the manufacturing equipment tends to increase year by year. As a result, for example, the time that the person in charge of equipment should devote to the original work such as the development of new manufacturing equipment is further reduced by the restoration work of the manufacturing equipment.

On the other hand, in recent years, widening the bandwidth of communication networks and remote control technology for personal computers (also referred to as personal computers) (also referred to as remote control technology for personal computers) have been developed. For this reason, for example, by utilizing the wide band of the communication network and the remote control technology of the personal computer, the person in charge of the equipment can check the situation of the manufacturing equipment in which the accident occurred and deal with the situation without going to the manufacturing site. Can be considered. In this case, for example, the time required for the person in charge of equipment to make a round trip to the manufacturing site is reduced, and the person in charge of equipment can secure the time allocated to the original work.

However, with the remote control technology of a personal computer, for example, it is possible to confirm and modify the software installed in the personal computer, confirm and modify the settings of the personal computer, and confirm the information that can be acquired by the sensor under the control of the personal computer. Degree. The information that can be acquired by the sensor may include, for example, information in the field of view that can be captured by the camera. For this reason, for example, in the case of remote control technology for personal computers, the placement and removal of products, semi-finished products or parts in manufacturing equipment, the adjustment of the physical position of the sensor, and the confirmation of the situation outside the measurement range by the sensor are not possible. Difficult to carry out. The confirmation of the situation outside the measurement range by the sensor may include, for example, confirmation of the situation outside the field of view of the camera.

On the other hand, the person in charge of the equipment, for example, for the part that cannot be executed by the remote control technology of the personal computer, uses a report from the on-site worker and a telephone conversation with the on-site worker to provide the manufacturing equipment. It is assumed that the situation will be grasped. In addition, it is assumed that the person in charge of equipment, for example, further grasps the situation and explains the work contents to the manufacturing equipment by telephone with the on-site worker. At this time, the person in charge of the equipment hears the situation of the manufacturing equipment and the work contents, for example, without looking at the manufacturing equipment at the site and without knowing the field of view and the point of interest of the field worker who is the other party of the call. I will explain. Further, at this time, the field worker will explain the situation of the manufacturing equipment within the range that can be seen from his / her own viewpoint, for example, in a state where the person in charge of the equipment does not accurately understand the information that he / she wants to know. Furthermore, at this time, if the presupposed knowledge and terms are not shared, for example, the person in charge of equipment and the field worker use different names for the same part, the content of the explanation will be the other party. There is a risk that the content will not be transmitted correctly and will be understood by the other party. As a result, for example, it is assumed that the person in charge of equipment takes an extremely large amount of time to grasp the status of the manufacturing equipment and to convey the work contents to the manufacturing equipment. Specifically, for example, if the person in charge of equipment explains unnecessary work contents based on grasping an erroneous situation about the manufacturing equipment, it takes more time than necessary for the restoration of the manufacturing equipment. It may be necessary. Such waste of time can occur, for example, when the person in charge of equipment and the field worker communicate with each other by relying only on words.

For such problems, for example, a system that promotes smooth communication between equipment personnel and field workers by using not only words but also images such as still images or moving images and other information. (Also called a remote work support system) can be considered. In this system, for example, a field worker captures an image of the situation at hand with a camera of a portable device such as a smart glass or a smartphone, and a person in charge of the device via a communication line such as a network. A mode of transmitting to the terminal device of is conceivable. In such an aspect, for example, the person in charge of the equipment can visually recognize what the on-site worker is seeing in real time without going to the manufacturing site by visually recognizing the image received by the terminal equipment. Further, for example, the explanation of the work contents to the field worker is shown not only by voice but also by pictures and characters written on the image obtained by the field worker on the display of the terminal device of the field worker. A mode is also conceivable. As a result, the field worker can understand the explanation of the work contents relatively easily, for example. At this time, for example, by correctly telling the on-site worker the place that the person in charge of equipment wants to see, the on-site worker can acquire the image that the person in charge of equipment wants to see. Specifically, for example, the person in charge of equipment can inform the field worker of the desired location and shooting angle of the manufacturing equipment. Then, the field worker can acquire the image that the person in charge of the device wants to see by using, for example, a device with a camera that he / she carries. As a result, for example, the person in charge of the equipment can easily visually recognize the place to be seen, and it becomes easy to grasp the state of the manufacturing equipment. In other words, for example, the person in charge of equipment can correctly grasp the situation of the manufacturing equipment and can correctly convey the work contents to the field worker. As a result, as compared with the remote control technology for personal computers, for example, the person in charge of equipment can quickly and accurately grasp the situation of the manufacturing equipment and then take measures suitable for the situation.

However, even if the remote work support system is adopted, since the on-site worker takes a picture according to the intention of the person in charge of the equipment, for example, the on-site worker is restrained as a photographer for a long time. At this time, for example, if the person in charge of the equipment looks at the same place over and over again in order to find the cause of the accident of the manufacturing equipment, or looks at the manufacturing equipment while operating it, the on-site work is performed. The time required for shooting by a person becomes longer. As a result, for example, the work load on the field worker may increase. Further, for example, for the person in charge of equipment, the more work requested to the field worker, the greater the feeling of regret for the field worker. As a result, for example, the mental burden on the person in charge of the device may increase. For this reason, for example, if there is a technology that allows the equipment staff to check the manufacturing equipment from various angles around them at their own pace without forcing the field workers to do a lot of work, the equipment staff and the field workers Work efficiency can be improved.

In response to such a request, for example, it is conceivable to apply an image processing technique for obtaining an image of a manufacturing device viewed from various angles around it. As such an image processing technique, for example, a technique for generating and displaying a reconstructed image of an imaged object viewed from an arbitrary angle by using an image (multi-viewpoint image) obtained by imaging from many viewpoints. Conceivable. In this technique, for example, parameters required for image reconstruction calculated from the viewpoint position of the observer and the type of image output device are used, and for each pixel of the reconstructed image, the corresponding pixel of the multi-view image is obtained. Obtained by calculation, a reconstructed image is generated.

However, in the image processing for generating such a reconstructed image, a large amount of calculation is required for calculating the parameters necessary for reconstructing the image. Therefore, there is room for improvement in the image processing for obtaining an image of the manufacturing equipment viewed from various angles around it in terms of reducing the amount of calculation.

Such a problem is common not only when the manufacturing equipment in which the accident has occurred is an image pickup object, but also in a general scene where it is desired to obtain an image of a three-dimensional image pickup object viewed from various angles around it. That is, there is room for improvement in the image processing for obtaining an image of the imaged object viewed from various angles around it in terms of reducing the amount of calculation.

Therefore, the inventors of the present application have created a technique capable of reducing the amount of calculation for image processing for obtaining an image of an imaged object viewed from various angles around it.

Hereinafter, various embodiments will be described with reference to the drawings. In the drawings, parts having the same structure and function are designated by the same reference numerals, and duplicate description will be omitted in the following description. The drawings are schematically shown. A right-handed XYZ coordinate system is attached to FIGS. 4, 7, 8 and 11. In this XYZ coordinate system, the first direction along the horizontal direction shown in FIG. 4 is the + X direction, the second direction orthogonal to the + X direction and along the horizontal direction is the + Y direction, and the + X direction and the + Y direction. The direction orthogonal to both of them is the + Z direction.

<1. First Embodiment>
<1-1. Image processing system>
As shown in FIG. 1, the image processing system 1 includes a first device 2, a second device 3, a communication line 4, and a base station 5. Here, the first device 2 is communicably connected to the base station 5 by, for example, a wireless system or a wired system. The base station 5 is communicably connected to the communication line 4 by, for example, a wired system or a wireless system. The second device 3 is communicably connected to the communication line 4 by, for example, a wired system or a wireless system. For example, a network line such as the Internet is applied to the communication line 4. Therefore, the second device 3 is communicably connected to the first device 2 via, for example, the communication line 4.

The first device 2 is, for example, a device that can be used by the first user U1. In the first embodiment, the first device 2 has, for example, an imaging function and a communication function. Specifically, a mobile device having an imaging function and a communication function, such as a smartphone or a tablet terminal, is applied to the first device 2. The first user U1 includes, for example, a person (also referred to as a field worker) who works on an object to be imaged (also referred to as an imaged object) at a site such as a factory. The object to be imaged may include various devices such as manufacturing equipment, inspection equipment, and analysis equipment.

The second device 3 is, for example, a device that can be used by the second user U2. In the first embodiment, the second device 3 has, for example, a communication function and an image display function. Specifically, a device having a communication function and an image display function, such as a personal computer or a tablet terminal, is applied to the second device 3. The second user U2 includes, for example, a person in charge of a manufacturer (also referred to as a person in charge of equipment) who has developed, produced, or sold various devices as an image pickup object. The person in charge of the device includes, for example, a developer of the device.

In such an image processing system 1, for example, the second device 3 can send various instructions to the first device 2 via the communication line 4. Further, for example, the second device 3 can acquire various information acquired by the first device 2 via the communication line 4.

<1-2. Configuration of the first device>
<1-2-1. Electrical configuration of the first device>
As shown in FIG. 2, the first device 2 includes, for example, a control unit 200, an imaging unit 21, a position sensor 22, a direction sensor 23, a distance measuring sensor 24, a display unit 25, a speaker 26, a communication unit 27, and a touch panel 28. And the operation button group 29 is provided.

The control unit 200 is, for example, a kind of arithmetic processing unit, and includes, for example, an electric circuit such as a CPU (Central Processing Unit) 210, a DSP (Digital Signal Processor) 220, and a storage medium 230. The control unit 200 can collectively manage the operation of the first device 2 by controlling other components of the first device 2, for example. The first device 2 may further include a co-processor such as a SoC (System-on-a-Chip), an MCU (Micro Control Unit), and an FPGA (Field-Programmable Gate Array). good. Further, the first device 2 may perform various controls by coordinating both the CPU (Central Processing Unit) and the sub-processing device, or by selectively using one of the two.

The control unit 200 includes at least one processor to provide control and processing power for performing various functions, as described in more detail below.

According to various embodiments, at least one processor may be run as a single integrated circuit (IC) or as multiple communicable integrated and / or discrete circuits. .. At least one processor can be run according to a variety of known techniques.

In one embodiment, the processor comprises, for example, one or more circuits or units configured to perform one or more data calculation procedures or processes by executing instructions stored in the associated memory. In other embodiments, the processor may be firmware (eg, a discrete logic component) configured to perform one or more data computation procedures or processes.

According to various embodiments, the processor is one or more processors, controllers, microprocessors, microcontrollers, application specific integrated circuits (ASICs), digital signal processing devices, programmable logic devices, field programmable gate arrays, or these. Any combination of devices or configurations, or other known device and configuration combinations, may be included to perform the functions described below.

The storage medium 230 includes a non-temporary recording medium such as a ROM (Read Only Memory) and a RAM (Random Access Memory) that can be read by the CPU 210 and the DSP 220. The ROM included in the storage medium 230 is, for example, a flash ROM (flash memory) which is a non-volatile memory 230b. The RAM included in the storage medium 230 is, for example, the volatile memory 230a. The storage medium 230 stores a main program for controlling the first device 2, a plurality of application programs (also referred to as applications), and the like. Various functions of the control unit 200 are realized by the CPU 210 and the DSP 220 executing various programs in the storage medium 230. The storage medium 230 stores, for example, an imaging application for capturing a still image or a moving image using the imaging unit 21, and an application for performing communication. The application stored in the storage medium 230 includes, for example, a control program 2Pr for controlling the first device 2. Here, in the first device 2, the function as the first image processing device 20 can be realized by executing the control program 2Pr on the CPU 210 or the like. In other words, the first image processing device 20 can be controlled by the control program 2Pr.

The storage medium 230 may include computer-readable non-temporary recording media other than ROM and RAM. The storage medium 230 may include, for example, a small hard disk drive and an SSD (Solid State Drive). A storage medium other than the storage medium 230 may exist. This storage medium may be located outside the control unit 200. The storage destination of the information described later may be the storage medium 230 or another storage medium. All the functions of the control unit 200 or some functions of the control unit 200 may be realized by hardware that does not require software to realize the functions. The control unit 200 may be formed by a circuit. The configuration of the control unit 200 can be applied to other control units described later.

The image pickup unit 21 includes, for example, a lens and an image pickup device. The imaging unit 21 can be controlled by the control unit 200. The image pickup unit 21 receives light from the image pickup device via the lens to generate an image pickup image. As a result, the imaging unit 21 can acquire the captured image. The imaging unit 21 can output the obtained captured image to the control unit 200. The captured image acquired by the imaging unit 21 may be in the form of a still image, or may be in the form of a moving image by repeatedly (for example, periodically) generating an captured image. Here, for example, if the first user U1 repeatedly moves and images the imaged object while carrying the first device 2, the imaging unit 21 determines the position of the viewpoint (also referred to as the viewpoint position) and the direction of the line of sight. An imaging object can be imaged under conditions in which combinations (also referred to as line-of-sight directions) are different from each other. As a result, the image pickup unit 21 can obtain, for example, a plurality of images (also referred to as captured images) related to the image pickup target. Here, the viewpoint position indicates, for example, the position of the imaging unit 21. The line-of-sight direction indicates, for example, the direction in which the subject is located on the optical axis of the imaging unit 21 with reference to the viewpoint position (also referred to as the imaging direction). In addition, the imaging unit 21 can output the obtained captured image to the control unit 200. At this time, in the control unit 200, a plurality of captured images sequentially acquired by the imaging unit 21 are stored in the storage medium 230. As a result, for example, a database (also referred to as an image DB) 2Id in which a plurality of captured images are stored is constructed in the non-volatile memory 230b or the like.

The position sensor 22 can obtain information for measuring the position of the first device 2, for example. For example, a GPS (Global Positioning System) sensor or the like is applied to the position sensor 22. The position sensor 22 can output the obtained information to the control unit 200. At this time, in the control unit 200, for example, the information obtained by the position sensor 22 is stored in the storage medium 230 in the image DB 2Id in a state associated with the corresponding captured image.

The direction sensor 23 can obtain information for measuring the orientation of the first device 2, for example. An electronic compass such as a 3-axis geomagnetic sensor is applied to the direction sensor 23. The direction sensor 23 can output the obtained information to the control unit 200. At this time, in the control unit 200, for example, the information obtained by the direction sensor 23 is stored in the storage medium 230 in the image DB 2Id in a state associated with the corresponding captured image.

The distance measuring sensor 24 can obtain information for measuring the distance from the first device 2 to the image pickup target, for example. For example, a distance image sensor is applied to the distance measuring sensor 24. As the method of the distance image sensor, for example, a method of measuring the distance to the image pickup object by using infrared rays in a non-contact manner is applied. The distance measuring sensor 24 can output the obtained information to the control unit 200. At this time, in the control unit 200, for example, the information obtained by the distance measuring sensor 24 is stored in the storage medium 230 in the image DB 2Id in a state associated with the corresponding captured image.

The display unit 25 has, for example, a display panel and a display screen. The display panel is located, for example, on the front side, which is one surface of the first device 2. For this display panel, for example, a liquid crystal panel or an organic EL (Electro Luminescence) panel can be adopted. The display panel can display various information such as images, characters, symbols, and figures by being controlled by the control unit 200. Various information visually output by the display panel is displayed on the display screen. As a result, the display unit 25 can visually output various types of information in a manner recognizable by the first user U1, for example. Here, the various information may include, for example, an captured image acquired by the imaging unit 21, various information received by the communication unit 27, and the like.

The speaker 26 can output sound, for example. For example, a Thaimic speaker may be applied to the speaker 26. For example, in the speaker 26, an electrical sound signal from the control unit 200 can be converted into sound and output. As a result, the speaker 26 can output, for example, voice information in a manner that can be audibly recognized by the first user U1.

The communication unit 27 is, for example, an interface for communication and has an antenna. The communication unit 27 receives, for example, a signal from a device other than the first device 2 or a signal from the second device 3 connected to the communication line 4 by an antenna via, for example, the base station 5. It is possible.

As described above, for example, the display unit 25 can visually output the information, the speaker 26 can output the information audibly, and the communication unit 27 can output the information as data. It is possible to output in the format of. Therefore, for example, the display unit 25, the speaker 26, and the communication unit 27 may form an output unit 2ou that outputs information.

Further, the communication unit 27 can, for example, appropriately perform predetermined signal processing on the reception signal at the antenna and output the processed reception signal to the control unit 200. For example, the communication unit 27 can perform predetermined signal processing on the signal generated by the control unit 200 as appropriate, and wirelessly transmit the processed transmission signal from the antenna. The transmission signal from the antenna is received by, for example, a device other than the first device 2 or a second device 3 connected to the communication line 4 through the base station 5. As a result, the communication unit 27 can communicate with the second device 3 via, for example, the communication line 4.

The touch panel 28 can detect an operation by an operator such as a finger on the display screen, for example. For example, a projection type capacitance type touch panel may be applied to the touch panel 28. When the first user U1 operates the display screen with an operator such as a finger, the touch panel 28 inputs an electric signal corresponding to the operation to the control unit 200 from the touch panel 28. As a result, the control unit 200 can specify the content of the operation performed on the display screen based on the electric signal from the touch panel 28, and perform processing according to the content. The first user U1 can also give various instructions to the first device 2 by operating the display screen with an operator other than the finger, for example, a pen for an electrostatic touch panel such as a stylus pen. The touch panel 28 can scroll the range of the image displayed on the display screen of the display unit 25 in response to a swipe operation on the display screen, for example. Further, the touch panel 28 can enlarge the image displayed on the display screen of the display unit 25, for example, in response to a pinch-in operation on the display screen. Further, the touch panel 28 can reduce the image displayed on the display screen of the display unit 25, for example, in response to a pinch-out operation on the display screen.

When the operation button included in the operation button group 29 is operated by the first user U1, the operation button group 29 can output an operation signal indicating that the operation is performed to the control unit 200. As a result, the control unit 200 can determine whether or not the operation button has been operated based on the operation signal from the operation button. The control unit 200 can perform processing according to the operated operation button. The operation button may be a software button displayed on the display screen instead of a hardware button such as a push button. In this case, for example, an operation on the software button by the first user U1 is detected by the touch panel 28, and the control unit 200 can perform processing according to the operated software button.

As described above, for example, the touch panel 28 and the operation button group 29 can be an input unit 2in for inputting an operation signal in response to the operation of the first user U1. The input unit 2in may include, for example, a pointing device other than the touch panel 28 and the operation button group 29.

<1-2-2. Functional configuration of the first image processing device>
FIG. 3 is a functional block diagram showing an example of the functional configuration of the first image processing device 20 when the first device 2 functions as the first image processing device 20. As shown in FIG. 3, the first image processing device 20 includes, for example, an image pickup unit 21, a position measurement unit 221; a direction measurement unit 231 and a distance measurement unit 241; a resolution calculation unit 211, a communication unit 27, and an instruction generation unit 212. , The output unit 2 ou and the input unit 2 in are provided. Here, the resolution calculation unit 211 and the instruction generation unit 212 are functional configurations that can be realized by, for example, executing the control program 2Pr stored in the storage medium 230 in the control unit 200.

The imaging unit 21 can image, for example, the situation of the object to be imaged and its surroundings.

The position measurement unit 221 can be realized as a functional configuration by the cooperation of the position sensor 22 and the control unit 200, for example. The position measurement unit 221 measures, for example, the position of the imaging unit 21 at the time of imaging for each of the plurality of captured images acquired by the imaging unit 21, so that information indicating the viewpoint position at the time of imaging (viewpoint position information). Also called) is obtained. In this position measurement unit 221, the viewpoint position can be calculated by, for example, a calculation based on the GPS measurement technique. Here, for example, a correction operation may be added according to the transmission / reception of a signal between the base station of the wireless LAN (Local Area Network) and the position sensor 22. Here, for example, instead of GPS, GLONASS (Global Navigation Satellite System), Galileo, or Quasi-Zenith Satellite System (QZSS) in the Global Navigation Satellite System (GNSS), etc. Other positioning systems may be employed. Further, here, for example, a gyro and an acceleration sensor may be applied to the position sensor 22, and the position measurement unit 221 may calculate the viewpoint position based on the detection value of the position sensor 22. In the control unit 200, for example, the viewpoint position information obtained by the position measurement unit 221 is stored in the storage medium 230 in a state associated with the corresponding captured image in the image DB 2Id.

The direction measuring unit 231 can be realized as a functional configuration by the cooperation of the direction sensor 23 and the control unit 200, for example. The direction measurement unit 231 measures, for example, the orientation of the imaging unit 21 at the time of imaging for each of the plurality of captured images acquired by the imaging unit 21, thereby indicating information indicating the line-of-sight direction at the time of imaging (line-of-sight direction information). Also called) is obtained. In this direction measuring unit 231, for example, the line-of-sight direction to which the imaging unit 21 is facing can be calculated based on the detected value of the three-axis geomagnetic sensor. For example, a tilt sensor may be added to the direction sensor 23. In the control unit 200, for example, the line-of-sight direction information obtained by the direction measurement unit 231 is stored in the storage medium 230 in a state associated with the corresponding captured image in the image DB2Id.

As described above, in the acquisition unit 2ac including the image pickup unit 21, the position measurement unit 221 and the direction measurement unit 231, the plurality of captured images and the viewpoint position information and the line-of-sight direction related to each of the plurality of captured images at the time of imaging are obtained. Information and can be obtained.

Here, as shown in FIG. 4, for example, it is assumed that the first user U1 orbits the image-imaging object Ob0 and images the image-imaging object Ob0 and its surroundings in chronological order. In this case, for example, imaging in the following viewpoint position and line-of-sight direction can be performed in order. First, the first user U1 performs imaging in the line-of-sight direction v1 and imaging in the line-of-sight direction v2 at the viewpoint position p1. Next, the first user U1 moves from the viewpoint position p1 to the viewpoint position p2, and at the viewpoint position p2, takes an image in the line-of-sight direction v3 and an image in the line-of-sight direction v4. Next, the first user U1 moves from the viewpoint position p2 to the viewpoint position p3, and at the viewpoint position p3, takes an image in the line-of-sight direction v5 and an image in the line-of-sight direction v6. Next, the first user U1 moves from the viewpoint position p3 to the viewpoint position p4, and performs imaging in the line-of-sight direction v7 and imaging in the line-of-sight direction v8 at the viewpoint position p4. Next, the first user U1 moves from the viewpoint position p4 to the viewpoint position p5, and performs imaging in the line-of-sight direction v9 and imaging in the line-of-sight direction v10 at the viewpoint position p5. At this time, for example, in the acquisition unit 2ac, 10 captured images can be acquired by 10 times of imaging, and the viewpoint position information and the line-of-sight direction information can be acquired for each captured image.

The distance measuring unit 241 can be realized as a functional configuration by the cooperation of the distance measuring sensor 24 and the control unit 200, for example. In the distance measuring unit 241 for example, information indicating the distribution of the distance from the viewpoint position at the time of imaging to the surface of the imaged object Ob0 for each of the plurality of captured images acquired by the imaging unit 21 (also referred to as distance distribution information). Is obtained. Here, for example, for all the pixels constituting the captured image acquired by the imaging unit 21, the distance from the viewpoint position to the surface portion of the imaging target Ob0 captured by each pixel can be obtained. The distance distribution information is also referred to as, for example, a distance image. In the distance measuring unit 241 for example, the distribution of the distance from the viewpoint position to the surface of the image pickup object Ob0 can be calculated by the calculation according to the TOF (Time of Flight) method based on the detected value of the distance measuring sensor 24. Here, for example, the distance measuring sensor 24 is a sensor that obtains information for performing triangulation using two viewpoints, and the distance measuring unit 241 acquires the distance distribution information by a calculation based on the triangulation technique. May be good. Here, for example, if the acquisition unit 2ac includes the distance measurement unit 241, the acquisition unit 2ac can also acquire the distance distribution information related to each of the plurality of captured images at the time of imaging.

For example, for each captured image obtained by the imaging unit 21, the resolution calculation unit 211 sets the imaging target Ob0 based on the distance distribution information obtained by the distance measuring unit 241 at the time of imaging and the characteristic information of the imaging unit 21. The resolution of the captured image portion (also called the imaging resolution) can be calculated. Here, the characteristic information includes, for example, information on the number of effective pixels in the imaging unit 21 (also referred to as pixel number information) and information indicating the angle of view (also referred to as viewing angle) of the imaging unit 21 (also referred to as angle of view information). Is included. The pixel number information may be, for example, information indicating the number of pixels of the captured image. The number of effective pixels of the imaging unit 21 and the number of pixels of the captured image include, for example, the number of pixels in the horizontal direction and the number of pixels in the vertical direction. As the imaging resolution, for example, the number of pixels corresponding to the unit length of the surface of the imaging object Ob0 in the captured image is adopted. Here, for example, for each pixel of the captured image, the unit length portion of the surface of the imaged object Ob0 is set based on the distance to the surface of the imaged object Ob0 captured from the viewpoint position and the angle of view. The number of pixels being represented can be calculated. Here, for example, if the acquisition unit 2ac includes the resolution calculation unit 211, the acquisition unit 2ac can also acquire information indicating the imaging resolution (also referred to as imaging resolution information) for each of the plurality of captured images.

For example, the communication unit 27 transmits the plurality of captured images acquired by the imaging unit 21 and the viewpoint position information and the line-of-sight direction information for each of the plurality of captured images to the second device 3 via the communication line 4. Can be sent to. These information can be transmitted by the communication unit 27 to the second device 3 via the communication line 4 at each timing when the captured image is acquired by the imaging unit 21, or at a preset timing, for example. The preset timing may be, for example, a timing at a fixed time interval, a timing for each acquisition of a predetermined number of captured images, or in response to an operation of the first user U1 or the like. It may be the timing to respond to the signal input by the input unit 2in. Here, in the communication unit 27, for example, along with the captured image, other information including at least one of the line-of-sight position information and the line-of-sight direction information corresponding to the captured image, and the distance distribution information and the imaging resolution information corresponding to the captured image. May be transmitted to the second device 3 via the communication line 4. In addition, other information may include, for example, characteristic information of the imaging unit 21.

The instruction generation unit 212 generates information (also referred to as instruction information) indicating an instruction according to the request based on, for example, information indicating various requests (also referred to as request information) received by the communication unit 27. The instruction information includes, for example, information (also referred to as imaging instruction information) for instructing imaging in a specific viewpoint position and line-of-sight direction with respect to the imaging object Ob0. The imaging instruction information may include, for example, information (also referred to as guide information) including a map that guides the first user U1 to a certain viewpoint position and a certain line-of-sight direction.

The output unit 2ou can visually output various information such as the captured image acquired by the imaging unit 21 and the imaging instruction information generated by the instruction generation unit 212. For example, the first user U1 carries the first device 2 and confirms the imaging instruction information shown on the display unit 25 constituting the output unit 2ou, and then images the imaging object Ob0 according to the imaging instruction information. Can be carried out.

The input unit 2in can input various signals in the first image processing device 20 according to the operation of the first user U1, for example. At this time, in the first image processing device 20, for example, an operation such as imaging according to various input signals can be performed.

<1-3. Configuration of the second device>
<1-3-1. Electrical configuration of the second device>
As shown in FIG. 5, the second device 3 includes, for example, a control unit 300, a display unit 35, a speaker 36, a communication unit 37, a touch panel 38, and an operation unit 39.

The control unit 300 is, for example, a kind of arithmetic processing unit, and includes, for example, a CPU (Central Processing Unit) 310 which is an electric circuit, a storage medium 330, and the like. The control unit 300 can collectively manage the operation of the second device 3 by controlling other components of the second device 3, for example. The second device 3 is, for example, an auxiliary processing device (co) such as a SoC (System-on-a-Chip), an MCU (Micro Control Unit), and an FPGA (Field-Programmable Gate Array), similarly to the first device 2. -processor) may be further included. Further, in the second device 3, for example, similarly to the first device 2, both the CPU (Central Processing Unit) and the sub-processing device are made to cooperate, or one of the two is selectively used. Various controls may be performed.

The control unit 300, like the control unit 200 of the first device 2, for example, is at least one in order to provide control and processing power for performing various functions, as described in more detail below. Includes one processor.

The storage medium 330 has, for example, a non-volatile memory 330b such as a ROM (Read Only Memory) and a volatile memory 330a such as a RAM (Random Access Memory). The ROM may include, for example, a flash ROM (flash memory). The storage medium 330 stores a main program for controlling the second device 3, a plurality of application programs (also referred to as applications), and the like. Various functions of the control unit 300 are realized by the CPU 310 executing various programs in the storage medium 330. The storage medium 330 stores, for example, an image display application for displaying a plurality of captured images transmitted from the first device 2. The application stored in the storage medium 330 includes, for example, a control program 3Pr for controlling the second device 3. Here, in the second device 3, the function as the second image processing device 30 can be realized by executing the control program 3Pr on the CPU 310 or the like. In other words, the second image processing device 30 can be controlled by the control program 3Pr.

The storage medium 330 may include computer-readable non-temporary recording media other than ROM and RAM. The storage medium 330 may include, for example, a hard disk drive and an SSD (Solid State Drive). A storage medium other than the storage medium 330 may exist. This storage medium may be located outside the control unit 300. The storage destination of the information described later may be the storage medium 330 or another storage medium. All the functions of the control unit 300 or some functions of the control unit 300 may be realized by hardware that does not require software to realize the functions. The control unit 300 may be formed by a circuit.

The display unit 35 has, for example, a display panel and a display screen, similarly to the display unit 25 of the first device 2. The display panel can display various information such as images, characters, symbols, and figures by being controlled by the control unit 300. Various information visually output by the display panel is displayed on the display screen. As a result, the display unit 35 can visually output various types of information in a manner recognizable by the second user U2, for example. Here, various types of information may include, for example, an captured image received by the communication unit 37.

The speaker 36 can output sound, for example, in the same manner as the speaker 26 of the first device 2. For example, in the speaker 36, an electrical sound signal from the control unit 300 can be converted into sound and output. As a result, the speaker 36 can output voice information in a manner that can be audibly recognized by the second user U2, for example.

The communication unit 37 is, for example, a communication interface similar to the communication unit 37 of the first device 2, and is connected to the communication line 4 in a manner capable of transmitting and receiving data. Therefore, the communication unit 37 is communicably connected to the first device 2 via, for example, the communication line 4. For example, the communication unit 37 can appropriately perform predetermined signal processing on the signal generated by the control unit 300, and then transmit the processed transmission signal to the communication line 4. The transmission signal from the communication unit 37 to the communication line 4 is received, for example, by the first device 2 connected to the communication line 4.

As described above, for example, the display unit 35 can visually output the information, the speaker 36 can output the information audibly, and the communication unit 37 can output the information as data. It is possible to output in the format of. Therefore, for example, the display unit 35, the speaker 36, and the communication unit 37 may form an output unit 3ou that outputs information.

Further, the communication unit 37 can receive, for example, a signal from a device other than the second device 3, or a signal from the first device 2 connected to the communication line 4. Here, the communication unit 37 includes, for example, a plurality of captured images transmitted from the first device 2 via the communication line 4, and the viewpoint position information and the line-of-sight direction related to each of the plurality of captured images at the time of imaging. It has a function as a receiving unit 3re capable of receiving information and information. In the receiving unit 3re, for example, the distance distribution information related to each of the plurality of captured images at the time of imaging may be received, or the information indicating the imaging resolution (imaging resolution information) for each of the plurality of captured images is received. May be good.

In other words, here, the communication unit 37 has, for example, a plurality of captured images transmitted from the first device 2 via the communication line 4, and a viewpoint position related to each of the plurality of captured images at the time of imaging. It has a function as an acquisition unit 3ac capable of acquiring information and line-of-sight direction information. In this acquisition unit 3ac, for example, distance distribution information related to each of a plurality of captured images at the time of imaging may be acquired, or information indicating an imaging resolution (imaging resolution information) for each of the plurality of captured images is acquired. May be good.

Further, the communication unit 37 can, for example, appropriately perform predetermined signal processing on the received signal and output the processed reception signal to the control unit 300. For example, the communication unit 37 can output the received captured image to the control unit 300, for example. At this time, in the control unit 300, a plurality of captured images sequentially acquired by the communication unit 37 are stored in the storage medium 330. As a result, for example, a database (also referred to as an image DB) 3Id in which a plurality of captured images are stored is constructed in the non-volatile memory 330b or the like.

The touch panel 38 can detect an operation by an operator such as a finger on the display screen, as in the touch panel 28 of the first device 2, for example. When the second user U2 operates the display screen with an operator such as a finger, the touch panel 38 inputs an electric signal corresponding to the operation to the control unit 300 from the touch panel 38. As a result, the control unit 300 can specify the content of the operation performed on the display screen based on the electric signal from the touch panel 38, and perform processing according to the content. The second user U2 can also give various instructions to the second device 3 by operating the display screen with an operator other than the finger, for example, a pen for an electrostatic touch panel such as a stylus pen. The touch panel 38 can scroll the range of the image displayed on the display screen of the display unit 35 in response to a swipe operation on the display screen, for example. Further, the touch panel 38 can enlarge the image displayed on the display screen of the display unit 35 in response to a pinch-in operation on the display screen, for example. Further, the touch panel 38 can reduce the image displayed on the display screen of the display unit 35 in response to a pinch-out operation on the display screen, for example.

The operation unit 39 includes, for example, a keyboard and a mouse, and can output an operation signal corresponding to the operation by the second user U2 to the control unit 300. As a result, the control unit 300 can determine whether or not the operation unit 39 has been operated based on the operation signal from the operation unit 39. The control unit 300 can perform processing according to the operation of the second user U2 on the operation unit 39, for example. The operation unit 39 may include, for example, a software button displayed on the display screen. In this case, for example, the touch panel 38 detects the operation of the software button by the second user U2, and the control unit 300 can perform the process according to the software button operated by the second user U2.

As described above, for example, the touch panel 38 and the operation unit 39 can be an input unit 3in for inputting an operation signal in response to the operation of the second user U2.

<1-3-2. Functional configuration of the second image processing device>
FIG. 6 is a functional block diagram showing an example of the functional configuration of the second image processing device 30 when the second device 3 functions as the second image processing device 30. As shown in FIG. 6, the second image processing device 30 includes, for example, an acquisition unit 3ac, an output unit 3ou, an input unit 3in, a storage medium 330, an estimation unit 311, a detection unit 312, an image generation unit 313, and a resolution setting unit. It includes 314, an area setting unit 315, an information generation unit 316, and a switching unit 317. Here, the estimation unit 311, the detection unit 312, the image generation unit 313, the resolution setting unit 314, the area setting unit 315, the information generation unit 316, and the switching unit 317 control, for example, the control program 3Pr stored in the storage medium 330. It is a functional configuration that can be realized by executing the part 300.

The acquisition unit 3ac uses, for example, a plurality of captured images transmitted from the first image processing device 20 and viewpoint position information and line-of-sight direction information related to each of the plurality of captured images at the time of imaging as the receiving unit 3re. It can be acquired by reception by the communication unit 37 of the above. Here, for example, if the distance distribution information related to each of the plurality of captured images at the time of imaging is also transmitted from the first image processing device 20, the acquisition unit 3ac receives the distance distribution by the communication unit 37 as the receiving unit 3re. Information can be obtained. Here, for example, if the first image processing device 20 transmits imaging resolution information indicating the imaging resolution for each of the plurality of captured images, the acquisition unit 3ac is imaged by reception by the communication unit 37 as the receiving unit 3re. Resolution information can be acquired.

The output unit 3ou can visually output, for example, an captured image acquired by the acquisition unit 3ac, a composite image generated by the image generation unit 313 described later, and the like. The composite image can be generated, for example, by synthesizing the first image portion and the second image portion in which the first surface and the second surface sandwiching the corner portion of the image pickup object Ob0 are captured, respectively.

The input unit 3in can input various signals in the second image processing device 30 according to the operation of the second user U2, for example. In the input unit 3in, for example, a signal instructing enlargement or reduction of various images displayed on the display unit 35 may be input according to the operation of the second user U2. Further, in the input unit 3in, for example, a signal instructing scrolling of various images displayed on the display unit 35 may be input according to the operation of the second user U2.

The storage medium 330 stores, for example, a plurality of captured images acquired by the acquisition unit 3ac. As a result, in the storage medium 330, for example, an image DB3Id in which a plurality of captured images are stored can be constructed. In the image DB3Id, for example, each captured image can be stored in a state in which the corresponding information is associated with the image DB3Id. Here, the corresponding information includes, for example, viewpoint position information and line-of-sight direction information. The corresponding information may include, for example, distance distribution information or imaging resolution information.

The estimation unit 311 can estimate the existence position of the image pickup target Ob0 based on, for example, the viewpoint position information and the line-of-sight direction information related to each of the plurality of captured images acquired by the acquisition unit 3ac. As a result, the estimation unit 311 can acquire, for example, information (also referred to as existence position information) indicating the existence position of the imaging target Ob0.

Here, a method of estimating the existence position of the imaging object Ob0 in the estimation unit 311 will be described with reference to FIGS. 7 and 8.

Here, for example, for the purpose of reducing the amount of calculation, an approximately representative line-of-sight direction (also referred to as a representative line-of-sight direction) is obtained from the line-of-sight direction at the time of a plurality of imaging as shown in FIG. Be done. In the example of FIG. 7, the range that can be taken in the horizontal 360-degree line-of-sight direction is the angle range Raj (j is an integer of 0 or more, in this case j is an integer of 0 to 8) of 45 degrees as a specific angle. It is divided into eight parts. Here, the representative line-of-sight direction Vj is set for each angle range Raj. Specifically, the representative line-of-sight direction V0 is set in the angle range Ra0. The representative line-of-sight direction V1 is set in the angle range Ra1. The representative line-of-sight direction V2 is set in the angle range Ra2. The representative line-of-sight direction V3 is set in the angle range Ra3. The representative line-of-sight direction V4 is set in the angle range Ra4. The representative line-of-sight direction V5 is set in the angle range Ra5. The representative line-of-sight direction V6 is set in the angle range Ra6. The representative line-of-sight direction V7 is set in the angle range Ra7. Then, the estimation unit 311 obtains the representative line-of-sight direction of the angle range Raj to which the line-of-sight direction belongs with respect to the line-of-sight direction indicated by the line-of-sight direction information. As a result, the representative line-of-sight direction can be obtained for each captured image. Here, each angle range Raj is, for example, an angle other than 45 degrees, and an integer that j can take may be arbitrarily set.

Next, as shown in FIG. 8, a virtual space (also referred to as an object space) Sp0 having a certain size is set so as to include the image pickup object Ob0. This target space Sp0 is divided into a large number of voxels Bx0. FIG. 8 shows a simple example in which the target space Sp0 is divided into 11 sections from the coordinate x1 to the coordinate x11 in the X direction and 7 sections from the coordinate y1 to the coordinate y7 in the Y direction. That is, an example is shown in which the target space Sp0 is divided into 77 (= 11 × 7) voxels Bx0. At this time, each voxel Bx0 has a coordinate xm in the X direction (m is a natural number, m is a natural number from 1 to 11 in the example of FIG. 8) and a coordinate yn in the Y direction (n is a natural number, n is 1 in the example of FIG. 8). It is specified by the coordinates (xm, yn) in combination with the natural numbers from 7 to 7. Here, for example, the larger the number of divisions of the target space Sp0 in the X direction and the Y direction, the more detailed the existing position of the imaging target Ob0 can be estimated.

Here, for example, for each voxel Bx0, the existence position of the image-imaging object Ob0 can be estimated by counting the directions imaged by the first user U1. Specifically, for example, for each captured image, the line-of-sight direction at the time of imaging is effective for each voxel Bx0 in the target space Sp0 through which the line of sight passes at the time of imaging. Here, the voxel Bx0 in the target space Sp0 through which the line of sight passes at the time of imaging can be specified, for example, based on the viewpoint position and the line-of-sight direction at the time of imaging for each captured image. The line-of-sight direction at the time of imaging, which is enabled for each voxel Bx0, is, for example, the representative line-of-sight direction obtained above for the purpose of reducing the amount of calculation.

In the example of FIG. 8, the representative gaze direction enabled for all of the plurality of captured images is indicated by a thick arrow on each voxel Bx0. In the example of FIG. 8, for example, in the voxel Bx0 at the coordinates (x4, y1), the representative line-of-sight direction V4 is valid. For example, in the voxel Bx0 of coordinates (x6, y4), the four most representative line-of-sight directions V0, V3, V4, V6 are considered to be effective. Therefore, for example, it can be estimated that the voxel Bx0 at the coordinates (x6, y4) is the existence position of the image pickup target Ob0. Here, for example, one or more voxels Bx0 in which a number of representative line-of-sight directions equal to or larger than the threshold value are valid may be estimated as the existence position of the image pickup target Ob0. Further, for example, one or more voxels Bx0 surrounded by the locus of movement of the first user U1 may be presumed to be the existence position of the imaging target Ob0.

In this way, when the existence position of the image-imaging object Ob0 is estimated, the first user U1 determines whether the image-imaging object Ob0 is captured in the image-imaging image, or is at his feet or around the image-imaging object Ob0 when moving. It is possible to take multiple images without being aware of whether or not the image is captured. As a result, a mode in which captured images are acquired at regular time intervals in the first device 2 can be easily adopted.

Further, here, the target space Sp0 may be divided not only in the horizontal direction but also in the vertical direction. In this case, for example, the line-of-sight direction at the time of imaging may include an inclination angle calculated by using an inclination sensor. In this case, for example, the control unit 300 can calculate the value of the tilt angle of the first device 2 from the value of the gravitational acceleration detected by the tilt sensor.

The resolution setting unit 314, for example, determines the resolution (display resolution) of the image portion of the image displayed on the display unit 35 that captures the image-imaging object in response to the signal input in response to the operation of the second user U2. Also called) can be set. For example, when the composite image is displayed on the display unit 35, the resolution setting unit 314 responds to the signal input in response to the operation of the second user U2, and the enlarged display image portion of the composite image is displayed. In addition to setting the range of, the display resolution in this enlarged display image portion can be set. The enlarged display image portion may be, for example, the range of the portion of the composite image displayed on the display unit 35 that is enlarged and displayed on the display unit 35.

Here, examples of the operation of the second user U2 include rotation of the mouse wheel or pinch-in and pinch-out operations on the display screen. For example, as shown in FIG. 9A, it is assumed that the image portion Ip0 in which the region Ar0 of the imaging target is captured is displayed on the display screen Sc0. When a pinch-in operation is performed on the display screen Sc0 in this state, the image portion Ip1 in which the region Ar1 included in the region Ar0 of the imaging target is captured is shown in FIG. 9B. , Can be enlarged and displayed on the entire display screen Sc0. Here, for the display resolution, for example, the number of pixels corresponding to the unit length of the surface of the image pickup object Ob0 in the image is adopted as in the above image pickup resolution. The display resolution is, for example, the display range of the image portion that captures the imaging object Ob0 set on the display screen Sc0, the unit length portion of the surface of the imaging object Ob0 in this image portion, and the display screen Sc0. It can be calculated based on the number of pixels in the vertical and horizontal directions. The unit length portion includes, for example, the distance of each pixel of the captured image to the surface of the imaged object Ob0 captured from the viewpoint position, the angle of view, and the enlargement ratio for displaying the captured image. Can be calculated based on. The magnification can be determined, for example, according to the operation of the second user U2.

The area setting unit 315 is, for example, a display target captured by an image visually output by the display unit 35 of the imaging target Ob0 in response to a signal input in response to the operation of the second user U2. The area can be set. Here, the display target area may be, for example, an area that is visually output by the display unit 35 of the image pickup target Ob0. The display target area is, for example, a spatial position of the display target area in the target space Sp0 based on the viewpoint position information, the line-of-sight direction information, the characteristic information including the angle of view of the imaging unit 21, and the existence position of the imaging target Ob0. It can be managed by the coordinates indicating. Here, for example, if the distance distribution information related to the captured image is added, the shape of the portion of the imaged object Ob0 captured in the captured image can be recognized, and the spatial position of the display target area in the target space Sp0 can be recognized. The accuracy of the coordinates indicating the above can be improved.

Further, the area setting unit 315 changes, for example, the display target area captured in the image visually output by the display unit 35 according to the signal input in response to the operation of the second user U2. Can be done. Further, for example, when the composite image is displayed on the display unit 35, the area setting unit 315 displays the imaged object according to the signal input in response to the operation of the second user U2. The display target area captured by the composite image visually output by the unit 35 can be changed.

Here, examples of the operation of the second user U2 include a swipe operation on the display screen Sc0 by the operator. For example, as shown in FIG. 10A, it is assumed that the image portion Ip2 in which the display target area (also referred to as the already displayed area) Ar2 of the imaging target object is captured on the display screen Sc0 is displayed. In this state, for example, when a swipe operation is performed on the display screen Sc0 by the finger Fg0, which is an operator, the image portion Ip2 in which the already displayed area Ar2 is captured moves as shown in FIG. 10B. At this time, on the display screen Sc0, for example, a part of the image portion Ip2 that captures the already displayed area Ar2 disappears outside the display screen Sc0, and the image portion Ip21 that captures the remaining portion Ar21 of the already displayed area Ar2 is displayed. Further, for example, in the portion of the display screen Sc0 where the image portion Ip2 is displaced and does not exist, the image portion Ip22 in which the region (non-display region) Ar22 adjacent to the already displayed region Ar2 of the image pickup object is captured is captured. It may be newly displayed. In this way, for example, the second user U2 can scroll the image displayed on the display screen Sc0 of the display unit 25 by swiping the display screen Sc0.

The detection unit 312 can detect, for example, an image portion corresponding to the display resolution set by the resolution setting unit 314 among a plurality of captured images stored in the image DB 3Id of the storage medium 330.

Here, for example, as shown in FIG. 11, the first user U1 moves around the imaging object Ob0 located in front of the background 600 including the wall 600 wl and the window 600 wd, and the viewpoint positions p11 and p12. , P13, p14, it is assumed that imaging is performed in order using the first device 2. In the example of FIG. 11, the line-of-sight directions from the viewpoint positions p11, p12, p13, and p14 are drawn by arrows. Between imaging, the pair of viewpoint positions p11, p12, p13, p14 and the line-of-sight direction is different from each other. In the example of FIG. 11, the image pickup object Ob0 has a first surface SfA and a second surface SfB located so as to sandwich the corner portion Cn1.

When the imaging is performed under the conditions shown in FIG. 11, for example, FIG. 12A shows the captured image Im11 acquired by the imaging at the viewpoint position p11. For example, FIG. 12B shows the captured image Im12 acquired by imaging at the viewpoint position p12. For example, FIG. 12 (c) shows the captured image Im13 acquired by imaging at the viewpoint position p13. For example, FIG. 12D shows the captured image Im14 acquired by imaging at the viewpoint position p14.

Here, as shown in FIG. 12A, for example, the captured image Im11 is composed of an image portion Ipa that captures the first surface SfA. As shown in FIG. 12B, for example, the captured image Im12 is composed of an image portion Ipa that captures the first surface SfA, an image portion Ipwl that captures the wall 600 wl, and an image portion Ipwd that captures the window 600 wd. There is. Specifically, in the captured image Im12, for example, the first surface SfA is captured in the left portion, and the wall 600 wl and the window 600 wd included in the background 600 are captured in the portion extending from the central portion to the right end. As shown in FIG. 12 (c), for example, the captured image Im13 includes an image portion Ipa that captures the first surface SfA, an image portion Ipb that captures the second surface SfB, an image portion Ipwl that captures the wall 600wl, and a window. It is composed of an image portion Ipwd that captures 600 wd. Specifically, in the captured image Im13, for example, the first surface SfA and the second surface SfB are captured in the portion near the left end portion, and the wall 600 wl and the window 600 wd included in the background 600 are captured in the remaining portion. There is. As shown in FIG. 12D, for example, the captured image Im14 is composed of an image portion Ipa that captures the first surface SfA, an image portion Ipb that captures the second surface SfB, and an image portion Ipwl that captures the wall 600wl. Has been done. Specifically, in the captured image Im14, for example, the first surface SfA is captured in the portion near the left end portion, the second surface SfB is captured in the wide portion including the central portion, and the background 600 is captured in the remaining left portion. The included wall 600 wl is captured.

Here, for example, it is assumed that the display target area is included in the first surface SfA of the image pickup target Ob0. In this case, for example, in the detection unit 312, whether or not the imaging resolution of the image portion to be displayed by the display unit 35 at that time is equal to or higher than the display resolution changed by the resolution setting unit 314. Can be detected.

Further, here, for example, it is assumed that the display target region includes the first surface SfA and the second surface SfB that sandwich the corner portion Cn1 of the image pickup target Ob0. In this case, for example, the detection unit 312 captures the first surface SfA of the imaging target Ob0 from the image DB3Id, detects the first image portion corresponding to the display resolution, and detects the second surface of the imaging target Ob0. SfB can be captured and a second image portion corresponding to the display resolution can be detected.

In this way, the detection unit 312 can detect, for example, the first image portion whose imaging resolution is equal to or higher than the display resolution, in which the first surface SfA of the imaging target Ob0 is captured from the image DB3Id. In the detection unit 312, for example, a second image portion having an imaging resolution equal to or higher than the display resolution, in which the second surface SfB of the imaged object Ob0 is captured, can be detected from the image DB3Id. As a result, the first image portion and the second image portion for compositing can be detected. As a result, for example, in the image generation unit 313 described later, a composite image in which the second user U2 can easily see the image pickup target Ob0 can be easily generated.

Further, the detection unit 312 can detect, for example, an image portion corresponding to the display target area set by the area setting unit 315 among the plurality of captured images stored in the image DB 3Id of the storage medium 330. If the image portion corresponding to the display target area is, for example, an image portion whose imaging resolution is equal to or higher than the display resolution, an image for display having excellent visibility can be generated.

Here, for example, it is assumed that the display target area is included in the first surface SfA of the image pickup target Ob0. In this case, the detection unit 312 can detect, for example, an image portion corresponding to the display target region from the captured image in which the first surface SfA is captured.

Further, here, for example, it is assumed that the display target area after the setting or the change includes the first surface SfA and the second surface SfB that sandwich the corner portion Cn1 of the image pickup target Ob0. In this case, the detection unit 312 captures, for example, from the image DB3Id, a region (also referred to as a first region) included in the display target region after setting or change of the first surface SfA of the imaging target Ob0. The first image portion obtained can be detected. Further, in the detection unit 312, for example, a second image portion in which a region (also referred to as a second region) included in the display target region after setting or change of the second surface SfB of the imaging target Ob0 is captured is captured. Can be detected.

Here, the detection unit 312, for example, for a plurality of captured images in the image DB3Id, based on the viewpoint position information, the line-of-sight direction information, the existence position information, and the characteristic information of the imaging unit 21, the imaged object. Each image portion in which Ob0 is captured can be recognized. At this time, for example, even if the imaging is performed with the imaging object Ob0 deviated from the line of sight, the image portion in which the imaging object Ob0 is captured can be recognized from the captured image.

Then, in the detection unit 312, for example, for the captured image, the first region in the image pickup target Ob0 is set based on the viewpoint position information, the line-of-sight direction information, the existence position information, and the characteristic information of the image pickup unit 21. The first image portion captured in the first line-of-sight direction from the one viewpoint position can be detected. Here, for example, if the first region is a region included in the display target region of the first surface SfA of the imaging target Ob0 and the display target region is managed by the coordinates in the target space Sp0, the first region is the first. The position of the area can be easily recognized. Then, for example, based on the position of the first region and the viewpoint position, line-of-sight direction, and angle of view at the time of imaging, the first image portion in which the first region is captured from the first viewpoint position in the first line-of-sight direction is detected. Can be done.

Further, in the detection unit 312, for example, with respect to the captured image, the second region in the image pickup target Ob0 is set based on the viewpoint position information, the line-of-sight direction information, the existence position information, and the characteristic information of the image pickup unit 21. The second image portion captured in the second line-of-sight direction from the two viewpoint positions can be detected. Here, for example, if the second region is a region included in the display target region of the second surface SfB of the image pickup target Ob0 and the display target region is managed by the coordinates in the target space Sp0, the second region is the second. The position of the area can be easily recognized. Then, for example, based on the position of the second region and the viewpoint position, line-of-sight direction, and angle of view at the time of imaging, the second image portion in which the second region is captured from the second viewpoint position in the second line-of-sight direction is detected. Can be done. Here, for example, the combination of the first viewpoint position and the first line-of-sight direction and the combination of the second viewpoint position and the second line-of-sight direction may be different.

Here, for example, in the imaging target Ob0, if the first region and the second region are adjacent regions, a composite image obtained by synthesizing the adjacent portions that can be seen from different directions of the imaging target Ob0 is obtained. Can be generated. As a result, for example, for the second user U2, it is possible to generate an image having a good listability when the imaged object Ob0 is viewed from various directions around it. The fact that the first region and the second region are adjacent to each other can be recognized, for example, by the presence of an captured image in which the first region and the second region are captured adjacent to each other. At this time, for example, a corner portion exists at the boundary portion between the first region and the second region, and the first region and the second region are based on at least one of the existence position information and the distance distribution information. It can be recognized that and are adjacent at the corners.

By the way, for example, if the first line-of-sight direction is perpendicular to the first surface SfA of the first region, the first image portion is a distortion-free image portion in which the first region is captured from the front. Therefore, the detection unit 312 may detect the first image portion in which the first region is captured from the front, for example, based on the position information, the viewpoint position information, and the line-of-sight direction information of the first region. Further, in the detection unit 312, for example, the first image portion may be detected for a plurality of captured images based on the viewpoint position information, the line-of-sight direction information, the distance distribution information, and the existence position information. At this time, for example, if the distribution of the distance from the viewpoint position to the first region captured in the captured image is substantially constant, the first image portion capturing the first region from the front can be easily and accurately detected. Here, for example, when the surface of the imaged object Ob0 has irregularities or the like, the first area is set to the front if the condition that the variation in the distance from the viewpoint position to the first area captured in the captured image is small is satisfied. The first image portion captured from can be detected. The variability can be represented by statistical values such as variance or standard deviation.

Further, for example, if the second line-of-sight direction is perpendicular to the second surface SfB of the second region, the second image portion is a distortion-free image portion in which the second region is captured from the front. Therefore, the detection unit 312 may detect the second image portion in which the second region is captured from the front, for example, based on the position information, the viewpoint position information, and the line-of-sight direction information of the second region. Further, in the detection unit 312, for example, the second image portion may be detected for a plurality of captured images based on the viewpoint position information, the line-of-sight direction information, the distance distribution information, and the existence position information. At this time, for example, if the distribution of the distance from the viewpoint position to the second region captured in the captured image is substantially constant, the second image portion capturing the second region from the front can be easily and accurately detected. Here, for example, when the surface of the imaged object Ob0 has irregularities or the like, if the condition that the variation in the distance from the viewpoint position to the second region captured in the captured image is small is satisfied, the second region is set to the front. The first image portion captured from can be detected. The variability can be represented by statistical values such as variance or standard deviation.

By the way, as described above, for example, as shown in FIGS. 10A and 10B, when scrolling the range of the image displayed on the display screen Sc0 of the display unit 25, the area is used. The display target area is changed by the setting unit 315. Further, for example, if the composite image is displayed on the display unit 35, the area setting unit 315 changes the display target area captured by the composite image visually output by the display unit 35 of the imaged objects. Can be done.

Here, the changed display target area is, for example, the already-displayed area captured by the composite image (also referred to as the first composite image) that has already been visually output by the display unit 35 in the image pickup target Ob0. The remaining Ar21 of Ar2 and the non-display area Ar22 not captured in the first composite image are included.

In response to the setting of the non-display area Ar22, for example, the detection unit 312 sets the non-display area Ar22 based on the viewpoint position information, the line-of-sight direction information, and the existence position information for the plurality of captured images of the image DB3Id. The captured image portion (also referred to as the third image portion) Ip22 is detected. Here, the third image portion Ip22 can be detected, for example, by a method similar to the detection of the first image portion and the second image portion by the detection unit 312 described above.

The image generation unit 313 can generate a composite image by, for example, combining the first image portion and the second image portion detected by the detection unit 312. Here, for example, the first image portion and the second image portion obtained by imaging the imaging object Ob0 from different directions are synthesized so as to be arranged side by side. Thereby, for example, a composite image which is a pseudo three-dimensional image obtained by capturing the image pickup object Ob0 from a plurality of surrounding angles can be easily acquired with a relatively small amount of calculation.

The composite image generated by the image generation unit 313 may be output by the output unit 3ou or may be stored in the storage medium 330, for example. At this time, the output of the composite image in the output unit 3ou may include, for example, the visible output of the composite image in the display unit 35, the transmission of the composite image data by the communication unit 37, and the like. This makes it possible, for example, to visually recognize or store the composite image in the second image processing device 30, or to visually recognize or store the composite image in another device. Here, if the composite image is visually output on the display unit 35, for example, the second user U2 can visually recognize the composite image in the second image processing device 30.

Further, even when the composite image (first composite image) is displayed on the display screen Sc0, the range of the image may be scrolled on the display screen Sc0. In this case, the image generation unit 313 has, for example, an image portion (also referred to as a fourth image portion) Ip21 that captures the remaining portion Ar21 excluding a part of the already displayed area Ar2, and a third image detected by the detection unit 312. A second composite image can be generated by synthesizing the partial Ip22. Here, if the second composite image is visually output on the display unit 35, for example, in the second image processing device 30, the second user U2 can scroll the composite image.

For example, as shown in FIGS. 13 (a) to 13 (c), the second user U2 can scroll the image displayed on the display unit 35. Specifically, first, as shown in FIG. 13A, the image portion Ipa that captures the first surface SfA of the imaging target Ob0 is displayed on the entire display screen Sc0 of the display unit 35. Next, as shown in FIGS. 13 (a) to 13 (b), the display is performed in response to a swipe operation in which the finger Fg0, which is an operator, is slid from the right side portion of the display screen Sc0 toward the center portion. The image on the screen Sc0 is scrolled from right to left. At this time, on the display screen Sc0, the second surface SfB of the imaging target Ob0 is moved so that the image portion Ipa that captures the first surface SfA of the imaging target Ob0 shifts to the left and continues to the right side of the image portion Ipa. The captured image portion Ipb appears. Further, as shown in FIGS. 13 (b) to 13 (c), the display screen is responded to by a swipe operation in which the finger Fg0, which is an operator, is slid from the central portion of the display screen Sc0 toward the left portion. The image on Sc0 is scrolled from right to left. At this time, on the display screen Sc0, the second surface SfB of the imaging target Ob0 is moved so that the image portion Ipa that captures the first surface SfA of the imaging target Ob0 is further shifted to the left and continues to the right side of the image portion Ipa. The captured image portion Ipb is further shifted to the left and displayed in a large size.

In this way, for example, if the first user U1 performs a series of imaging of the imaged object Ob0 a plurality of times using the first image processing device 20 and accumulates the plurality of captured images, the second user The U2 can arbitrarily scroll the image displayed on the display unit 35. Therefore, for example, the second user U2 can confirm the image pickup target Ob0 from various angles around it at his / her own pace without forcing the first user U1 to perform a great deal of work. Therefore, for example, the work efficiency can be improved for the first user U1 and the second user U2. Further, here, by generating a composite image which is a pseudo three-dimensional image, it is possible to easily obtain an image of the imaged object Ob0 viewed from various angles around it with a relatively small amount of calculation.

The information generation unit 316 can generate, for example, information that requests the imaging of a missing captured image (also referred to as a missing captured image).

Here, for example, when the display unit 35 scrolls the image, the detection unit 312 may not detect the image portion in which the non-display area Ar22 is captured in the plurality of captured images stored in the image DB3Id. In this case, the detection unit 312 has detected a shortage of the image portion that captures the non-display area Ar22. In response to the detection of this shortage, for example, the information generation unit 316 can generate information related to the shortage of the image portion that captures the non-display area Ar22. The information generated here can be output by, for example, the output unit 3ou. Here, the information generated by the information generation unit 316 includes, for example, information requesting imaging of a insufficiently captured image capturing the non-display region Ar22 (also referred to as imaging request information), or imaging capturing the non-display region Ar22. It may include information indicating that the image is missing.

Further, here, for example, when enlarging an image on the display unit 35, one or more enlarged displays having an imaging resolution equal to or higher than the display resolution in a plurality of captured images stored in the image DB 3Id by the detection unit 312 are used. The image part may not be detected. Here, the imaging resolution may be, for example, an imaging resolution related to the range of the enlarged display image portion set by the resolution setting unit 314. In this case, the detection unit 312 has detected a shortage of captured images for enlarged display. In response to the detection of this shortage, for example, the information generation unit 316 may generate information indicating the shortage of the captured image for enlarged display. The information generated here can be output by, for example, the output unit 3ou. Here, the information generated by the information generation unit 316 is, for example, lacking information requesting imaging of a insufficiently captured image for enlarged display (imaging request information) or insufficient captured image for enlarged display. The information shown may be included.

The imaging request information may include, for example, information indicating the imaging conditions of the insufficiently captured image. This imaging condition may include, for example, information such as a viewpoint position, a line-of-sight direction, and an imaging resolution. Then, the information generated by the information generation unit 316 can be transmitted to the first image processing device 20 by, for example, the communication unit 37 constituting the output unit 3ou. At this time, in the first image processing device 20, for example, the instruction generation unit 212 generates instruction information indicating an instruction according to the image pickup request based on the image pickup request information, and the output unit 2ou generates the image pickup instruction information. Various information such as can be output visually. Here, the imaging instruction information includes, for example, instruction information for instructing imaging in a specific viewpoint position and line-of-sight direction with respect to the imaging object Ob0. This instruction information may include, for example, guide information such as a map that guides the first user U1 to a specific viewpoint position and line-of-sight direction. As a result, for example, when a shortage of captured images occurs, the first user U1 may be required to perform additional imaging.

Further, the information generated by the information generation unit 316 may be visually output by, for example, the display unit 35 constituting the output unit 3ou, or audibly output by the speaker 36 or the like constituting the output unit 3ou. May be done. Thereby, for example, the second user U2 can recognize the shortage of the image portion for compositing the scroll destination or the shortage of the image portion necessary for the enlarged display. At this time, the image generation unit 313, for example, generates a temporary image portion by interpolation processing or the like for the missing image portion until the insufficient captured image is acquired by the acquisition unit 3ac, and generates a composite image or the like. It may be used for. At this time, for example, the display unit 35 may display a comment on the temporary image portion indicating that the image is requested to be captured or that the image is waiting to be replenished. Then, in the image generation unit 313, for example, if the insufficient captured image is acquired by reception in the acquisition unit 3ac, an appropriate image portion is detected from the supplemented captured image instead of the temporary image portion, and the composite image is displayed. It may be used for generation and the like. At this time, for example, in the display unit 35, the temporary image portion may be replaced with an appropriate image portion, and the comment indicating that imaging is requested may be deleted.

The switching unit 317 is, for example, in a state where two or more captured images are acquired in time sequence by the acquisition unit 3ac, and the image generation unit 313 responds to a signal input in response to the operation of the second user U2. The mode for generating an image for output (also called an image generation mode) can be switched. In the switching unit 317, for example, the image generation mode can be selectively switched between the first image generation mode, the second image generation mode, and the third image generation mode. Thereby, for example, in a state where two or more captured images are acquired in chronological order, an image for display according to the intention of the second user U2 can be generated and displayed.

For example, in the first image generation mode, the image generation unit 313 can prepare an image for output using the captured image each time the captured image is acquired by the acquisition unit 3ac.

Here, for example, as shown in FIG. 14A, it is assumed that a plurality of captured images in which the imaged object Ob0 is captured by the path indicated by the arrow of the thick alternate long and short dash line are acquired. In the example of FIG. 14A, it is shown that a plurality of captured images capturing the region Ai1 of the imaging target Ob0 have already been acquired by the acquisition unit 3ac. At this time, for example, if the captured image It0 shown in FIG. 14B is acquired by the acquisition unit 3ac at time t0, as shown in FIG. 14C, an image for output is obtained from the captured image It0. Will be generated. Next, for example, if the captured image It1 shown in FIG. 14D is acquired by the acquisition unit 3ac at time t1, an image for output is obtained from the captured image It1 as shown in FIG. 14E. Will be generated. Next, for example, at time t2, if a new captured image It2 is not acquired by the acquisition unit 3ac as shown in FIG. 14 (f), the captured image It1 is shown in FIG. 14 (g). An image for output is generated from. Such a first image generation mode may be selected, for example, when the second user U2 does not perform an operation for changing the image generation mode.

For example, in the second image generation mode, the image generation unit 313 synthesizes the first image portion and the second image portion detected by the detection unit 312 with respect to the plurality of captured images already acquired by the acquisition unit 3ac. This makes it possible to generate a composite image as an image for output.

Here, for example, as shown in FIG. 15A, it is assumed that a plurality of captured images capturing the region Ai1 of the imaging target Ob0 have already been acquired by the acquisition unit 3ac. In this case, for example, first, as shown in FIG. 15A, by swiping the display screen Sc0 of the second user U2 with the finger Fg0, which is an operator, the image-imaging object Ob0 is within the already imaged area Ai1. The display target area Ar2 is set in. At this time, as shown in FIG. 15B, the image portion Ip2 in which the display target area Ar2 of the image pickup target Ob0 is captured is displayed on the display screen Sc0.

Next, for example, as shown in FIGS. 15 (b) to 15 (c), by swiping the display screen Sc0 with the finger Fg0, as shown in FIG. 15 (a), the imaging target Ob0 is imaged. In the completed area Ai1, the display target area Ar2 is changed to the display target area Ar3. At this time, as shown in FIGS. 15 (b) to 15 (c), the image portion Ip2 in which the display target area Ar2 is captured moves. At this time, on the display screen Sc0, for example, a part of the image portion Ip2 that captures the display target area Ar2 disappears outside the display screen Sc0, and the image portion that captures the remaining Ar21 of the display target area Ar2 (fourth image portion). Ip21 is displayed. Further, as shown in FIG. 15C, for example, the portion of the display screen Sc0 where the image portion Ip2 is displaced and does not exist is captured by the region Ar22 adjacent to the display target region Ar2 of the image pickup target. The combined image portion (third image portion) Ip22 can be used to display a composite image. At this time, for example, as described above, the image generation unit 313 synthesizes the fourth image portion Ip21 and the third image portion Ip22 to generate a composite image, and the composite image is displayed on the display unit 35. In this way, for example, even while a plurality of captured images are being acquired, the second user U2 can swipe the display screen Sc0 to display the image displayed on the display screen Sc0 of the display unit 25. You can scroll the range of.

For example, in the third image generation mode, the image generation unit 313 uses the first image portion detected by the detection unit 312 for a plurality of captured images already acquired by the acquisition unit 3ac and the most recently acquired image. The second image portion detected by the detection unit 312 can be combined with the second image portion. Here, the second image portion can be detected by the detection unit 312 from the captured image, for example, in response to the acquisition of the captured image by the acquisition unit 3ac. Then, in the image generation unit 313, for example, a composite image as an image for output can be generated by synthesizing the first image portion and the second image portion.

Here, for example, as shown in FIG. 16A, it is assumed that a plurality of captured images capturing the region Ai1 of the imaging target Ob0 have already been acquired by the acquisition unit 3ac. In this case, for example, as shown in FIG. 16A, by swiping the display screen Sc0 of the second user U2 with the finger Fg0, which is an operator, the image-imaging object Ob0 is placed in the already imaged area Ai1. The display target area Ar2 is set. At this time, as shown in FIG. 16B, the image portion Ip2 in which the display target area Ar2 of the image pickup target Ob0 is captured is displayed on the display screen Sc0.

Next, for example, as shown in FIGS. 16B to 16C, the display target area Ar3 is set as shown in FIG. 16A by swiping the display screen Sc0 with the finger Fg0. Will be done. In the example of FIG. 16A, the display target area Ar3 is located so as to straddle the boundary portion between the imaged region Ai1 of the image pickup target Ob0 and the region Acn newly acquired by the acquisition unit 3ac. .. At this time, as shown in FIGS. 16 (b) to 16 (c), the image portion Ip2 in which the display target area Ar2 is captured moves. At this time, on the display screen Sc0, for example, a part of the image portion Ip2 that captures the display target area Ar2 disappears outside the display screen Sc0, and the fourth image portion Ip21 that captures the remaining portion Ar21 of the display target area Ar2 is displayed. NS. Further, as shown in FIG. 16C, for example, in the portion of the display screen Sc0 where the image portion Ip2 is displaced and does not exist, the region Ar22 adjacent to the display target region Ar2 of the image pickup target is captured. The resulting third image portion Ip22 can be used to display a composite image.

However, the third image portion Ip22 has a portion that is not covered by the captured image acquired by the acquisition unit 3ac. Therefore, for example, if the latest captured image Icn (FIG. 16 (e)) that captures the region Acn of the imaged object Ob0 is acquired by the acquisition unit 3ac, a part of the latest captured image Icn is generated as a composite image. Can be used for. Then, as shown in FIG. 16D, a composite image as an image for output is generated and displayed on the display unit 35. In this way, for example, even while a plurality of captured images are being acquired, the second user U2 can swipe the display screen Sc0 to display the image displayed on the display screen Sc0 of the display unit 25. You can scroll the range of.

<1-4. Image processing system operation flow>
<1-4-1. Multiple imaging operations>
FIG. 17 is a flow chart showing an example of an operation flow when performing a plurality of imaging operations (also referred to as imaging operations) for an image-imaging object Ob0 in the image processing system 1. This operation flow is started, for example, in response to the input of a signal by the first user U1 via the input unit 2in of the first image processing device 20. In the example of FIG. 17, an operation flow focusing on a portion of a plurality of times of imaging in which two imaging operations are performed for convenience is shown. FIG. 18 is a flow chart showing an example of an operation flow in one imaging operation in the first image processing apparatus 20.

First, in step S1 of FIG. 17, for example, in the first image processing device 20, an imaging operation is performed in the first image processing device 20 in response to a signal input by the first user U1 via the input unit 2in. .. In the imaging operation, the processes of steps S101 to S105 in FIG. 18 are performed in order. In step S101, the imaging unit 21 acquires the captured image and the distance measuring unit 241 acquires the distance image (distance distribution information). In step S102, the resolution calculation unit 211 calculates the imaging resolution of the image portion that captures the imaging object Ob0 based on the distance distribution information and the characteristic information of the imaging unit 21. In step S103, the position measurement unit 221 measures the viewpoint position at the time of imaging and acquires the viewpoint position information. In step S104, the direction measurement unit 231 measures the line-of-sight direction at the time of imaging, and the line-of-sight direction information is acquired. The combination of the process of step S101 and the process of step S102, the process of step S103, and the process of step S104 may be executed in any order or may be executed in parallel. In step S105, the image captured in step S101 by the communication unit 27 and attached information such as distance image information, viewpoint position information, and line-of-sight direction information acquired in steps S102 to S104 (also referred to as imaging accessory information). ) Is transmitted to the second device 3.

Next, in step S2 of FIG. 17, the captured image and the imaging accessory information are transmitted from the first image processing device 20 to the second image processing device 30 via the communication line 4 and the like.

In step S3, in the second image processing device 30, the communication unit 37 of the acquisition unit 3ac receives the captured image and the imaging accessory information from the first image processing device 20, and the captured image and the imaging attachment information are associated with each other. It is stored in the storage medium 330 in this state.

In step S4, in the second image processing device 30, the captured image received and stored in step S3 is visually output on the display unit 35.

Subsequently, in steps S5 to S8, the same processing as in steps S1 to S4 is performed. As a result, for example, in the second image processing apparatus 30, a plurality of captured images obtained by imaging an imaged object under conditions where the combination of the viewpoint position and the line-of-sight direction are different from each other by the acquisition unit 3ac, and each captured image The viewpoint position information and the line-of-sight direction information related to the time of imaging are acquired.

<1-4-2. Estimating the location of the object to be imaged>
FIG. 19 shows an example of an operation flow when the estimation unit 311 of the second image processing apparatus 30 estimates the existence position of the image pickup target Ob0 captured by M images (M is a natural number of 2 or more). It is a flow chart which shows. This operation flow may be started after the operation flow related to the plurality of imaging operations shown in FIGS. 17 and 18 is completed, or may be executed in parallel with the operation flow related to the plurality of imaging operations. May be done. Here, for example, by performing the processes of steps S11 to S19, the existence position of the imaged object is estimated based on the viewpoint position information and the line-of-sight direction information related to each of the plurality of captured images.

In step S11, the virtual target space Sp0 including the imaging target Ob0 is divided into a large number of voxels Bx0 in a grid pattern.

In step S12, the numerical value i that defines the order of the captured images is set to 1. This numerical value i means that the i-th captured image is the target of processing.

In step S13, the viewpoint position pi and the line-of-sight direction vi are read from the storage medium 330 for the i-th captured image. For example, when i = 1, the viewpoint position p1 and the line-of-sight direction v1 are read from the storage medium 330 for the i-th captured image.

In step S14, the representative line-of-sight direction Vj is obtained for the line-of-sight direction vi for the i-th captured image.

In step S15, the representative line-of-sight direction Vj obtained in step S14 is valid for each voxel Bx0 that the line of sight passed through in the target space Sp0 for the i-th captured image. The line of sight at the time of shooting is obtained from, for example, the viewpoint position pi and the line of sight direction vi.

In step S16, it is determined whether or not the numerical value i has reached the numerical value M. In other words, it is determined whether or not the i-th captured image to be processed reaches the M-th captured image. Here, if the numerical value i has not reached the numerical value M, the numerical value i is increased by one in step S17, and the process returns to step S13. If the numerical value i has reached the numerical value M, the process proceeds to step S18.

In step S18, for each voxel Bx0 in the target space Sp0, a number of valid representative line-of-sight directions is recognized.

In step S19, it is estimated that one or more voxels Bx0 in which the number of representative line-of-sight directions equal to or greater than the threshold value is valid are the existing positions of the imaging target Ob0.

<1-4-3. Enlarging and reducing the displayed image>
FIG. 20 shows an example of an operation flow when enlarging and reducing an image displayed on the display unit 35 based on a plurality of captured images stored in the storage medium 330 in the second image processing apparatus 30. It is a flow chart. This operation flow can be executed, for example, after the operation flow for estimating the existence position of the image pickup target Ob0 captured in the M images captured in FIG. 19 is completed.

In step S21, the area setting unit 315 sets the display target area for the image pickup target Ob0 in response to the signal input by the input unit 3in according to the operation of the second user U2. The display target area is an area for which an image is to be visually output to the display unit 35.

In step S22, the resolution setting unit 314 displays in response to the signal input by the input unit 3in according to the operation of the second user U2 related to the enlargement or reduction of the image such as rotation of the mouse wheel, pinch out or pinch in. The resolution (Ra) is set. At this time, the size of the display target area is adjusted.

In step S23, the detection unit 312 compares the display resolution (Ra) set in step S22 with the imaging resolution (Rb) of the captured image in which the display target area is captured.

In step S24, the detection unit 312 determines whether or not the display resolution (Ra) is higher than the imaging resolution (Rb). Here, if the display resolution (Ra) is higher than the imaging resolution (Rb), the process proceeds to step S25, and if the display resolution (Ra) is not higher than the imaging resolution (Rb), the process proceeds to step S30. Here, if the display resolution (Ra) is higher than the imaging resolution (Rb), it means that the captured image corresponding to the display resolution is insufficient.

In step S25, the information generation unit 316 generates imaging request information requesting the imaging of the insufficiently captured image, and the output unit 3ou outputs the imaging request information. For example, the communication unit 37 transmits the image pickup request information to the first image processing device 20 via the communication line 4 or the like.

In step S26, the image generation unit 313 acquires an image of the imaging resolution (Rb) from the image DB3Id instead of the image of the display resolution (Ra), and the enlarged image corresponding to the display resolution set in step S22 is obtained. It is forcibly generated. Then, this enlarged image is tentatively displayed on the display unit 35. At this time, when the enlarged image is generated, a temporary enlarged image may be generated for the portion of the insufficiently captured image by interpolation processing or the like.

In step S27, the display unit 35 is shown to indicate that the temporary enlarged image portion is waiting for image replenishment.

In step S28, the acquisition unit 3ac acquires the captured image and the imaging accessory information transmitted from the first image processing device 20. This captured image is an image that meets the imaging request.

In step S29, the control unit 300 stores the captured image acquired in step S28 in the image DB3Id of the storage medium 330 instead of the captured image having an imaging resolution (Rb) lower than the display resolution (Ra). When the process of step S29 is completed, the process returns to step S23.

In step S30, the display target area and display set in steps S21 and S22 are displayed from the captured image having the image resolution (Rb) determined by the image generation unit 313 to be higher than the display resolution (Ra) in step S24. An image corresponding to the resolution is generated and displayed on the display unit 35.

<1-4-4. Imaging to supplement the insufficient captured image>
FIG. 21 is a flow chart showing an example of an operation flow when an image processing system 1 captures an insufficiently captured image related to an imaged object Ob0. This operation flow is started, for example, in response to the detection of a shortage of captured images with respect to the generation of images for display in the second image processing apparatus 30. In the example of FIG. 21, an operation flow in which one insufficiently captured image is captured is shown.

In step S31 of FIG. 21, in the second image processing apparatus 30, similarly to step S25 of FIG. 20, the information generation unit 316 generates imaging request information requesting imaging of the insufficiently captured image.

In step S32, in the second image processing device 30, the communication unit 37 transmits the imaging request information generated in step S31 via the communication line 4 and the like, and receives the image processing request information to the first image processing device 20.

In step S33, in the second image processing apparatus 30, the image generation unit 313 generates a temporary image including a forced enlarged image, a blank part, or a blackened part in place of the missing captured image. Then, a temporary image is displayed by the display unit 35.

In step S34, the second image processing device 30 waits for the display unit 35 to replenish the temporary image portion in the same manner as in step S27 of FIG. 20. Display (also referred to as image replenishment standby display) is displayed.

In step S35, in the first image processing device 20, the instruction generation unit 212 generates image pickup instruction information indicating an image pickup instruction based on the image pickup request information received by the communication unit 27. The imaging instruction information includes, for example, a specific viewpoint position and line-of-sight direction with respect to the imaging object Ob0. The imaging instruction information may include guide information including, for example, a map that guides the first user U1 to a specific viewpoint position and line-of-sight direction for capturing a missing captured image.

Here, if the imaging request information indicates a region on the imaging target object Ob0 (also referred to as an imaging target region) and a desired display resolution, for example, the instruction generation unit 212 captures the imaging target region. A region (also referred to as an imageable region) Sa in which a viewpoint position suitable for the above can be taken is calculated. Further, for example, the instruction generation unit 212 calculates a region (imaging possible region) Sb in which a viewpoint position suitable for imaging with an imaging resolution for an imaging target Ob0 having a desired display resolution or higher can be obtained. Then, for example, the instruction generation unit 212 sets a region (imaging region) Sc where the imageable region Sa and the imageable region Sb overlap. At this time, the image pickup instruction information may include, for example, information for guiding the first user U1 to move to the image pickup possible area Sc.

In step S36, in the first image processing device 20, the display unit 25 of the output unit 2ou visually outputs the imaging instruction information generated in step S35. Here, for example, information for guiding the first user U1 to a specific viewpoint position and line-of-sight direction for capturing the missing captured image is displayed.

In step S37, in the first image processing device 20, similarly to steps S1 and S5 in FIG. 17, in response to the input of the signal by the first user U1 via the input unit 2in, the first image processing device 20 An imaging operation is performed.

In step S38, as in step S2 of FIG. 17, the captured image and the imaging accessory information are transmitted from the first image processing device 20 to the second image processing device 30 via the communication line 4 and the like.

In step S39, similarly to step S3 of FIG. 17, in the second image processing device 30, the communication unit 37 of the acquisition unit 3ac receives the captured image from the first image processing device 20 and the imaging accessory information. The captured image and the imaging accessory information are stored in the storage medium 330 in a state of being associated with each other.

In step S40, similarly to step S4 of FIG. 17, in the second image processing apparatus 30, the image portion based on the captured image received and stored in step S39 is the portion of the temporary image displayed in step S33. It is visually output on the display unit 35 so as to be replaced. At this time, the imaging replenishment standby display displayed in step S34 is also deleted.

<1-4-5. Scrolling the displayed image>
22 to 24 show an example of an operation flow when scrolling the images displayed on the display unit 35 based on the plurality of captured images stored in the storage medium 330 in the second image processing device 30. It is a flow chart which shows. Here, the imaged object Ob0 as shown in FIG. 11 is imaged, and the captured images as shown in FIGS. 12 (a) to 12 (d) are acquired, and FIGS. 13 (a) to 13 (a). An example of scrolling an image on the display screen Sc0 as shown in 13 (c) will be described. This operation flow is started in response to a change in the display target area by, for example, a swipe operation on the display screen Sc0.

In step S41 of FIG. 22, the detection unit 312 recognizes an element (also referred to as element A) of the image pickup target Ob0 that was captured in the image displayed on the display screen Sc0 before the display target area was changed. .. Here, it is assumed that only the element A is captured in the image displayed on the display screen Sc0 before the display target area is changed. Here, as the element A, for example, the first surface SfA of FIG. 11 is adopted.

In step S42, the detection unit 312 searches for the image portion in which the portion of the imaging target Ob0 corresponding to the display target region changed by translation by translation operation such as swiping is captured for the plurality of captured images in the image DB3Id. Will be done. As described above, the display target area is a coordinate indicating a spatial position in the target space Sp0 based on, for example, viewpoint position information, line-of-sight direction information, characteristic information of the imaging unit 21, and the existence position of the imaging object Ob0. Can be managed by. Therefore, for example, for a plurality of captured images, the imaging target Ob0 corresponding to the display target region is captured based on the viewpoint position information, the line-of-sight direction information, the existence position information, and the characteristic information of the imaging unit 21. The image part can be searched. Here, for example, if the distance distribution information related to the captured image is added, the shape of the portion of the imaged object Ob0 captured in the captured image can be recognized, and the spatial position of the display target area in the target space Sp0 can be recognized. The accuracy of the coordinates indicating the above can be improved. As a result, the image portion in which the image pickup target Ob0 corresponding to the display target region is captured can be searched with high accuracy.

In step S43, the detection unit 312 determines in step S42 whether or not the image portion corresponding to the display target area changed by translation is detected. Here, if the image portion is not detected, the process proceeds to step S44, and if the image portion is detected, the process proceeds to step S45.

In step S44, as in step S25 of FIG. 20, the information generation unit 316 generates imaging request information requesting the imaging of the insufficiently captured image, and the output unit 3ou outputs this imaging request information. Specifically, for example, the communication unit 37 transmits the image pickup request information to the first image processing device 20 via the communication line 4 or the like. Here, for example, when proceeding from step S43, the insufficient captured image may be an captured image corresponding to the display target area after the change due to translation. For example, when proceeding from step S55 of FIG. 23, which will be described later, the insufficient captured image may be an captured image including an image portion in which the element E is captured from the front. For example, when proceeding from step S63 of FIG. 24 described later, the insufficiently captured image may be a boundary-containing image described later. For example, when proceeding from step S65 of FIG. 24, which will be described later, the insufficient captured image may be an captured image including an image portion in which the element D is captured from the front. Here, after the processing of step S44 is performed, the process returns to step S42. At this time, in the image processing system 1, for example, as shown in FIG. 21, the operation flow for capturing the insufficiently captured image related to the imaged object Ob0 is executed.

In step S45, the detection unit 312 analyzes the image portion (also referred to as the detected image portion) detected in step S42. In this analysis, for example, in the detected image portion, a portion that captures the imaging target Ob0 (also referred to as a target portion) and a portion that captures a region other than the imaging target Ob0 (background 600, etc.) (also referred to as a non-target portion). And are detected. Here, for example, the image pickup target Ob0 is captured based on the viewpoint position information, the line-of-sight direction information, the existence position information, and the characteristic information of the imaging unit 21 for the detected image portion to be analyzed. The image part can be recognized. As a result, the detected image portion is divided into a portion that captures the image pickup target Ob0 and a portion that captures the background 600. At this time, for example, if the distance distribution information is used, the detected image portion is more accurately divided into a portion that captures the image pickup target Ob0 and a portion that captures the background 600. In this case, for example, it is utilized that the distance from the viewpoint position to the imaging object Ob0 is clearly shorter than the distance from the viewpoint position to the background 600.

In step S46, the detection unit 312 determines whether or not the detected image portion has a non-target portion corresponding to a portion other than the imaging target Ob0 (background 600, etc.) based on the analysis result of step S45. Will be done. Here, if the non-target portion does not exist in the detected image portion, the process proceeds to step S51 in FIG. 23, and if the non-target portion exists in the detected image portion, the process proceeds to step S61 in FIG. 24.

In step S51 of FIG. 23, the detection unit 312 further analyzes the detected image portion detected in step S42. In this analysis, for example, an image portion in which the element A recognized in step S41 is captured and an image portion in which the element E other than the element A is captured are detected. As the element E, for example, the second surface SfB adjacent to the first surface SfA in FIG. 11 is adopted. Here, for example, a corner portion forming a boundary between the element A and the element E based on at least one of the existence position information, the viewpoint position, the line-of-sight direction, and the angle of view related to the image pickup object Ob0, and the distance distribution information. Can be detected. Thereby, the element A and the element E can be distinguished.

In step S52, the detection unit 312 determines whether or not the detected image portion includes an image portion in which the element E other than the element A is captured, based on the analysis result of step S51. Here, if the image portion in which the element E is captured does not exist in the detected image portion, the process proceeds to step S53, and if the image portion in which the element E is captured exists in the detected image portion, the process proceeds to step S54. ..

In step S53, the display unit 35 updates the image (also referred to as a display image) displayed on the display screen Sc0 to the detected image portion detected in step S42. As a result, for example, the image portion of the first surface SfA related to the display target area after translation can be displayed on the display screen Sc0.

In step S54, the detection unit 312 searches for an image portion in which the element E of the imaged object Ob0 is captured from the front with respect to the plurality of captured images in the image DB3Id. Here, whether or not the element E is captured from the front can be determined by, for example, whether or not the line-of-sight direction is substantially perpendicular to the element E. At this time, for example, the image portion related to the element E, which is in contact with the corner portion which is the boundary with the element A and has an imaging resolution equal to or higher than the imaging resolution of the image portion in which the element A is captured in the detected image portion, can be searched. ..

In step S55, the detection unit 312 determines, based on the search result of step S54, whether or not an image portion in which the element E of the imaged object Ob0 is captured from the front is detected. Here, if the image portion in which the element E is captured from the front is not detected, the process proceeds to step S44 in FIG. If the image portion in which the element E is captured from the front is detected, the process proceeds to step S56.

In step S56, the image generation unit 313 describes the image portion (corresponding to the first image portion) related to the element A detected in step S51 and the element E detected in step S54 with respect to the detected image portion detected in step S42. The image portion (corresponding to the second image portion) according to the above is combined. As a result, a composite image is generated. At this time, for example, the first image portion and the second image portion are combined so as to be arranged so as to sandwich the portion corresponding to the boundary between the element A and the element E, so that the combined image can be calculated with a relatively small number of operations. Can be generated. Here, for example, the outer shape of the portion of the second image portion separated from the first image portion may be appropriately adjusted so that the outer shape of the composite image matches the display screen Sc0.

In step S57, the display unit 35 updates the image (also referred to as a display image) displayed on the display screen Sc0 with the composite image generated in step S56.

In step S61, the detection unit 312 divides the detected image portion detected in step S42 into an image portion B that captures the image-imaging object Ob0 and an image portion C that captures an image other than the image-imaging object Ob0. As a result, the image portion B as the first image portion can be detected. As the image portion C, for example, an image portion that captures the background 600 of FIG. 11 is adopted.

In step S62, the detection unit 312 has an image portion in which the element B of the imaging target object Ob0 corresponding to the image portion B is captured for a plurality of captured images in the image DB3Id, and the remaining portion is the imaging target object Ob0. The boundary-containing image that captures the element D is searched. The boundary-containing image is an image including the boundary between the element B and the element D. Here, as the element B, for example, the first surface SfA of FIG. 11 is adopted. As the element D, for example, the second surface SfB adjacent to the first surface SfA in FIG. 11 is adopted. Here, for example, the boundary between the element B and the element D can be detected based on at least one of the existence position information related to the image pickup object Ob0, the viewpoint position, the line-of-sight direction, and the angle of view, and the distance distribution information. .. Thereby, it can be recognized whether or not each captured image is a boundary-containing image.

In step S63, the detection unit 312 determines whether or not the boundary-containing image has been detected. Here, if the boundary-containing image is detected, the process proceeds to step S64, and if the boundary-containing image is not detected, the process proceeds to step S44 of FIG.

In step S64, the detection unit 312 searches for an image portion in which the element D of the imaged object Ob0 is captured from the front with respect to the plurality of captured images in the image DB3Id. Here, whether or not the element D is captured from the front can be determined by, for example, whether or not the line-of-sight direction is substantially perpendicular to the element D. At this time, for example, an image portion related to element D that is in contact with a corner portion that is a boundary with element B and has an imaging resolution equal to or higher than the imaging resolution of the image portion that captures element B in the detected image portion can be searched. ..

In step S65, the detection unit 312 determines, based on the search result of step S64, whether or not the image portion in which the element D of the image pickup target Ob0 is captured from the front is detected. Here, if the image portion in which the element D is captured from the front is not detected, the process proceeds to step S44 in FIG. If the image portion in which the element D is captured from the front is detected, the process proceeds to step S66.

In step S66, the image generation unit 313 recognizes the image portion B (corresponding to the first image portion) in the division of step S61 and the image portion related to the element D detected in step S64 (corresponding to the second image portion). ) And are synthesized. As a result, a composite image is generated. At this time, for example, the first image portion and the second image portion are combined so as to be arranged so as to sandwich the portion corresponding to the boundary between the element B and the element D, so that the combined image is generated with a relatively small number of operations. Can be done. Here, for example, the outer shape of the portion of the second image portion separated from the first image portion may be appropriately adjusted so that the outer shape of the composite image matches the display screen Sc0.

In step S67, the display unit 35 updates the image (display image) displayed on the display screen Sc0 with the composite image generated in step S66.

<1-4-6. Switching image generation mode>
FIG. 25 is a flow chart showing an example of an operation flow in which the image generation mode is switched in a state where a plurality of captured images are sequentially acquired by the acquisition unit 3ac in the second image processing apparatus 30.

In step S71, the area setting unit 315 accepts the change of the display target area according to the signal input in response to the operation of the second user U2.

In step S72, the area setting unit 315 determines whether or not the display target area has been changed. Here, if there is a change in the display target area, the process proceeds to step S73. On the other hand, if there is no change in the display target area, the switching unit 317 sets the image generation mode to the first image generation mode described above, and the process proceeds to step S77.

In step S73, the area setting unit 315 confirms the necessity of additional imaging in order to generate an image for display. Here, for example, as shown in FIG. 15, if the changed display target area is included in the already imaged area Ai1 of the imaging target object Ob0, no additional imaging is necessary. On the other hand, for example, as shown in FIG. 16, if the changed display target area has a portion outside the region Ai1 of the image pickup target Ob0, additional imaging is required.

In step S74, the area setting unit 315 proceeds to step S75 according to the confirmation result of step S73 if additional imaging is not required. At this time, the switching unit 317 sets the image generation mode to the second image generation mode described above. If additional imaging is required, the process proceeds to step S76. At this time, the switching unit 317 sets the image generation mode to the third image generation mode described above.

In step S75, as shown in FIG. 15, the image generation unit 313 detects the first image portion and the second image portion of the plurality of captured images already acquired by the acquisition unit 3ac by the detection unit 312. And are synthesized. As a result, a composite image as an image for output is generated.

In step S76, as shown in FIG. 16, the image generation unit 313 immediately acquires the first image portion detected by the detection unit 312 for the plurality of captured images already acquired by the acquisition unit 3ac. The second image portion detected by the detection unit 312 is combined with the captured image. As a result, a composite image as an image for output is generated.

In step S77, the acquisition unit 3ac confirms the reception of the captured image from the first device 2.

In step S78, the acquisition unit 3ac determines whether or not the captured image has been received from the first device 2. Here, if the captured image from the first device 2 is not received, this operation flow ends. On the other hand, if the captured image from the first device 2 is received, the process proceeds to step S79.

In step S79, the image generation unit 313 generates an output image from the image captured most recently received by the acquisition unit 3ac.

As described above, in the first image generation mode, for example, each time the image generation unit 313 acquires an image captured by the acquisition unit 3ac, the captured image is used to prepare an image for output. As a result, a real-time display that is displayed each time a captured image is obtained can be realized.

<1-5. Summary of the first embodiment>
As described above, in the image processing system 1 and the second image processing apparatus 30 according to the first embodiment, for example, a plurality of captured images and viewpoint position information and line-of-sight direction information at the time of imaging for each captured image are acquired. Will be done. The existence position of the imaged object Ob0 is estimated based on the viewpoint position information and the line-of-sight direction information related to each captured image. For a plurality of captured images, the first region in the imaged object Ob0 is determined based on the viewpoint position information and the line-of-sight direction information, the estimated existing position information of the imaged object Ob0, and the characteristic information of the imaging unit 21. The first image portion captured in the first line-of-sight direction from the first viewpoint position and the second region different from the first region in the imaged object Ob0 are changed from the second viewpoint position different from the first viewpoint position to the first line-of-sight direction. Is detected as a second image portion captured in a different second line-of-sight direction. Then, a composite image can be generated by synthesizing the detected first image portion and the second image portion.

Here, for example, the first image portion and the second image portion obtained by imaging the imaging object Ob0 from different directions are synthesized so as to be arranged side by side. Thereby, for example, a composite image which is a pseudo three-dimensional image obtained by capturing the image pickup object Ob0 from a plurality of surrounding angles can be easily acquired with a relatively small amount of calculation. Therefore, for example, the amount of calculation in image processing for obtaining an image of an imaged object viewed from various angles around it can be reduced. Therefore, for example, a processor that performs calculations can be effectively used. Further, for example, the time required to generate the composite image in the processor is shortened, and it is difficult to interfere with other processing using the processor. As a result, the usability of the image processing system 1 and the second image processing device 30 is improved.

Further, for example, if the first user U1 performs a series of imaging of the imaged object Ob0 a plurality of times using the first image processing device 20 and accumulates a plurality of captured images, the second user U2 can be used. , The image displayed on the display unit 35 can be arbitrarily scrolled. Therefore, for example, the second user U2 can confirm the image pickup target Ob0 from various angles around it at his / her own pace without forcing the first user U1 to perform a great deal of work. Therefore, for example, the work efficiency can be improved for the first user U1 and the second user U2. Further, here, by generating a composite image which is a pseudo three-dimensional image, it is possible to easily obtain an image of the imaged object Ob0 viewed from various angles around it with a relatively small amount of calculation.

<2. Other embodiments>
The present disclosure is not limited to the above-described first embodiment, and various changes, improvements, and the like can be made without departing from the gist of the present disclosure.

<2-1. Second Embodiment>
In the first embodiment, the first image processing device 20 takes an image of the imaged object Ob0, and the second image processing device 30 generates and displays a composite image using a plurality of captured images. On the other hand, for example, in the first image processing device 20, the image pickup target Ob0 may be imaged, and further, a composite image using a plurality of captured images may be generated and displayed. If such a configuration is adopted, for example, in one device, it is easy to perform from imaging for acquiring a plurality of captured images to image processing for obtaining images of an imaged object viewed from various angles around it. can do.

Specifically, as shown in FIG. 26, the first image processing device 20A according to the second embodiment has, for example, the first image processing device 20A according to the first embodiment in addition to the functions of the first image processing device 20. It may have the same function as the second image processing apparatus 30 according to the first embodiment. Here, the functions similar to those of the second image processing apparatus 30 according to the first embodiment include, for example, an estimation unit 311, a detection unit 312, an image generation unit 313, a resolution setting unit 314, an area setting unit 315, and information generation. Functional configurations such as unit 316 and switching unit 317 are included. In the first image processing apparatus 20A according to the second embodiment, for example, the output unit 2ou plays the same role as the output unit 3ou, the input unit 2in plays the same role as the input unit 3in, and the acquisition unit 2ac plays the same role as the acquisition unit 3in. It plays the same role as 3ac. Further, for example, based on the imaging request information for requesting the imaging of the insufficiently captured image generated by the information generation unit 316, the instruction generation unit 212 indicates the imaging instruction information indicating the instruction according to the request based on the imaging request information or the like. Is generated. For example, the visible output of this imaging instruction information can be performed by the display unit 25 of the output unit 2ou.

<3. Others>
In each of the above embodiments, for example, the first image portion and the second image portion used for generating the composite image do not capture the first region and the second region adjacent to each other of the imaging object Ob0. May be good. For example, in the image pickup target Ob0, the first region and the second region may be located so as to sandwich the corner portion at the boundary between the first surface SfA and the second surface SfB. Here, as the corner portion, for example, a curved R surface or a chamfered C surface portion can be adopted. If such a configuration is adopted, the state of scrolling the display images shown in FIGS. 13 (a) to 13 (c) can be seen in the display images shown in FIGS. 27 (a) to 27 (c). Can be seen as scrolling. In the composite image of FIGS. 27 (b) and 27 (c), the corner portion corresponds to the space between the first image portion Ipa related to the first surface SfA and the first image portion Ipb related to the second surface SfB. There is a blank part to do. Further, for example, a composite image is generated so that the first image portion Ipa related to the first surface SfA and the first image portion Ipb related to the second surface SfB are arranged without a gap without providing the blank portion. May be done.

In each of the above embodiments, for example, the composite image may be temporarily generated at a timing necessary for display, or the composite image once generated may be stored in the storage media 230, 330, or the like.

In each of the above embodiments, for example, a composite image may be generated by synthesizing the first image portion and the second image portion obtained by capturing each surface of the imaging target Ob0 from the front, or the imaging target Ob0 may be generated. The first image portion and the second image portion obtained by capturing each surface of the above from different directions may be combined.

In each of the above embodiments, for example, the image of the surface captured by the image portion of the central portion of the display screen Sc0 is selectively displayed on the display screen without synthesizing two or more image portions of different surfaces of the imaging object Ob0. It may be displayed on Sc0. At this time, for example, by scrolling the image on the display screen Sc0, the second surface SfB is captured on the display screen Sc0 at the timing when the end portion of the image portion capturing the first surface SfA passes through the center portion of the display screen Sc0. It is conceivable that the image portion is started to be displayed.

In each of the above embodiments, for example, the first user U1 is not limited to the field worker, and may be, for example, a general person (also referred to as an imager) who images a three-dimensional image-imaging object.

In each of the above embodiments, for example, the second user U2 is not limited to the person in charge of the device, but is, for example, a person (also referred to as an observer) who observes an image of a three-dimensional imaged object viewed from various angles around it. It may be general.

In each of the above embodiments, for example, the object to be imaged Ob0 may be, for example, a sarcophagus of an old burial mound that cannot be easily approached, or a solid that can be observed from other surroundings such as clothing, a bag, and hair. It may be a general one having a typical shape.

In each of the above embodiments, for example, the second image processing device 30 may be in a form in which the main body unit and the display unit 35 are divided. In this case, for example, a mode in which the main body unit and the display unit 35 are communicably connected can be adopted. Further, for example, the first image processing device 20A may have a form in which the main body unit and the display unit 25 are divided. In this case, for example, a mode in which the main body unit and the display unit 25 are communicably connected can be adopted.

In each of the above embodiments, for example, the image DBs 2Id and 3Id do not need to be constructed in the storage medium 230 of the first device 2 and the storage medium 330 of the second device 3, and are various servers or various types in the concept of cloud computing. It may be constructed on a storage medium.

Needless to say, all or a part of each of the above-described embodiments and various modifications can be combined as appropriate and within a consistent range.

1 Image processing system 2 1st device 2Pr, 3Pr Control program 2ac, 3ac Acquisition unit 2in, 3in Input unit 2ou, 3ou Output unit 3 2nd device 3re Receiver 4 Communication line 20, 20A 1st image processing device 21 Imaging unit 22 Position sensor 23 Direction sensor 24 Distance measurement sensor 25,35 Display unit 27,37 Communication unit 30 Second image processing device 200, 300 Control unit 210, 310 CPU
211 Resolution calculation unit 212 Instruction generation unit 221 Position measurement unit 230, 330 Storage medium 231 Direction measurement unit 241 Distance measurement unit 311 Estimating unit 312 Detection unit 313 Image generation unit 314 Resolution setting unit 315 Area setting unit 316 Information generation unit 317 Switching unit Ai1, Ar0, Ar1 area Ar2 Display target area (already displayed area)
Ar21 Remaining Ar3 Display target area Ar22 Non-display area 2Id, 3Id Image database (image DB)
Im11, Im12, Im13, Im14, It0, It1, It2 Captured image Ip0, Ip1, Ip2, Ip21, Ip22 Image part Ip21 image part (4th image part)
Ip22 image part (third image part)
Ob0 Image target U1 1st user U2 2nd user

Claims (5)

A plurality of captured images that capture an object under conditions where the combination of the viewpoint position and the line-of-sight direction are different from each other, and the viewpoint position information indicating the viewpoint position at the time of imaging and the line-of-sight direction indicating the line-of-sight direction for each of the plurality of captured images. Information, acquisition department to acquire,
An estimation unit that acquires the existence position information indicating the existence position by estimating the existence position of the object based on the viewpoint position information and the line-of-sight direction information related to each of the plurality of captured images.
With respect to the plurality of captured images, the first region in the object is determined based on the viewpoint position information, the line-of-sight direction information, the existing position information, and the characteristic information of the imaging unit used at each imaging. The first image portion captured in the first line-of-sight direction from the first viewpoint position and the second region of the object different from the first line-of-sight position are viewed from the second viewpoint position different from the first viewpoint position. A detection unit that detects a second image portion captured in a second line-of-sight direction different from the direction, and a detection unit.
An image generation unit that generates a composite image by synthesizing the first image portion and the second image portion.
An output unit that outputs the composite image and at least one of a storage medium that stores the composite image.
Equipped with an area setting unit
The output unit includes a display unit that visually outputs the composite image.
The area setting unit changes the display target area captured in the composite image that is visually output by the display unit of the object in response to a signal input in response to the user's operation. ,
The area setting unit is one of the already displayed areas of the object captured in the first composite image visually output by the display unit in response to a signal input in response to the user's operation. The changed display target area including the remaining portion excluding the portion and the non-display area of the object that is not captured in the first composite image that is visually output by the display unit is set.
In response to the setting of the display target area after the change by the area setting unit, the detection unit receives the viewpoint position information, the line-of-sight direction information, the existence position information, and the said existence position information for the plurality of captured images. Based on the characteristic information of the imaging unit, the third image portion that captures the non-display area is detected.
The image generation unit generates a second composite image by synthesizing the fourth image portion that captures the displayed region in the first composite image and the third image portion.
The display unit is an image processing device that visually outputs the second composite image. The image processing apparatus according to claim 1.
The output unit outputs information relating to the shortage of the image portion capturing the non-display area in response to the detection by the detection unit of the shortage of the image portion capturing the non-display area in the plurality of captured images. , Image processing equipment. A plurality of captured images that capture an object under conditions where the combination of the viewpoint position and the line-of-sight direction are different from each other, and the viewpoint position information indicating the viewpoint position at the time of imaging and the line-of-sight direction indicating the line-of-sight direction for each of the plurality of captured images. Information, acquisition department to acquire,
An estimation unit that acquires the existence position information indicating the existence position by estimating the existence position of the object based on the viewpoint position information and the line-of-sight direction information related to each of the plurality of captured images.
With respect to the plurality of captured images, the first region in the object is determined based on the viewpoint position information, the line-of-sight direction information, the existing position information, and the characteristic information of the imaging unit used at each imaging. The first image portion captured in the first line-of-sight direction from the first viewpoint position and the second region of the object different from the first line-of-sight position are viewed from the second viewpoint position different from the first viewpoint position. A detection unit that detects a second image portion captured in a second line-of-sight direction different from the direction, and a detection unit.
An image generation unit that generates a composite image by synthesizing the first image portion and the second image portion.
Resolution calculation unit and
Output section and
Equipped with a resolution setting unit
The acquisition unit acquires distance distribution information indicating the distribution of the distance from the viewpoint position to the surface of the object at the time of imaging for each of the plurality of captured images.
The resolution calculation unit calculates the imaging resolution of the image portion that captures the object in each of the captured images based on the distance distribution information and the characteristic information.
The output unit visually outputs the composite image.
The resolution setting unit sets the display resolution of the image portion that captures the object that is visually output by the output unit in response to the signal input in response to the user's operation.
The detection unit is an image processing device that detects the first image portion and the second image portion whose imaging resolution is equal to or higher than the display resolution in the plurality of captured images. The image processing apparatus according to claim 3.
The resolution setting unit is an enlarged display image portion of the composite image that is visually output to the output unit in response to a signal input in response to a user's operation and is enlarged and displayed by the output unit. Along with setting the range, the display resolution in the enlarged display image portion is set.
In the output unit, the detection unit has detected that the plurality of captured images lack an image portion for enlarged display in which the captured image resolution related to the range of the enlarged display image portion is equal to or higher than the display resolution. An image processing device that responds and outputs information relating to a shortage of the image portion for enlarged display. A plurality of captured images that capture an object under conditions where the combination of the viewpoint position and the line-of-sight direction are different from each other, and the viewpoint position information indicating the viewpoint position at the time of imaging and the line-of-sight direction indicating the line-of-sight direction for each of the plurality of captured images. Information, acquisition department to acquire,
An estimation unit that acquires the existence position information indicating the existence position by estimating the existence position of the object based on the viewpoint position information and the line-of-sight direction information related to each of the plurality of captured images.
With respect to the plurality of captured images, the first region in the object is determined based on the viewpoint position information, the line-of-sight direction information, the existing position information, and the characteristic information of the imaging unit used at each imaging. The first image portion captured in the first line-of-sight direction from the first viewpoint position and the second region of the object different from the first line-of-sight position are viewed from the second viewpoint position different from the first viewpoint position. A detection unit that detects a second image portion captured in a second line-of-sight direction different from the direction, and a detection unit.
An image generation unit that generates a composite image by synthesizing the first image portion and the second image portion.
An image generation mode that generates an image for output in the image generation unit in response to a signal input in response to a user's operation in a state where two or more captured images are sequentially acquired by the acquisition unit. Is provided with a switching unit for selectively switching between the first image generation mode, the second image generation mode, and the third image generation mode.
The first image generation mode includes a mode in which an image for output is prepared by using the captured image each time the captured image is acquired by the acquisition unit.
The second image generation mode is the combination of the first image portion and the second image portion detected by the detection unit with respect to the plurality of captured images already acquired by the acquisition unit. Includes a mode to generate the composite image as an output image.
The third image generation mode responds to the acquisition of the first image portion detected by the detection unit and the captured image by the acquisition unit for the plurality of captured images already acquired by the acquisition unit. An image processing apparatus including a mode of generating the composite image as an image for output by synthesizing the second image portion detected by the detection unit from the captured image.

Images