JP7581638B2 - Information processing device and information processing method - Google Patents

Description

The present invention relates to an information processing device and an information processing method.

Patent document 1 describes a distance measuring device that can stably and accurately measure the distance to an object.

Patent document 2 describes an imaging device that performs image processing to reduce the effect of inclusion of a finger or other object in the image.

The present invention aims to provide an information processing device and information processing method that allows easy confirmation that the desired three-dimensional information has been acquired.

The information processing device of the present invention outputs three-dimensional information determined based on received light and comprises an imaging unit that captures two-dimensional image information , a projection unit that projects light onto an object, a light receiving unit that has an angle of view that at least partially coincides with the angle of view of the imaging unit and receives the light reflected from the object, a three-dimensional information determination unit that determines the three-dimensional information based on the output of the light receiving unit, and an output unit that outputs the two-dimensional image information before determining the three -dimensional information.

The present invention provides an information processing device and information processing method that can easily confirm that the desired three-dimensional information has been acquired.

1 is a diagram showing an example of the appearance of an imaging device according to an embodiment of the present invention. 2 is a diagram for explaining a configuration of an imaging device in the embodiment. FIG. 2A to 2C are diagrams for explaining a usage state of the imaging device in the embodiment. FIG. 2 is a diagram showing an example of a configuration of a processing block of a processing circuit in the embodiment. 10 is a flow chart showing an example of the operation of a processing circuit of the imaging device according to the embodiment. FIG. 11 is a flowchart showing a flow of generating spherical image data in the embodiment. FIG. 11 is a flowchart showing a nearby object determination process according to the embodiment. 4A to 4C are diagrams for explaining display contents of a display unit in the embodiment. 13 is a diagram showing the appearance of an imaging device according to a modified example of the embodiment. FIG. 13 is a diagram showing a configuration of a processing block of a processing circuit in a modified example. 13 is a diagram showing the external appearance of an imaging device according to a second modified example of an embodiment of the present invention. FIG. 13 is a diagram showing a configuration of a processing block of a processing circuit in a second modified example. FIG. 11 is a flowchart showing a process for determining an adjacent object in the second modified example. FIG. 13 is a diagram for explaining a configuration of an imaging device according to a third modified example of an embodiment of the present invention.

Embodiments of the imaging device and imaging processing method are described in detail below with reference to the attached drawings.

Fig. 1 is a diagram showing an example of the external appearance of an imaging device according to an embodiment of the present invention. Fig. 2 is a diagram for explaining the configuration of the imaging device. Fig. 2 shows the internal configuration of the imaging device of Fig. 1.

The imaging device 1 is an example of an information processing device that outputs three-dimensional information determined based on received light, and has an imaging unit (camera) 11, a projection unit (corresponding to the light-emitting unit of the distance sensor) 12 that projects light other than visible light, and a distance information acquisition unit (corresponding to the light-receiving unit of the distance sensor) 13 that acquires distance information based on the light projected by the projection unit 12, all of which are provided integrally with the housing 10. Each unit is electrically connected to a processing circuit 14 inside the housing 10 by a synchronization signal line L, and operates synchronously with each other.

The photographing switch 15 is for the user to input a photographing instruction signal to the processing circuit 14. The display unit 20 displays the contents according to the output signal of the processing circuit 14 and is composed of a liquid crystal screen or the like. The display unit 20 may be composed of a touch panel or the like and be configured to accept user operation input. Based on the photographing instruction, the processing circuit 14 controls each part to acquire RGB image and distance information data, and performs processing to reconstruct the acquired distance information data into high-density three-dimensional point cloud data based on the RGB image and distance information data. The distance information data can be used as is to construct three-dimensional point cloud data, but in that case, the accuracy of the three-dimensional point cloud data is limited by the number of pixels (resolution) of the distance information acquisition unit 13. In this example, the processing for reconstructing it into high-density three-dimensional point cloud data is also shown. The reconstructed data is output to an external PC or the like via a portable recording medium or communication, and is used to display a three-dimensional reconstruction model.

Each unit and the processing circuit 14 are supplied with power from a battery housed inside the housing 10. Alternatively, power may be supplied from outside the housing 10 via a connection cord.

The imaging unit 11 has imaging elements 11a, 11A, fisheye lenses (wide-angle lenses) 11b, 11B, etc. The projection unit 12 has light source units 12a, 12A, wide-angle lenses 12b, 12B, etc. The distance information acquisition unit 13 has TOF (Time Of Flight) sensors 13a, 13A, wide-angle lenses 13b, 13B, etc. Each unit may be configured as an optical system such as a prism or a lens group, although not shown. For example, the imaging unit 11 may be configured as an optical system for forming an image on the imaging elements 11a, 11A using light collected by the fisheye lenses 11b, 11B. The projection unit 12 may be configured as an optical system for guiding the light of the light source units 12a, 12A to the wide-angle lenses 12b, 12B. In addition, an optical system may be configured in the distance information acquisition unit 13 to focus the light collected by the wide-angle lenses 13b and 13B on the TOF sensors 13a and 13A. Each optical system may be appropriately determined depending on the configuration and arrangement of the image pickup elements 11a and 11A, the light source units 12a and 12A, the TOF sensors 13a and 13A, etc., and the optical systems such as prisms and lens groups will not be described here.

The image sensors 11a, 11A, the light sources 12a, 12A, and the TOF sensors 13a, 13A are housed integrally inside the housing 10. The fisheye lens 11b, the wide-angle lens 12b, the wide-angle lens 13b, and the display unit 20 are each provided on a first surface on the front side of the housing 10. On the first surface, the inner ranges of the fisheye lens 11b, the wide-angle lens 12b, and the wide-angle lens 13b are each open.

The fisheye lens 11B, the wide-angle lens 12B, the wide-angle lens 13B, and the shooting switch 15 are each provided on a second surface on the rear side of the housing 10. On the second surface, the inner ranges of the fisheye lens 11B, the wide-angle lens 12B, and the wide-angle lens 13B are each open.

The imaging elements 11a and 11A are two-dimensional resolution image sensors (area sensors). The imaging elements 11a and 11A have an imaging area in which a large number of light receiving elements (photodiodes) for each pixel are arranged in a two-dimensional direction. The imaging area is provided with R (Red), G (Green), and B (Blue) color filters such as a Bayer array to receive visible light, and light that passes through the color filters is stored in the photodiodes. Here, an image sensor with a large number of pixels is used so that a wide-angle (for example, a range of a hemisphere of 180 degrees around the imaging direction shown in FIG. 2) two-dimensional image can be obtained with high resolution. The imaging elements 11a and 11A convert the light imaged in the imaging area into an electrical signal using the pixel circuit of each pixel, and output a high-resolution RGB image. The fisheye lenses 11b and 11B collect light from a wide angle (for example, a hemispherical range of 180 degrees around the imaging direction shown in FIG. 2) and focus the light on the imaging areas of the imaging elements 11a and 11A.

The light source units 12a and 12A are semiconductor lasers that emit laser light in a wavelength band other than the visible light region used for measuring distances (infrared is used here as an example). For the light source units 12a and 12A, one semiconductor laser may be used, or multiple semiconductor lasers may be used in combination. In addition, a surface-emitting semiconductor laser such as a VCSEL (Vertical Cavity Surface Emitting LASER) may be used as the semiconductor laser. In addition, the light of the semiconductor laser may be shaped to be vertically elongated by an optical lens, and the vertically elongated light may be scanned in a one-dimensional direction of the measurement range by an optical deflection element such as a MEMS (Micro Electro Mechanical Systems) mirror. In this embodiment, the light source units 12a and 12A are configured to expand the light of the semiconductor laser LA to a wide angle range via wide-angle lenses 12b and 12B without using an optical deflection element such as a MEMS mirror.

The wide-angle lenses 12b and 12B of the light source units 12a and 12A have the function of expanding the light emitted by the light source units 12a and 12A to a wide-angle range (for example, a hemispherical range of 180 degrees around the imaging direction as shown in FIG. 2).

The wide-angle lenses 13b and 13B of the distance information acquisition unit 13 capture reflected light of the light from the light source units 12a and 12A projected by the projection unit 12 from each direction of the wide angle (for example, a hemispherical range of 180 degrees around the imaging direction shown in FIG. 2) that is the measurement range, and form an image of that light on the light receiving area of the TOF sensors 13a and 13A. The measurement range includes one or more projection objects (for example, buildings, etc.), and light reflected by the projection objects (reflected light) enters the wide-angle lenses 13b and 13B. The reflected light may be captured by, for example, providing a filter that cuts light with wavelengths in the infrared range or longer on the entire surface of the wide-angle lenses 13b and 13B. Note that this is not limited to this, and since it is sufficient that light in the infrared range enters the light receiving area, a means for passing light with wavelengths in the infrared range, such as a filter, may be provided on the optical path from the wide-angle lenses 13b and 13B to the light receiving area.

TOF sensors 13a and 13A are optical sensors with two-dimensional resolution. TOF sensors 13a and 13A have a light receiving area in which many light receiving elements (photodiodes) are arranged in a two-dimensional direction. In this sense, they can be called "second imaging light receiving means." TOF sensors 13a and 13A receive reflected light from each area (each area is also called a position) in the measurement range using light receiving elements corresponding to each area, and measure (calculate) the distance to each area based on the light detected by each light receiving element.

In this embodiment, distance is measured using a phase difference detection method. In this method, laser light that has been amplitude modulated at a fundamental frequency is irradiated onto the measurement range, the reflected light is received, and the phase difference between the irradiated light and the reflected light is measured to determine the time, which is then multiplied by the speed of light to calculate the distance. An advantage of this method is that it can provide a certain degree of resolution.

TOF sensors 13a and 13A are driven in synchronization with the irradiation of light by projection unit 12, calculate the distance corresponding to each pixel from the phase difference with the reflected light at each light receiving element (corresponding to a pixel), and output distance information image data (hereinafter also referred to as "distance image" or "TOF image") in which pixel information is associated with information indicating the distance to each area within the measurement range. TOF sensors 13a and 13A may output phase information image data in which pixel information is associated with phase information, and acquire distance information image data based on the phase information image data in post-processing.

The number of areas into which the measurement range can be divided is determined by the resolution of the light receiving area. Therefore, if a low resolution is used to reduce size, the number of pixels of distance image data will decrease, and the number of 3D point clouds will also decrease.

As another embodiment, distance may be measured using a pulse method instead of the phase difference detection method. In this case, for example, the light source unit 12a, 12A emits an ultrashort pulse irradiation pulse P1 with a rise time of several nanoseconds (ns) and a strong optical peak power, and the TOF sensor 13a, 13A measures the time (t) required to receive the reflected pulse P2, which is the reflected light of the irradiation pulse P1 emitted by the light source unit 12a, 12A, in synchronization with this. When this method is adopted, for example, the TOF sensor 13a, 13A is used that is equipped with a circuit that measures time on the output side of the light receiving element. In each circuit, the time required from the light source unit 12a, 12A emitting the irradiation pulse P1 to receiving the reflected pulse P2 is converted into a distance for each light receiving element, and the distance to each area is obtained.

This method is suitable for widening the angle of the imaging device 1 because it can output powerful light using peak light. Also, if a MEMS mirror or the like is used to deflect (scan) the light, powerful light can be emitted farther while suppressing the spread, which leads to an increase in the measurement distance. In this case, the laser light emitted from the light source units 12a and 12A is arranged so that it is scanned (deflected) by the MEMS mirror toward the wide-angle lenses 12b and 12B.

Note that, although it is desirable that the effective angle of view of the imaging unit 11 and the effective angle of view of the distance information acquisition unit 13 are the same, for example, at 180 degrees or more, they do not necessarily have to be the same. If necessary, the effective angle of view of the imaging unit 11 and the effective angle of view of the distance information acquisition unit 13 may be reduced. In this embodiment, the effective pixels of the imaging unit 11 and the distance information acquisition unit 13 are reduced, for example, to within a range of 100 degrees to 180 degrees so that there is no interference with the angle of view. In addition, the resolution of the TOF sensors 13a and 13A may be set lower than the resolution of the imaging elements 11a and 11A, prioritizing the miniaturization of the imaging device 1. By making the TOF sensors 13a and 13A lower resolution than the imaging elements 11a and 11A, the size expansion of the light receiving area can be suppressed, which can lead to the miniaturization of the imaging device 1. For this reason, the TOF sensors 13a and 13A have low resolution, and the 3D point clouds obtained by the TOF sensors 13a and 13A have low density, but because the processing circuit 14, which is the "acquisition means," is provided, it can be converted into a high-density 3D point cloud. The process of converting into a high-density 3D point cloud in the processing circuit 14 will be described later.

In this embodiment, as an example, the image sensor 11a, the light source unit 12a, and the TOF sensor 13a are arranged so as to be aligned in a straight line in the longitudinal direction of the housing 10. Similarly, the image sensor 11A, the light source unit 12A, and the TOF sensor 13A are arranged so as to be aligned in a straight line in the longitudinal direction of the housing 10. Below, an example of the image sensor 11a, the light source unit 12a, and the TOF sensor 13a will be described.

The imaging area (imaging surface) of the imaging element 11a and the light receiving area (light receiving surface) of the TOF sensor 13a may be arranged in a direction perpendicular to the longitudinal direction as shown in FIG. 2, or may be arranged in the longitudinal direction by providing a prism that changes the straight-line direction (optical path) of light by 90 degrees and makes it incident. In addition, they may be arranged in any direction depending on the configuration. In other words, the imaging element 11a, the light source unit 12a, and the TOF sensor 13a are arranged so that the same measurement range is the target. The imaging unit 11, the projection unit 12, and the distance information acquisition unit 13 are arranged from one side of the housing 10 toward the measurement range. In this case, it is sufficient that the imaging element 11a and the TOF sensor 13a are arranged on the same baseline so as to be parallel stereo. By arranging them so as to be parallel stereo, even if there is only one imaging element 11a, it is possible to obtain parallax data using the output of the TOF sensor 13a. The light source unit 12a is configured to irradiate light into the measurement range of the TOF sensor 13a.

(Processing circuit)
Next, the processing of the processing circuit 14 will be described. The TOF image obtained only by the TOF sensors 13a and 13A has low resolution as it is. For this reason, in this example, the resolution is increased by the processing circuit 14, and high-density three-dimensional point cloud data is reconstructed. Note that some or all of the processing described below as "information processing means" in the processing circuit 14 may be performed by an external device.

As mentioned above, the 3D point cloud data reconstructed by the imaging device 1 is output to an external device such as a PC via a portable recording medium or communication, and is used to display the 3D reconstruction model.

This makes it possible to provide an imaging device 1 that is faster, smaller, and lighter, making it more portable than when the imaging device 1 itself displays the 3D reconstruction model.

However, after leaving the site where the 3D information was acquired and restoring the 3D information using an external device, it may be noticed that the photographer or the tripod is still visible in the captured image, or that the 3D information was not captured in the desired layout. In such cases, it may be necessary to go to the trouble of revisiting the site where the 3D information was acquired.

One solution to this problem would be to bring in a 3D reconstruction device on-site, but this would negate the benefits of high speed, compact size, and lightweight design.

It is also possible to transmit the acquired 3D information to an external device via a communication line and receive the restored 3D information, but this would eliminate the benefit of high speed, and in addition, 3D information contains a large amount of information, making it difficult to visually check whether the photographer, the tripod, etc. are reflected in the captured image.

In particular, when it comes to spherical 3D information, it is extremely difficult to visually check whether the photographer, a tripod, or other objects are reflected in the captured image.

In view of the above problems, this embodiment aims to provide an imaging device 1 that makes it easy to check in real time whether the photographer himself or a tripod is captured in the captured image, or whether 3D information of the desired layout has not been acquired.

Figure 3 is a diagram for explaining the usage status of the imaging device in this embodiment.

In the state shown in FIG. 3(a), the photographer M and the selfie stick 1A supporting the imaging device 1 are not included in the spherical imaging range R, and the photographer M and the selfie stick 1A are not captured in the spherical captured image.

In the state shown in FIG. 3(b), the photographer M is included in the spherical imaging range R, and the photographer M is captured in the spherical captured image.

In the state shown in FIG. 3(c), the tripod 1B supporting the imaging device 1 is included in the omnidirectional imaging range R, and the tripod 1B is captured in the omnidirectional captured image.

In the state shown in FIG. 3(d), the photographer M and the selfie stick 1A supporting the imaging device 1 are not included in the spherical imaging range R, and the photographer M and the selfie stick 1A are not captured in the spherical captured image. However, because of the strong external light, there is a possibility that the photographer M and the selfie stick 1A are erroneously determined to be captured.

In addition, in the conditions shown in Figures 3(b) and 3(c), the colors and types of objects reflected in the image, as well as the way they appear, are diverse, making it difficult to uniformly determine whether or not an object is reflected.

In the above situation, when determining whether or not there is a specific object (nearby object) such as the photographer himself or a tripod based on the distance information image data output from the TOF sensors 13a and 13A, it is difficult to distinguish whether a specific object is actually present or whether the external light is too strong.

In other words, when the charge storage amount of a particular pixel of the TOF sensors 13a and 13A becomes saturated, it is difficult to distinguish from the output of the TOF sensors 13a and 13A alone whether this is due to the presence of a particular object or because the intensity of external light is too strong.

In view of the above problems, another objective of this embodiment is to provide an imaging device 1 that can accurately check whether or not a specific object, such as the photographer himself or herself or a tripod, is reflected in a captured image, distinguishing it from the effects of external light.

Figure 4 is a diagram showing an example of the configuration of the processing block of the processing circuit 14. The processing circuit 14 shown in Figure 4 has a control unit 141, an RGB image data acquisition unit 142, a monochrome processing unit 143, a TOF image data acquisition unit 144, a high resolution unit 145, a matching processing unit 146, a reprojection processing unit 147, a semantic segmentation unit 148, a parallax calculation unit 149, a 3D reconstruction processing unit 150, a judgment unit 160, a display control unit 170 which is an example of an output unit, and a transmission/reception unit 180 which is an example of an output unit. In Figure 4, the solid arrows indicate the flow of signals, and the dashed arrows indicate the flow of data.

When the control unit 141 receives an ON signal (shooting start signal) from the shooting switch 15, it outputs a synchronization signal to the image pickup elements 11a, 11A, the light source units 12a, 12A, and the TOF sensors 13a, 13A, and controls the entire processing circuit 14. The control unit 141 first outputs a signal to the light source units 12a, 12A to instruct them to emit ultrashort pulses, and at the same timing outputs a signal to the TOF sensors 13a, 13A to instruct them to generate TOF image data. Furthermore, the control unit 141 outputs a signal to the image pickup elements 11a, 11A to instruct them to capture images. Note that the image pickup at the image pickup elements 11a, 11A may be during the period when light is emitted from the light source units 12a, 12A, or may be during the period immediately before or after that.

The RGB image data acquisition unit 142 acquires RGB image data captured by the image capture elements 11a and 11A based on an image capture instruction from the control unit 141, and outputs RGB image data of the entire celestial sphere. The monochrome processing unit 143 performs processing to align data types for matching processing with TOF image data obtained from the TOF sensors 13a and 13A. In this example, the monochrome processing unit 143 performs processing to convert the RGB image data of the entire celestial sphere into a monochrome image of the entire celestial sphere.

The TOF image data acquisition unit 144 acquires the TOF image data generated by the TOF sensors 13a and 13A based on an instruction to generate TOF image data from the control unit 141, and outputs TOF image data of the entire celestial sphere.

The high-resolution unit 145 treats the TOF image data of the celestial sphere as a monochrome image and increases the resolution of the image. Specifically, the high-resolution unit 145 replaces the distance value associated with each pixel of the TOF image data of the celestial sphere with the value (grayscale value) of the monochrome image of the celestial sphere and uses the result. Furthermore, the high-resolution unit 145 increases the resolution of the monochrome image of the celestial sphere to the resolution of the RGB image data of the celestial sphere obtained from the image sensors 11a and 11A. The conversion to high resolution is performed, for example, by performing a normal up-conversion process. As another conversion method, for example, multiple frames of TOF image data of the celestial sphere that are generated continuously may be obtained, and the distances of adjacent points may be added using the frames to perform super-resolution processing.

The matching processing unit 146 extracts features of textured portions of a monochrome image of the celestial sphere obtained by increasing the resolution of the TOF image data of the celestial sphere and a monochrome image of the celestial sphere obtained by increasing the resolution of the TOF image data of the celestial sphere, and performs a matching process using the extracted features. For example, the matching processing unit 146 extracts edges from each monochrome image and performs a matching process between the extracted edge information. As another method, the matching process may be performed using a method that converts changes in texture into features, such as SIFT. Here, the matching process refers to a search for corresponding pixels.

One example of a specific matching technique is block matching. Block matching is a method that calculates the similarity between pixel values cut out as a block of M x M pixels (M is a positive integer) around a reference pixel and pixel values cut out as a block of M x M pixels around a pixel that is the center of the search in another image, and then determines the central pixel with the highest similarity as the corresponding pixel.

There are various methods for calculating similarity. For example, an equation showing the normalized autocorrelation coefficient (CNCC) may be used. The higher the normalized autocorrelation coefficient (CNCC) value, the higher the similarity, and the value is 1 when the pixel values of the blocks are perfectly identical.

In addition, because distance data for textureless regions can also be obtained from spherical TOF image data, weighting may be applied to the matching process depending on the region. For example, in calculating the formula showing CNCC, a calculation may be performed in which weighting is applied to areas other than edges (textureless regions).

In addition, instead of the formula indicating NCC, selective correlation coefficient (SCC) or the like may be used.

The reprojection processing unit 147 performs a process of reprojecting the TOF image data of the celestial sphere, which indicates the distance of each position in the measurement range, onto the two-dimensional coordinates (screen coordinate system) of the imaging unit 11. Reprojection means determining at which coordinates the three-dimensional points calculated by the TOF sensors 13a and 13A are reflected in the images of the imaging elements 11a and 11A. The TOF image data of the celestial sphere indicates the position of the three-dimensional point in a coordinate system centered on the distance information acquisition unit 13 (mainly the wide-angle lenses 13b and 13B). Therefore, the three-dimensional points indicated by the TOF image data of the celestial sphere are reprojected onto a coordinate system centered on the imaging unit 11 (mainly the fisheye lenses 11b and 11B). For example, the reprojection processing unit 147 translates the coordinates of three-dimensional points in the TOF image data of the celestial sphere to the coordinates of three-dimensional points centered on the imaging unit 11, and after the translation, performs processing to convert the coordinates into a two-dimensional coordinate system (screen coordinate system) indicated by the RGB image data of the celestial sphere.

The parallax calculation unit 149 calculates the parallax at each position from the difference in distance to the corresponding pixel obtained by the matching process.

The parallax matching process uses the reprojection coordinates converted by the reprojection processing unit 147 to search for surrounding pixels around the position of the reprojection coordinates, making it possible to reduce processing time and obtain more detailed, high-resolution distance information.

In addition, the segmentation data obtained by the semantic segmentation process of the semantic segmentation unit 148 may be used for the disparity matching process. In this case, it becomes possible to obtain even more detailed and high-resolution distance information.

In addition, disparity matching processing may be performed only on edges or only on parts with strong features, and propagation processing may be performed for other parts using spherical TOF image data, for example, using spherical RGB image features or probabilistic methods.

The semantic segmentation unit 148 uses deep learning to assign segmentation labels indicating the object to the input image in the measurement range. This makes it possible to constrain each pixel of the omnidirectional TOF image data to one of multiple distance regions divided by distance, further increasing the reliability of the calculation.

The 3D reconstruction processing unit 145 acquires RGB image data of the celestial sphere from the RGB image data acquisition unit 142, reconstructs three-dimensional data of the celestial sphere based on the distance information output by the parallax calculation unit 149, and outputs a high-density three-dimensional point cloud of the celestial sphere in which color information is added to each three-dimensional point. The 3D reconstruction processing unit 150 is an example of a three-dimensional information determination unit that determines three-dimensional information.

The determination unit 160 acquires RGB image data of the celestial sphere from the RGB image data acquisition unit 142, and acquires TOF image data of the celestial sphere converted into a two-dimensional coordinate system indicated by the RGB image data of the celestial sphere from the reprojection processing unit 147. Based on these data, the determination unit 160 determines whether or not a specific object is reflected in the captured image, and outputs the determination result to the display control unit 170.

The display control unit 170 acquires RGB image data of the entire celestial sphere from the RGB image data acquisition unit 142, and causes the display unit 20 to display two-dimensional image information based on the acquired RGB image data of the entire celestial sphere. The display control unit 170 also causes the display unit 20 to display information indicating the judgment result acquired from the judgment unit 160 by superimposing it on the two-dimensional image information.

The display control unit 170 is an example of an output unit that outputs two-dimensional image information captured by the imaging unit 11 separately from the three-dimensional information, and the display unit 20 is an example of an output destination that outputs the two-dimensional image information.

The display control unit 170 may acquire three-dimensional data of the entire celestial sphere from the three-dimensional reconstruction processing unit 145, and cause the display unit 20 to display the three-dimensional information. Specifically, the display control unit 170 may select, according to a predetermined condition, whether to display two-dimensional image information on the display unit 20 or to display three-dimensional information on the display unit 20. This allows the display control unit 170 to output the two-dimensional image information separately from the three-dimensional information.

The transmission/reception unit 180 communicates with an external device using wired or wireless technology, and transmits (outputs) the three-dimensional data of the celestial sphere output from the three-dimensional reconstruction processing unit 145 and the two-dimensional image information of the celestial sphere output from the RGB image data acquisition unit 142 via the network 400 to the external device 300 that performs three-dimensional reconstruction processing.

The transmitter/receiver 180 is an example of an output unit that outputs three-dimensional information, and the external device 300 is an example of an output destination that outputs three-dimensional information.

The transmitting/receiving unit 180 may transmit only the three-dimensional data of the celestial sphere, without transmitting the two-dimensional image information of the celestial sphere. The transmitting/receiving unit 180 may also be configured with an interface circuit for a portable storage medium such as an SD card, a personal computer, or the like.

(Operation of the Processing Circuit)
5 is a flow diagram showing an example of the operation of the processing circuit 14 of the imaging device 1. When the user turns on the imaging switch 15 and inputs an imaging instruction signal, the control unit 141 of the processing circuit 14 performs an operation of generating a high-density three-dimensional point cloud in the following manner (an example of an imaging processing method and an information processing method).

First, the control unit 141 drives the light source units 12a and 12A, the TOF sensors 13a and 13A, and the image sensors 11a and 11A to capture an image of the measurement range (step S1). Driven by the control unit 141, the light source units 12a and 12A irradiate infrared light (an example of a projection step), and the TOF sensors 13a and 13A receive the reflected light (an example of a light reception step). In addition, the image sensors 11a and 11A capture an image of the measurement range at the timing when the light source units 12a and 12A start to be driven or in the period immediately prior to that (an example of an image capture step).

Next, the RGB image data acquisition unit 142 acquires RGB image data of the measurement range from the image sensors 11a and 11A (step S2). The display control unit 170 then acquires RGB image data of the entire celestial sphere from the RGB image data acquisition unit 142, and causes the display unit 20 to display two-dimensional image information based on the acquired RGB image data of the entire celestial sphere (an example of a two-dimensional image information output step) (step S3).

The display control unit 170 displays two-dimensional image information of a partial area of the acquired spherical RGB image data on the display unit 20, and changes the area of the two-dimensional image information displayed on the display unit 20 according to various inputs from the user. Various inputs from the user can be realized by providing an operation switch other than the shooting switch 15, or by configuring the display unit 20 as an input unit such as a touch panel.

At this stage, the photographer can check from the two-dimensional image information displayed on the display unit 20 that the photographer himself or the tripod is not reflected in the captured image, and that two-dimensional image information of the desired layout has not been obtained.

Next, the TOF image data acquisition unit 144 acquires TOF image data indicating the distance from the TOF sensors 13a and 13A to each position in the two-dimensional area (step S4).

Next, the monochrome processing unit 143 converts the RGB image data into a monochrome image (step S5). The TOF image data and the RGB image data are different data types, namely distance data and RGB data, and matching cannot be performed as is. Therefore, each data is first converted into a monochrome image. For the TOF image data, the high resolution unit 145 converts the values indicating the distance of each pixel before high resolution conversion by directly replacing them with the values of the monochrome image.

Next, the high-resolution unit 145 increases the resolution of the TOF image data (step S6).

Next, the matching processing unit 146 extracts features of the textured parts of each monochrome image and performs matching processing using the extracted features (step S7).

Next, the parallax calculation unit 149 calculates the parallax at each position from the difference in distance between corresponding pixels to calculate the distance (step S8).

Next, the determination unit 160 acquires RGB image data of the entire celestial sphere from the RGB image data acquisition unit 142, and acquires TOF image data of the entire celestial sphere converted into the two-dimensional coordinate system indicated by the RGB image data from the reprojection processing unit 147, and determines whether or not a nearby object as a specific target is reflected in the captured image based on these data, and outputs the determination result to the display control unit 170. The display control unit 170 superimposes information indicating the determination result acquired from the determination unit 160 on the two-dimensional image information and displays it on the display unit 20 (an example of a display step) (step S9).

Then, the 3D reconstruction processing unit 145 acquires the RGB image data from the RGB image data acquisition unit 142, reconstructs the 3D data based on the distance information output by the parallax calculation unit 149, and outputs a high-density 3D point cloud with color information added to each 3D point (step S10).

Next, the transmission/reception unit 180 transmits the three-dimensional data output from the three-dimensional reconstruction processing unit 145 and the two-dimensional image information output from the RGB image data acquisition unit 142 via the network 400 to the external device 300 that performs the three-dimensional reconstruction process (an example of a three-dimensional information output step) (step S11).

The transmitter/receiver 180 may transmit the three-dimensional data output from the three-dimensional reconstruction processing unit 145 without transmitting the two-dimensional image information output from the RGB image data acquisition unit 142.

As described above, the imaging device 1 includes an imaging unit 11 and a display control unit 170 that outputs two-dimensional image information captured by the imaging unit 11 separately from the three-dimensional information.

This makes it easy to check from the 2D image information whether the photographer or a tripod is captured in the captured image, or whether 3D information of the desired layout has not been obtained, without checking the 3D information.

This makes it possible to reacquire 3D information while still at the site where the 3D information is being acquired, reducing the effort required to revisit the site where the 3D information is being acquired, compared to when, after leaving the site where the 3D information is being acquired, one realizes that the photographer or the tripod is still in the captured image, or that 3D information of the desired layout has not been acquired.

The three-dimensional information includes spherical three-dimensional information. In this case, even in the case of spherical three-dimensional information in which it is difficult to confirm that the photographer himself, the tripod, etc. are not captured in the captured image, or that three-dimensional information of the desired layout has not been obtained, it becomes possible to easily confirm from the two-dimensional image information captured by the imaging unit 11 that the photographer himself, the tripod, etc. are not captured in the captured image, or that three-dimensional information of the desired layout has not been obtained.

The display control unit 170 outputs the two-dimensional image information G in step S3 before the transmitting/receiving unit 180 transmits (outputs) the three-dimensional information in step S11. The display control unit 170 outputs the two-dimensional image information G in step S3 before the three-dimensional reconstruction processing unit 150 determines the three-dimensional information in step S10.

This makes it possible to check from the 2D image information before checking the 3D information whether the photographer or a tripod is captured in the captured image, or whether 3D information of the desired layout has not been obtained.

The display control unit 170 causes the display unit 20 to display two-dimensional image information. The imaging device 1 is equipped with the display unit 20.

This makes it easy to check from the two-dimensional image information displayed on the display unit 20 whether the photographer himself or the tripod is captured in the captured image, or whether three-dimensional information of the desired layout has not been obtained.

The display control unit 170 outputs the two-dimensional image information to a display unit 20 that is different from the external device 300 to which the transmission/reception unit 180 outputs the three-dimensional information.

This makes it possible to check whether the photographer himself/herself, the tripod, etc. are captured in the captured image, or whether 3D information of the desired layout has not been obtained, from the 2D image information output to a display unit 20 different from the external device 300, without checking the 3D information output to the external device 300.

The imaging device 1 includes a 3D reconstruction processing unit 150 that determines 3D information based on the output of the distance information acquisition unit 13. The 3D reconstruction processing unit 150 determines 3D information based on the output of the distance information acquisition unit 13 and 2D image information.

This makes it possible to check from the 2D image information captured by the imaging unit 11 whether the photographer himself or the tripod is captured in the captured image, or whether 3D information of the desired layout has not been obtained, without checking the 3D information determined by the 3D reconstruction processing unit 150.

Figure 6 shows a flow diagram of spherical image data generation in this embodiment.

Figure 6 (a) is a flowchart showing the process of generating spherical RGB image data corresponding to step S2 described in Figure 5.

The RGB image data acquisition unit 142 inputs two pieces of RGB image data in fisheye image format (step S201).

The RGB image data acquisition unit 142 converts each RGB image data into an equirectangular image format (step S202). The RGB image data acquisition unit 142 converts the two RGB image data into an equirectangular image format based on the same coordinate system, making it easier to combine the images in the next step.

Here, we will explain the equirectangular image format. The equirectangular image format is a method that can represent a full celestial sphere image, and is a format for images (equirectangular images) created using equirectangular projection. Equirectangular projection is a projection that represents three-dimensional directions with two variables, such as the latitude and longitude of a globe, and displays the image on a plane so that the latitude and longitude are orthogonal. Therefore, an equirectangular image is an image created using equirectangular projection, and is represented in coordinates with two axes that are the two angular variables of a spherical coordinate system.

The RGB image data acquisition unit 142 combines the two RGB image data generated in step S202 to generate one piece of spherical RGB image data (step S203). The two pieces of input RGB image data cover an area with a full angle of view of more than 180 degrees. Therefore, the spherical RGB image data generated by appropriately connecting the two pieces of RGB image data can cover the spherical area.

The combining process in step S203 can use existing technology for joining multiple images, and there are no particular limitations on the method.

Figure 6(b) is a flowchart showing the process of generating TOF image data of the entire celestial sphere, which corresponds to step S4 described in Figure 5.

The TOF image data acquisition unit 144 acquires two distance image data in fisheye image format (step S401).

The TOF image data acquisition unit 144 converts the two TOF image data in the fisheye image format into an equirectangular image format (step S402). As described above, the equirectangular image format is a method capable of expressing a celestial sphere image. In this step S402, the two TOF image data are converted into the equirectangular image format based on the same coordinate system, which makes it easier to combine the images in the next step S403.

The TOF image data acquisition unit 144 combines the two TOF image data generated in step S402 to generate one piece of spherical TOF image data (step S403). The two input TOF image data cover an area with a full angle of view of more than 180 degrees. Therefore, the spherical TOF image data generated by appropriately connecting the two TOF image data can cover the entire spherical area.

The combining process in step S403 can use existing technology for joining multiple images, and there are no particular limitations on the method.

Figure 7 shows a flow diagram for determining nearby objects in this embodiment.

Figure 7 is a flowchart showing the process of determining whether or not a nearby object is captured in the captured image, which corresponds to step S9 described in Figure 5.

Based on the TOF image data of the entire celestial sphere acquired from the reprojection processing unit 147, the determination unit 160 determines whether there is a pixel in the TOF image data of the entire celestial sphere whose charge amount is saturated (step S801).

When a pixel with saturated charge is found in step S801, the determination unit 160 determines whether the charge is saturated for pixels at the same coordinates as the pixel with saturated charge in step S801 in the RGB image data of the entire celestial sphere, based on the RGB image data of the entire celestial sphere acquired from the RGB image data acquisition unit 142 (step S802).

If the amount of stored power is saturated in step S802, the determination unit 160 determines that the pixel whose amount of stored power is saturated in step S801 is due to external light, and outputs error information to the display control unit 170. Based on the error information acquired from the determination unit 160, the display control unit 170 superimposes the error information on the two-dimensional image information and displays it on the display unit 20 (step S803).

If the charge amount is not saturated in step S802, the determination unit 160 determines that the pixel whose charge amount is saturated in step S801 is due to the presence of a nearby object, and outputs the coordinate position information of the pixel whose charge amount is saturated in step S801 to the display control unit 170. Based on the coordinate position information of the pixel acquired from the determination unit 160, the display control unit 170 superimposes identification information that identifies the nearby object on the two-dimensional image information and displays it on the display unit 20 (step S804).

If there are no pixels whose charge levels are saturated in step S801, the determination unit 160 determines whether there are any pixels in the TOF image data of the celestial sphere that indicate distance information of 0.5 m or less based on the TOF image data of the celestial sphere acquired from the reprojection processing unit 147 (step S805).

If there are no pixels indicating distance information of 0.5 m or less in step S805, the judgment unit 160 ends the process.

If there is a pixel showing distance information of 0.5 m or less in step S805, the determination unit 160 proceeds to step S804 described above, determines that the pixel showing distance information of 0.5 m or less in step S805 is due to the presence of a nearby object, and outputs the coordinate position information of the pixel showing distance information of 0.5 m or less in step S805 to the display control unit 170. The display control unit 170 superimposes identification information identifying the nearby object on the two-dimensional image information based on the coordinate position information of the pixel acquired from the determination unit 160 and displays it on the display unit 20.

As described above, the display control unit 170 superimposes the identification information on the two-dimensional image information when it determines that a nearby object exists, and does not superimpose the identification information on the two-dimensional image information when it does not determine that a nearby object exists.

In other words, the display control unit 170 causes the display unit 20 to display differently depending on whether or not a nearby object is present.

In addition, the display control unit 170 superimposes identification information that identifies nearby objects on the two-dimensional image information based on the pixel coordinate position information acquired from the determination unit 160 and displays it on the display unit 20.

In other words, the display control unit 170 causes the display unit 20 to display a different position depending on the position of the nearby object.

Figure 8 is a diagram for explaining the display content of the display unit in this embodiment.

Figure 8 is an explanatory diagram corresponding to step S2 shown in Figure 5 and steps S803 and S804 shown in Figure 7.

The display control unit 170 displays two-dimensional image information G on the display unit 20. The display control unit 170 also displays identification information G1, G2 for identifying nearby objects, and error information G3, each of which is superimposed on the two-dimensional image information G on the display unit 20.

As described above, the imaging device 1 includes an imaging unit 11 that images an object, a projection unit 12 that projects light onto the object, a distance information acquisition unit 13 that receives light reflected from the object, and a display control unit 170 that causes the display unit 20 to display different images depending on the presence or absence of a nearby object determined based on the output of the distance information acquisition unit 13 and the output of the imaging unit 11.

This allows the photographer to distinguish between the effects of external light and accurately check whether the photographer himself or nearby objects such as a tripod are reflected in the captured image.

The imaging device 1 is equipped with a display unit 20. This allows the photographer to reliably check whether or not a nearby object is captured in the captured image.

The display control unit 170 causes the display unit 20 to display different positions depending on the position of the nearby object. This allows the photographer to check the position of the nearby object reflected in the captured image.

The display control unit 170 causes the display unit 20 to display the image information G captured by the imaging unit 11, and also causes the display unit 20 to display the identification information G1, G2 for identifying nearby objects superimposed on the image information. This allows the photographer to reliably confirm the position of the nearby object reflected in the captured image.

The imaging device 1 includes a determination unit 160 that determines that an object is present nearby when the amount of charge stored by the light received by the distance information acquisition unit 13 is saturated and the amount of charge stored in the pixels of the imaging unit 11 is not saturated.

This allows the photographer to distinguish between the effects of external light and accurately check whether nearby objects are reflected in the captured image.

Figure 9 is a diagram showing the appearance of an imaging device according to a modified example of the embodiment. Figure 10 is a diagram showing the configuration of a processing block of a processing circuit according to the modified example.

In this modified example, the display control unit 170 acquires RGB image data of the entire celestial sphere from the RGB image data acquisition unit 142, and causes the display unit 520 of the display device 500 to display two-dimensional image information based on the acquired RGB image data of the entire celestial sphere. The display unit 520 is an example of an output destination that outputs the two-dimensional image information.

This makes it easy to check from the two-dimensional image information displayed on the display unit 520 whether the photographer himself or the tripod is captured in the captured image, or whether three-dimensional information of the desired layout has not been obtained.

The display control unit 170 outputs the two-dimensional image information to a display unit 520 that is different from the external device 300 to which the transmission/reception unit 180 outputs the three-dimensional information.

This makes it possible to check whether the photographer himself or the tripod is captured in the captured image, or whether 3D information of the desired layout has not been obtained, from the 2D image information output to a display unit 520 different from the external device 300, without checking the 3D information output to the external device 300.

The display control unit 170 may acquire three-dimensional data of the entire celestial sphere from the three-dimensional reconstruction processing unit 145, and display the three-dimensional information on the display unit 520. Specifically, the display control unit 170 may select whether to display two-dimensional image information on the display unit 520 or to display three-dimensional information on the display unit 520 according to a predetermined condition. This allows the display control unit 170 to output the two-dimensional image information separately from the three-dimensional information.

The display control unit 170 superimposes the error information obtained from the judgment unit 160 onto the two-dimensional image information and displays it on the display unit 520.

The display control unit 170 superimposes identification information for identifying nearby objects on the two-dimensional image information based on the pixel coordinate position information acquired from the determination unit 160 and displays it on the display unit 520.

In other words, the display control unit 170 causes the display unit 520 to display different images depending on the presence or absence of a nearby object determined based on the output of the distance information acquisition unit 13 and the output of the imaging unit 11.

This allows the photographer to distinguish between the effects of external light and accurately check whether the photographer himself or nearby objects such as a tripod are reflected in the captured image.

The display control unit 170 causes the display unit 520 to display different positions depending on the position of the nearby object. This allows the photographer to confirm the position of the nearby object reflected in the captured image.

The display control unit 170 causes the display unit 520 to display the image information captured by the imaging unit 11, and also causes the display unit 520 to display identification information for identifying the nearby object superimposed on the image information. This allows the photographer to reliably confirm the position of the nearby object reflected in the captured image.

Figure 11 is a diagram showing the external appearance of an imaging device according to a second modified embodiment of the present invention. Figure 12 is a diagram showing the configuration of a processing block of a processing circuit in the second modified embodiment.

In the second modified example shown in FIG. 11, the imaging device 1 has a plurality of display units 20A, 20a instead of the display unit 20 shown in FIG. 1. The display units 20A, 20a are composed of LEDs or the like, and blink or light up in response to the output signal of the processing circuit 14.

Display unit 20a is provided on a first surface on the front side of housing 10, and display unit 20A is provided on a second surface on the rear side of housing 10.

In the second modified example shown in FIG. 12, the display control unit 170 causes the display units 20A and 20a to display information indicating the judgment result obtained from the judgment unit 160.

The transmitter/receiver 180 also transmits (outputs) the spherical two-dimensional image information output from the RGB image data acquisition unit 142 to the display device 500 via the network 400. The display device 500 is an example of an output destination to which the two-dimensional image information is output.

That is, in the second modified example, in step S3 shown in FIG. 5, the transmitting/receiving unit 180 acquires RGB image data of the entire celestial sphere from the RGB image data acquiring unit 142, and transmits (outputs) two-dimensional image information based on the acquired RGB image data of the entire celestial sphere to the display device 500.

The transmitter/receiver 510 of the display device 500 receives the two-dimensional image information transmitted from the transmitter/receiver 180 of the imaging device 1.

The control unit 530 of the display device 500 causes the two-dimensional image information received by the transmission/reception unit 510 to be displayed on the display unit 520.

As described above, the imaging device 1 includes an imaging unit 11 and a transmission/reception unit 180 that outputs two-dimensional image information captured by the imaging unit 11 separately from the three-dimensional information.

This makes it easy to check from the 2D image information whether the photographer or a tripod is captured in the captured image, or whether 3D information of the desired layout has not been obtained, without checking the 3D information.

This makes it possible to reacquire 3D information while still at the site where the 3D information is being acquired, reducing the effort required to revisit the site where the 3D information is being acquired, compared to when, after leaving the site where the 3D information is being acquired, one realizes that the photographer or the tripod is still in the captured image, or that 3D information of the desired layout has not been acquired.

The transmitting/receiving unit 180 transmits (outputs) the two-dimensional image information G in step S3 before transmitting (outputting) the three-dimensional information in step S11. The transmitting/receiving unit 180 transmits (outputs) the two-dimensional image information G in step S3 before the three-dimensional reconstruction processing unit 150 determines the three-dimensional information in step S10.

This makes it possible to check from the 2D image information before checking the 3D information whether the photographer or a tripod is captured in the captured image, or whether 3D information of the desired layout has not been obtained.

The transmitter/receiver 180 transmits the two-dimensional image information to the display device 500, and the display device 500 displays the two-dimensional image information on the display unit 520.

This makes it easy to check from the two-dimensional image information displayed on the display unit 520 whether the photographer himself or the tripod is captured in the captured image, or whether three-dimensional information of the desired layout has not been obtained.

The transmitter/receiver 180 transmits the two-dimensional image information to a display device 500 that is different from the external device 300 that outputs the three-dimensional information.

This makes it possible to check whether the photographer himself/herself or the tripod is captured in the captured image, or whether 3D information of the desired layout has not been obtained, from the 2D image information output to the display unit 520 of the display device 500, which is different from the external device 300, without checking the 3D information output to the external device 300.

The transmitting/receiving unit 180 may transmit the three-dimensional information to the display device 500. Specifically, the transmitting/receiving unit 180 may select whether to transmit the two-dimensional image information to the display device 500 or the three-dimensional information to the display device 500 in accordance with a predetermined condition. This allows the transmitting/receiving unit 180 to transmit the two-dimensional image information to the display device 500 separately from the three-dimensional information.

Figure 13 shows a flow diagram for determining nearby objects in the second modified example.

Figure 13 is a flowchart showing the process of determining whether or not a nearby object is captured in a captured image, which corresponds to step S9 described in Figure 5 in the second modified example.

Based on the TOF image data of the entire celestial sphere acquired from the reprojection processing unit 147, the determination unit 160 determines whether there is a pixel in the TOF image data of the entire celestial sphere whose charge amount is saturated (step S811).

When a pixel with saturated charge is found in step S811, the determination unit 160 determines whether the charge is saturated for pixels at the same coordinates as the pixel with saturated charge in step S811 in the RGB image data of the entire celestial sphere, based on the RGB image data of the entire celestial sphere acquired from the RGB image data acquisition unit 142 (step S812).

If the amount of stored power is saturated in step S812, the determination unit 160 determines that the pixel whose amount of stored power is saturated in step S811 is due to external light, and outputs error information to the display control unit 170. The display control unit 170 causes the display units 20A and 20a to display the error information based on the error information acquired from the determination unit 160 (step S813).

If the charge amount is not saturated in step S812, the determination unit 160 determines that the pixel whose charge amount is saturated in step S811 is due to the presence of a nearby object, and outputs the coordinate position information of the pixel whose charge amount is saturated in step S811 to the display control unit 170. The display control unit 170 determines whether the coordinate position information is on the front side of the housing 10 based on the coordinate position information of the pixel acquired from the determination unit 160 (step S814).

If there are no pixels whose charge levels are saturated in step S811, the determination unit 160 determines whether there are any pixels in the TOF image data of the celestial sphere that indicate distance information of 0.5 m or less based on the TOF image data of the celestial sphere acquired from the reprojection processing unit 147 (step S815).

If there are no pixels indicating distance information of 0.5 m or less in step S815, the judgment unit 160 ends the process.

If there is a pixel showing distance information of 0.5 m or less in step S815, the determination unit 160 proceeds to step S814 described above, determines that the pixel showing distance information of 0.5 m or less in step S815 is due to the presence of a nearby object, and outputs the coordinate position information of the pixel showing distance information of 0.5 m or less in step S815 to the display control unit 170. The display control unit 170 determines whether the coordinate position information is on the front side of the housing 10 based on the coordinate position information of the pixel acquired from the determination unit 160.

If the display control unit 170 determines in step S814 that the display is on the front side, it causes the display unit 20a located on the front side of the housing 10 to blink (step S816).

If the display control unit 170 does not determine in step S814 that the display is on the front side, it causes the display unit 20A located on the rear side of the housing 10 to blink (step S817).

As described above, the display control unit 170 blinks the display unit 20a or the display unit 20A when it determines that a nearby object is present, and does not blink the display unit 20a or the display unit 20A when it does not determine that a nearby object is present.

In other words, the display control unit 170 causes the display unit 20a and the display unit 20A to display different images depending on whether or not a nearby object is present.

This allows the photographer to distinguish between the effects of external light and accurately check whether the photographer himself or nearby objects such as a tripod are reflected in the captured image.

In addition, the display control unit 170 blinks the display unit 20a or the display unit 20A based on the pixel coordinate position information obtained from the determination unit 160.

In other words, the display control unit 170 causes the display units 20a and 20A to display different positions depending on the position of the nearby object. This allows the photographer to confirm the position of the nearby object reflected in the captured image.

Then, the display control unit 170 causes the display unit 20A, 20a that is closer to the nearby object to display different information depending on whether or not the nearby object is present. This allows the photographer to reliably confirm the position of a specific object reflected in the captured image.

Figure 14 is a diagram for explaining the configuration of an imaging device according to a third modified embodiment of the present invention.

In the third modified example shown in FIG. 14, the imaging device 1 includes, in addition to the configuration shown in FIG. 2, another imaging section 111 having other imaging elements 111a, 111A, other fisheye lenses (wide-angle lenses) 111b, 111B, etc.

In the third modified example, the RGB imaging unit 11 and the other imaging unit 111 are provided on the same baseline. In this case, multi-eye processing becomes possible in the processing circuit 14. In other words, by simultaneously driving the imaging unit 11 and the other imaging unit 111, which are provided a specified distance apart on one surface, RGB images from two viewpoints can be obtained. This makes it possible to use parallax calculated based on the two RGB images, further improving the distance accuracy over the entire measurement range.

Specifically, when an RGB imaging unit 11 and another imaging unit 111 are provided, multi-baseline stereo (MSB) using SSSD and EPI processing can be used, as in conventional parallax calculations. Therefore, by using this, the reliability of parallax increases, making it possible to achieve high spatial resolution and accuracy.

As described above, the imaging device 1 includes another imaging unit 111, and the 3D reconstruction processing unit 150 determines 3D information based on the output of the distance information acquisition unit 13, the 2D image information, and other 2D image information captured by the other imaging unit 111.

The imaging device 1 may also include another imaging unit 111 and a three-dimensional information determination unit that determines three-dimensional information based on the two-dimensional image information and other two-dimensional image information captured by the other imaging unit 111, regardless of the output of the distance information acquisition unit 13.

This makes it possible to check from the two-dimensional image information captured by the imaging unit 11 whether the photographer himself or the tripod is captured in the captured image, or whether three-dimensional information of the desired layout has not been obtained, without checking the three-dimensional information determined by the three-dimensional reconstruction processing unit 150 based on the two-dimensional image information.

As described above, the imaging device 1 (an example of an information processing device) according to one embodiment of the present invention outputs three-dimensional information determined based on received light, and includes an imaging unit 11, and a display control unit 170 (an example of an output unit) or a transmission/reception unit 180 (an example of an output unit) that outputs two-dimensional image information G captured by the imaging unit 11 separately from the three-dimensional information.

This makes it easy to check from the 2D image information whether the photographer or a tripod is captured in the captured image, or whether 3D information of the desired layout has not been obtained, without checking the 3D information.

This makes it possible to reacquire 3D information while still at the site where the 3D information is being acquired, reducing the effort required to revisit the site where the 3D information is being acquired, compared to when, after leaving the site where the 3D information is being acquired, one realizes that the photographer or the tripod is still in the captured image, or that 3D information of the desired layout has not been acquired.

The display control unit 170 or the transmission/reception unit 180 outputs the two-dimensional image information G before outputting the three-dimensional information. The display control unit 170 or the transmission/reception unit 180 outputs the two-dimensional image information G before determining the three-dimensional information.

This makes it possible to check from the 2D image information before checking the 3D information whether the photographer or a tripod is captured in the captured image, or whether 3D information of the desired layout has not been obtained.

The display control unit 170 causes the display unit 20, 520 to display the two-dimensional image information G. The imaging device 1 includes a display unit 20.

This makes it easy to check from the 2D image information displayed on the display unit whether the photographer or the tripod is captured in the captured image, or whether 3D information of the desired layout has not been obtained.

The display control unit 170 or the transmission/reception unit 180 outputs the two-dimensional image information G to a display unit 20, 520 (an example of an output destination) that is different from the external device 300 (an example of an output destination) that outputs the three-dimensional information.

This makes it possible to check whether the photographer himself or the tripod is captured in the captured image, or whether 3D information of the desired layout has not been obtained, from the 2D image information output to a display unit 20, 520 other than the external device 300 (an example of an output destination), without checking the 3D information output to the external device 300.

The imaging device 1 includes a projection unit 12 that projects light onto an object, a distance information acquisition unit 13 (an example of a light receiving unit) that receives light reflected from the object, and a 3D reconstruction processing unit 150 (an example of a 3D information determination unit) that determines 3D information based on the output of the distance information acquisition unit 13. The 3D reconstruction processing unit 150 determines the 3D information based on the output of the distance information acquisition unit 13 and the 2D image information.

This makes it possible to check from the 2D image information captured by the imaging unit 11 whether the photographer himself or the tripod is captured in the captured image, or whether 3D information of the desired layout has not been obtained, without checking the 3D information determined by the 3D reconstruction processing unit 150.

The imaging device 1 includes another imaging unit 111, and the three-dimensional reconstruction processing unit 150 determines three-dimensional information based on the output of the distance information acquisition unit 13, the two-dimensional image information, and other two-dimensional image information captured by the other imaging unit 111.

The imaging device 1 may also include another imaging unit 111 and a three-dimensional information determination unit that determines three-dimensional information based on the two-dimensional image information and other two-dimensional image information captured by the other imaging unit 111, regardless of the output of the distance information acquisition unit 13.

This makes it possible to check from the two-dimensional image information captured by the imaging unit 11 whether the photographer himself or the tripod is captured in the captured image, or whether three-dimensional information of the desired layout has not been obtained, without checking the three-dimensional information determined by the three-dimensional reconstruction processing unit 150 based on the two-dimensional image information.

The three-dimensional information includes spherical three-dimensional information. In this case, even in the case of spherical three-dimensional information in which it is difficult to confirm that the photographer himself, the tripod, etc. are not captured in the captured image, or that three-dimensional information of the desired layout has not been obtained, it becomes possible to easily confirm from the two-dimensional image information captured by the imaging unit 11 that the photographer himself, the tripod, etc. are not captured in the captured image, or that three-dimensional information of the desired layout has not been obtained.

1. Imaging device (an example of an information processing device)
REFERENCE SIGNS LIST 10 Housing 11 Imaging unit 11a, 11A Imaging element 11b, 11B Fisheye lens 12 Projection unit 12a, 12A Light source unit 12b, 12B Wide-angle lens 13 Distance information acquisition unit (an example of a light receiving unit)
13a, 13A TOF sensor 13b, 13B wide-angle lens 14 processing circuit 15 shooting switch 20 display unit 20A, 20a display unit 111 other imaging unit 150 3D reconstruction processing unit (an example of a 3D information determination unit)
160 Determination unit 170 Display control unit (an example of an output unit)
180 Transmitting/receiving unit (an example of an output unit)
300 External device (an example of an output destination)
500 Display device (an example of an output destination)
520 Display unit (an example of an output destination)
L Synchronous signal line

JP 2018-077071 A Patent No. 5423287

Claims (9)

An information processing device that outputs three-dimensional information determined based on received light,
An imaging unit that captures two-dimensional image information ;
A projection unit that projects light onto a target;
a light receiving unit having an angle of view at least partially coincident with an angle of view of the imaging unit and configured to receive the light reflected from the object;
a three-dimensional information determination unit that determines the three-dimensional information based on an output from the light receiving unit;
an output unit that outputs the two-dimensional image information before determining the three-dimensional information;
An information processing device comprising: The information processing device according to claim 1, wherein the output unit causes a display unit to display the two-dimensional image information. The information processing device according to claim 2 , further comprising the display unit. 4. The information processing apparatus according to claim 1, wherein the output unit outputs the two-dimensional image information to an output destination different from an output destination to which the three-dimensional information is output. 5. The information processing apparatus according to claim 1, wherein the three-dimensional information determination section determines the three-dimensional information based on an output from the light receiving section and the two-dimensional image information. Another imaging unit is provided,
The information processing apparatus according to claim 5 , wherein the three-dimensional information determination section determines the three-dimensional information based on an output from the light receiving section, the two-dimensional image information, and other two-dimensional image information captured by the other imaging section. Other imaging units,
The information processing device according to any one of claims 1 to 6 , further comprising: a three-dimensional information determination unit that determines the three-dimensional information based on the two-dimensional image information and other two-dimensional image information captured by the other imaging unit. The information processing device according to claim 1 , wherein the three-dimensional information includes spherical three-dimensional information. an imaging step of capturing two-dimensional image information by an imaging unit ;
A projection step of projecting light onto an object;
a light receiving step of receiving the light reflected from the object by a light receiving unit having an angle of view at least partially coincident with an angle of view of the imaging unit;
a three-dimensional information determination step of determining three-dimensional information based on an output of the light receiving unit;
a two-dimensional image information output step of outputting the two-dimensional image information before determining the three-dimensional information;
An information processing method comprising: