JP7852001B2 - Imaging and analysis device, program, and imaging and analysis method - Google Patents

Description

This invention relates to an imaging and analysis device, program, and imaging and analysis method that support understanding the conditions of a work site by imaging the work site.

Heavy materials used in construction, such as steel sheet piles and H-beams, are transported in large trucks, often in mixed loads. During inspection, workers must visually check the quantity and condition of these materials by standing on the truck bed. This presents challenges in terms of labor costs and safety. Furthermore, monitoring progress at large construction sites incurs significant costs due to site travel, highlighting the need for efficiency improvements through remote video monitoring.

One method for addressing these challenges is to compare three-dimensional models of materials without defects or damage, and of buildings constructed according to the construction plan, with three-dimensional models representing the actual materials and buildings under construction. By comparing and identifying the differences, more efficient inspection and progress tracking become possible.

Another approach to solving these problems involves creating a three-dimensional mesh (three-dimensional model, digital twin) with textures that represent the loading status of truck cargo or the construction site. If the accuracy of such a three-dimensional model is high, it becomes possible to perform visual inspection from any viewpoint, as well as counting and dimensional measurement.

As a technique for comparing three-dimensional models, there is a material quantity calculation device described in Patent Document 1. This material quantity calculation device includes an acquisition unit that acquires a first three-dimensional model representing a predetermined space and a second three-dimensional model representing the same predetermined space but different from the first three-dimensional model. The material quantity calculation device also includes an alignment unit that aligns the first and second three-dimensional models based on the attribute information of each model. Furthermore, the material quantity calculation device includes a calculation unit that calculates the difference between the first and second three-dimensional models and outputs attribute information related to that difference and difference information indicating the amount of that difference.

International Publication No. 2020/179438

The material quantity calculation device described in Patent Document 1 has the problem of high processing costs because it recognizes attribute information and aligns the positions of two three-dimensional models. There is a need for support in understanding on-site conditions, including inspection of goods and confirmation of construction progress, by reproducing objects at the work site as three-dimensional models (digital twins) and comparing them with a reference three-dimensional model, or by improving the accuracy of the reproduction.
This invention was made in view of the above background, and aims to provide an imaging analysis device, program, and imaging analysis method that enable support for understanding on-site conditions using a three-dimensional model.

To solve the above-mentioned problems, the imaging analysis apparatus according to the present invention comprises: a three-dimensional model generation unit that generates a generated three-dimensional model, which is a three-dimensional model of the object to be imaged, based on imaging information; a comparison unit that compares the generated three-dimensional model with a reference three-dimensional model, which is a three-dimensional model of an object that is also a three-dimensional model of an object to be compared with the object to be imaged, and calculates the difference; and an additional imaging support unit that outputs a position and orientation suitable for imaging the additional imaging location, which is the part of the object to be imaged where the difference occurs.

According to the present invention, it is possible to provide an imaging analysis device, program, and imaging analysis method that enable support for understanding on-site conditions using a three-dimensional model. Other problems, configurations, and effects will be clarified by the following description of embodiments.

This is a functional block diagram of the imaging and analysis device according to the first embodiment. This is a screen configuration diagram of the imaging analysis results screen according to the first embodiment. This is a screen configuration diagram of the imaging analysis results screen, which displays a pointer indicating the position and orientation of additional imaging according to the first embodiment. This is a screen configuration diagram of the imaging analysis results screen after additional imaging according to the first embodiment. This is a flowchart of the imaging analysis process according to the first embodiment. This is a schematic diagram showing the video analysis device according to the second embodiment. This is an example of video data of a bridge pier. This is a flowchart of the measurement visualization process according to the second embodiment. This is a data flow diagram of the video analysis device according to the second embodiment. This is a hardware configuration diagram of the video analysis device according to the second embodiment. This is a schematic diagram showing the video analysis device according to the third embodiment. This is a data flow diagram of the video analysis device according to the third embodiment. This is a schematic diagram showing the video analysis device according to the fourth embodiment. This is a flowchart of the measurement visualization process according to the fourth embodiment. This is a hardware configuration diagram showing an example of a computer that implements the functions of the imaging and video analysis device and the video analysis device according to the above embodiment.

The following describes an imaging analysis device in an embodiment for carrying out the present invention. The imaging analysis device generates a three-dimensional model based on three-dimensional point cloud information and images (video) captured by multiple LiDAR (Light Detection and Ranging) and cameras at the work site. This three-dimensional model is, for example, a textured three-dimensional mesh of heavy materials such as steel sheet piles and H-beams loaded on a truck. Next, the imaging analysis device compares the generated three-dimensional model (generated three-dimensional model) with a reference three-dimensional model (reference three-dimensional model) and outputs the difference. The difference is, for example, the difference in length in the longitudinal direction of the steel sheet pile, and is the length of the portion cut from the reference steel sheet pile. In the inspection of heavy materials such as steel sheet piles and H-beams, in addition to understanding the extent of damage, determining the length and quantity of the heavy materials is included in the inspection.

Traditionally, workers would stand on the back of a truck to measure the length of heavy materials and then inspect them. By using an imaging and analysis device, not only can the efficiency of the workers' inspection process be improved, but the safety of the workers can also be enhanced.

Furthermore, the imaging and analysis system can also be used to understand and monitor the progress of construction. By comparing the generated 3D model with a 3D model of a building constructed according to the construction plan (reference 3D model), managers can understand the differences and progress compared to the construction plan.

Thus, imaging and analysis devices can support the confirmation of conditions at the work site. Below, we will explain imaging and analysis devices using the inspection of heavy materials such as steel sheet piles and H-beams as an example.

<<Configuration of the imaging and analysis system>>
Figure 1 is a functional block diagram of the imaging analysis device 100 according to the first embodiment. Multiple LiDARs 411 and cameras 412 are connected to the imaging analysis device 100. Pairs of LiDARs 411 and cameras 412 are assumed to be installed close together. The LiDAR 411 transmits captured three-dimensional point cloud information to the imaging analysis device 100. The camera 412 transmits captured images (video) to the imaging analysis device 100. Hereinafter, the LiDARs 411 and cameras 412 will be collectively referred to as the imaging device. The three-dimensional point cloud information and images will also be collectively referred to as imaging information. The LiDARs 411 and cameras 412 may be an integrated imaging device. Furthermore, since the LiDARs 411 and cameras 412 have different imaging positions and fields of view, their correspondence may be calculated using the alignment method for both described in the second embodiment and subsequent embodiments.

The imaging and analysis device 100 is a computer comprising a control unit 110, a storage unit 120, and an input/output unit 180. User interface devices such as a display, keyboard, and mouse are connected to the input/output unit 180. The input/output unit 180 is equipped with a communication device, enabling data transmission and reception with the LiDAR 411 and camera 412. A media drive may also be connected to the input/output unit 180, enabling data exchange using a recording medium.

≪Imaging and Analysis Device: Memory Unit≫
The memory unit 120 is composed of memory devices such as ROM (Read Only Memory), RAM (Random Access Memory), and SSD (Solid State Drive). The memory unit 120 stores an imaging information database 130, an imaging device information database 140, an object information database 150, a reference three-dimensional model database 160, a recognition model 121, a generated three-dimensional model 122, and a program 128. The various contents of the memory unit 120 may be stored in an external storage device such as a cloud server and read as needed.

≪Storage Unit: Imaging Information Database / Imaging Device Information Database≫
The imaging information database 130 stores three-dimensional point cloud information and images captured by the imaging device, associated with the imaging device's identification information and the time of acquisition.
The imaging device information database 140 stores information such as identification information, type, specifications, installation location, and installation orientation related to the imaging device, LiDAR 411, and camera 412.

≪Memory Unit: Object Information Database / Reference 3D Model Database≫
The object information database 150 stores information about the object to be imaged (the object to be imaged). In the first embodiment, the object to be imaged is heavy material, and information such as the name, type, size, imaged image, and identification information of the three-dimensional model (reference three-dimensional model) of the reference heavy material is associated with it and stored in the object information database 150. "Reference" means that the object (heavy material) is in the same condition as a new product before cutting or before any damage or deformation occurs, and serves as a standard for comparison with the heavy material subject to inspection.

The reference three-dimensional model database 160 stores reference three-dimensional models, which are three-dimensional models of standard heavy materials, associated with identification information. These three-dimensional models are not necessarily generated based on three-dimensional point cloud information; for example, they may be three-dimensional models (three-dimensional CAD models) used as design information. Furthermore, when the imaging analysis device 100 is used for construction progress verification, the reference three-dimensional model database 160 may also store design three-dimensional data such as BIM (Building Information Modeling) or CIM (Construction Information Modeling).

≪Memory Unit: Cognitive Model≫
The recognition model 121 is a machine learning model that recognizes/identifies and detects the type of object being imaged based on the image. The image is not limited to an image captured by the camera 412, but may also be an image that is a texture of a three-dimensional model representing the object being imaged.

Examples of types include steel sheet piles and H-beams, but other heavy materials or even more specific types may also be used. Furthermore, the recognition model 121 can detect individual objects based on images showing the cross-section (vertical plane in the long direction) of the objects (heavy materials).

≪Memory Unit: Generating Three-Dimensional Model Program≫
The generated three-dimensional model 122 is a textured three-dimensional mesh of the imaged object (heavy material) generated based on the imaging information. The method for generating the generated three-dimensional model 122 will be described later.
Program 128 includes a description of the processing to be performed by the functional unit of the control unit 110, which will be described later.

<<Imaging and Analysis System: Control Unit>>
The control unit 110 includes a CPU (Central Processing Unit) and comprises an image acquisition unit 111, a three-dimensional model generation unit 112, a comparison unit 113, a display control unit 114, and an additional image acquisition support unit 115. The control unit 110 may also include a GPU (Graphics Processing Unit), an NPU (Neural (network) Processing Unit), an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), etc.

≪Control Unit: Image Acquisition Unit≫
The imaging acquisition unit 111 stores the three-dimensional point cloud information and images captured and transmitted by the LiDAR 411 and camera 412 in the imaging information database 130. The imaging acquisition unit 111 also stores the identification information of the imaging device and the time of acquisition in the imaging information database 130.

≪Control Unit: Three-Dimensional Model Generation Unit≫
The three-dimensional model generation unit 112 generates a three-dimensional model, which is a textured three-dimensional mesh, based on three-dimensional point cloud information and images in the imaging information database 130. This three-dimensional model is a generated three-dimensional model that represents an object (heavy material) at the site. More specifically, the three-dimensional model generation unit 112 refers to the position and orientation of the LiDAR 411 and generates a three-dimensional mesh based on multiple three-dimensional point cloud information. Three-dimensional mesh generation methods include the Alpha Shape method and the Poisson method. The three-dimensional mesh is composed of polygons and positions that represent the surface of the object to be imaged.

One method for combining multiple three-dimensional point cloud data is ICP (Iterative Closest Point), but the three-dimensional model generation unit 112 may generate a three-dimensional mesh using a method different from ICP. For example, the three-dimensional model generation unit 112 may extract feature quantities contained in each three-dimensional point cloud data and generate a three-dimensional mesh by combining the three-dimensional point cloud data by referencing the corresponding feature quantities. Examples of feature quantities include edges and sharp points.

Next, the 3D model generation unit 112 generates a UV map and texture by referring to the 3D mesh, image, and the position and orientation of the camera 412. The texture is the image applied to the 3D mesh. The UV map shows the correspondence between the polygons (pixels) that make up the 3D mesh and the texture (pixels). In this case, the correspondence of the UV map may be calculated based on the image captured during imaging. In this case, the output texture image is an image that has not been converted from the image captured during imaging.

The three-dimensional model generation unit 112 calculates the texture based on the image captured by the camera 412 when there is only one camera 412 that captures the surface of the object to be imaged, which corresponds to a polygon. More specifically, the three-dimensional model generation unit 112 calculates the pixels of the texture corresponding to the pixels of the polygon based on the image (pixels) captured by the camera 412.

The following describes the case where multiple cameras 412 capture images of the surface of an object corresponding to a polygon. The three-dimensional model generation unit 112 calculates the polygon's pixels (the texture pixels corresponding to those pixels) based on the image captured by the corresponding camera 412, and the distance between the pixel's position (the polygon's position) and the camera 412. For example, the three-dimensional model generation unit 112 uses the image from the camera 412 at the shortest distance. The three-dimensional model generation unit 112 may use the average value of pixels included in multiple images, or it may use a weighted average value where closer distances are given greater weight.

The 3D model generation unit 112 may calculate the texture based on the imaging angle rather than the distance between the polygon and the camera 412. The 3D model generation unit 112 uses the image captured by the camera 412 from the most frontal angle, where the angle between the line connecting the polygon and the camera 412 and the perpendicular line of the polygon is minimized, as the image. The 3D model generation unit 112 may also use an average value for the pixels, weighting more heavily on the angle between the line connecting the polygon and the camera 412 and the perpendicular line of the polygon.

The texture includes images captured by camera 412 and images calculated according to distance or angle. Alternatively, the texture may include images corresponding to a three-dimensional mesh (polygon) represented by a UV map.

As described above, the imaging analysis device 100 includes a three-dimensional model generation unit 112 that generates a generated three-dimensional model 122, which is a three-dimensional model of the object to be imaged, based on the imaging information.
The imaging information consists of three-dimensional point cloud information and images of the object being imaged.
The generated three-dimensional model 122 is a textured three-dimensional mesh, which is a three-dimensional mesh to which a texture has been applied.

The imaging information consists of multiple three-dimensional point cloud data and images of the target object, captured by multiple imaging devices.
The three-dimensional model generation unit 112 generates a three-dimensional mesh and texture by referring to the positions and orientations of multiple imaging devices (see imaging device information database 140), and also generates a UV map which represents the correspondence between the three-dimensional mesh and the texture.

The three-dimensional model generation unit 112 calculates the pixels included in the texture based on images captured by multiple imaging devices, according to the distance between the point on the object to be imaged corresponding to the pixel and the imaging device.
The three-dimensional model generation unit 112 calculates pixels based on the image captured by the imaging device with the smallest distance.
The three-dimensional model generation unit 112 calculates pixels based on images captured by an imaging device, which are weighted according to distance.
The three-dimensional model generation unit 112 takes the pixels included in the texture and uses the average of images captured by multiple imaging devices corresponding to points on the object to be imaged that correspond to those pixels.

The texture includes an image captured by the imaging device and an image composed of pixels calculated by the three-dimensional model generation unit 112.
Textures include images that correspond to UV maps.

≪Control Unit: Comparison Unit≫
The comparison unit 113 compares the generated three-dimensional model 122, which is a three-dimensional model generated by the three-dimensional model generation unit 112, with the reference three-dimensional model, which is a three-dimensional model in the reference three-dimensional model database 160, and calculates the difference. More specifically, the comparison unit 113 first detects the type of object to be imaged using the recognition model 121 based on the texture of the generated three-dimensional model 122 or the image from the camera 412. The comparison unit 113 may also detect the type of object to be imaged by comparing the size and shape of the reference three-dimensional model and the generated three-dimensional model 122. In this case, the length in the longitudinal direction of the object to be imaged (for example, a steel sheet pile) is compared taking into account the possibility that it may have been cut and shortened, or that it may have been damaged or deformed.

Furthermore, the reference 3D model may be one other than those stored in the reference 3D model database 160. Alternatively, the end faces and sides of materials may be detected from the video, and their size and shape may be measured using the generated 3D model based on that information. In cases where multiple types of materials are mixed, certain clusters (similar in shape or distance) may be detected based on the video and 3D information, and the above comparisons may be performed for each cluster. This cluster detection may be determined by using a threshold or similar method based on distance similarity using video recognition or 3D information.

Next, the comparison unit 113 uses the recognition model 121 to detect and count individual target objects based on the image from the camera 412 showing the cross-section (vertical plane in the longitudinal direction) of the target object, or the texture of the generated three-dimensional model 122. The comparison unit 113 may also count the number of target objects based on the volume of the generated three-dimensional model 122 and the volume of the reference three-dimensional model. Furthermore, the comparison unit 113 may obtain the portion of the generated three-dimensional model 122 occupied by each target object based on the count result.

The comparison unit 113 also calculates the size of the object to be imaged, particularly its length in the longitudinal direction, based on the generated three-dimensional model 122. Furthermore, the comparison unit 113 compares the shape of the generated three-dimensional model 122 with that of a reference three-dimensional model to detect any damage or deformation as a difference. The comparison unit 113 obtains identification information for the reference three-dimensional model corresponding to the detected type of object to be imaged by referring to the object information database 150. Subsequently, the comparison unit 113 may obtain the reference three-dimensional model corresponding to this identification information from the reference three-dimensional model database 160 and compare it with the generated three-dimensional model 122 to detect any difference in size (length). It is desirable to perform these processes on the parts of the generated three-dimensional model 122 corresponding to each target object.

As described above, the imaging analysis device 100 includes a comparison unit 113 that compares the generated three-dimensional model 122 with a reference three-dimensional model, which is a three-dimensional model of an object, and the generated three-dimensional model 122, and calculates the difference.
The comparison unit 113 counts the number of objects to be imaged.
The comparison unit 113 detects the type of object to be imaged based on the image or texture, compares the reference three-dimensional model and the generated three-dimensional model 122 corresponding to the detected type, and calculates the difference in at least one of the size and shape.

≪Control Unit: Display Control Unit≫
The display control unit 114 outputs the imaging analysis result screen 510 (see Figure 2 below) to a display connected to the input/output unit 180. Figure 2 is a screen configuration diagram of the imaging analysis result screen 510 according to the first embodiment. In the area 511 where the generated three-dimensional model 122 for which a type has been detected is displayed, comments 512 including the detected type, number, size (length), and damage/deformation status are added. In the area 513 where the generated three-dimensional model 122 for which a type has not been detected is displayed, comments 514 indicating that it was not detected are added. Areas with dark hatching are areas that are blind spots of the imaging device. Furthermore, when multiple materials or multiple types of materials are mixed, the detected materials and material units that could not be detected (units with similar shapes or close distances measured in three dimensions) may be displayed side by side as shown in Figure 2.

Note that the undetected region 513 is not necessarily a region of type undetected. For imaging target objects where the size (length) is not expected to change, if the comparison unit 113 determines that there is a size difference between the generated three-dimensional model 122 and the reference three-dimensional model, the display control unit 114 displays the differing portion as an undetected region.

By pressing the "Edit" button for comments 512 and 514, an editing screen for the comment content (not shown) is displayed, allowing the operator, who is a user of the imaging analysis device 100, to modify comments 512 and 514. By pressing the "Add Comment" button 515, the operator can add a region or a comment about that region. By pressing the "Add Image" button 516, the operator can acquire a suitable position and orientation for imaging the undetected region 513.

Figure 3 is a screen configuration diagram of the imaging analysis results screen 520, which displays a pointer 526 indicating the position and orientation of additional imaging according to the first embodiment. The operator takes images using LiDAR or a camera at the position and orientation indicated by the pointer 526 and sends the imaging information to the imaging analysis device 100. The imaging analysis device 100 regenerates the generated three-dimensional model 122, re-executes type detection and counting, and redisplays the imaging analysis results screen 530 (see Figure 4 below).

Figure 4 is a screen configuration diagram of the imaging analysis result screen 530 after additional imaging according to the first embodiment. The undetected region 513 (see Figure 2) is gone, and the steel sheet pile region 531 is correctly detected and displayed. The correct length is displayed in comment 532. By performing additional imaging in this way, the imaging analysis device 100 can reduce the undetected portion and display more accurate information.

≪Control Unit: Additional Imaging Support Unit≫
Returning to Figure 1, let's continue the explanation of the control unit 110. The additional imaging support unit 115 calculates a suitable position and orientation (see pointer 526 in Figure 3) for imaging the undetected region 513 (see Figure 2). The additional imaging support unit 115 calculates a position and orientation that allows the additional imaging location, which is the undetected region 513, to be imaged from the front, and displays it as pointer 526.

The additional imaging support unit 115 may also transmit its position and orientation to the imaging device to guide the worker who is performing the imaging. The imaging device should preferably be a device equipped with LiDAR, a camera, a satellite positioning system antenna, an angular velocity sensor, etc., such as a smartphone with LiDAR. Such an imaging device may also measure its own position and orientation and guide the worker so that imaging can be performed at the received position and orientation.

As described above, the imaging analysis device 100 includes an additional imaging support unit 115 that outputs a suitable position and orientation for imaging the additional imaging area, which is the part of the object to be imaged where a difference has occurred.

<<Image Analysis Processing>>
Figure 5 is a flowchart of the imaging analysis process according to the first embodiment. At the start of the imaging analysis process, it is assumed that imaging information has already been stored in the imaging information database 130.
In step S11, the three-dimensional model generation unit 112 generates a three-dimensional mesh representing the object to be imaged based on multiple three-dimensional point cloud information.

In step S12, the 3D model generation unit 112 generates a UV map and texture by referring to the 3D mesh, image, and the position and orientation of the camera 412. Once the 3D mesh, UV map, and texture are generated, the 3D model (generated 3D model) is created.
In step S13, the comparison unit 113 uses the recognition model 121 to detect the type of the object to be imaged (generated three-dimensional model).

In step S14, the comparison unit 113 uses the recognition model 121 to detect individual objects to be imaged and counts the number of objects to be imaged.
In step S15, the comparison unit 113 calculates the size of the object to be imaged based on the generated three-dimensional model 122.

In step S16, the comparison unit 113 compares the shapes of the generated three-dimensional model 122 and the reference three-dimensional model to detect damage or deformation.
In step S17, the display control unit 114 outputs the imaging analysis result screen 510 (see Figure 2) to the display.

In step S18, the display control unit 114 receives an operation from the user (operator). If the operation is an instruction to finish (step S18 → finish), the imaging analysis process is terminated. If the operation is the "correct" button pressed (step S18 → correct), the display control unit 114 proceeds to step S19. If the operation is the "add comment" button 515 pressed (step S18 → add comment), the display control unit 114 proceeds to step S20. If the operation is the "add image" button 516 pressed (step S18 → additional image), the display control unit 114 proceeds to step S21.

In step S19, the display control unit 114 modifies comments 512 and 514 according to the operator's instructions and returns to step S17.
In step S20, the display control unit 114 adds an area and a comment for that area according to the operator's instructions and returns to step S17.

In step S21, the additional imaging support unit 115 calculates a suitable position and orientation for imaging the undetected region 513, displays it as a pointer 526 (see Figure 3), and returns to step S11. In steps S11 and S12, the three-dimensional model generation unit 112 generates a new three-dimensional model including the additionally acquired imaging information.

As explained above, the three-dimensional model generation unit 112 generates a new three-dimensional model 122 based on the additional imaging information obtained from capturing additional imaging locations and the imaging information (see steps S21, S11, and S12).

Features of the imaging and analysis device
The imaging analysis device 100 generates a three-dimensional model based on multiple three-dimensional point cloud data and images of the object to be imaged, and calculates the type, size, and number of the object to be imaged. With such an imaging analysis device 100, it is possible to remotely identify the type of object to be imaged, count the number of objects, and check the damage status of heavy materials such as steel sheet piles, which can streamline inspection work, for example. Alternatively, it can reduce on-site work, thereby improving worker safety.

The imaging and analysis device 100 can also be used to check the progress of construction work. By comparing the three-dimensional model of the building under construction as a reference three-dimensional model with the generated three-dimensional model 122, the work manager can understand the differences from the construction plan and check the progress.

The imaging analysis device 100, in addition to measuring the size using a three-dimensional model generated based on the imaging information, calculates the difference in size/size and shape by comparing it with a reference three-dimensional model. Furthermore, the imaging analysis device 100 designates this difference as an undetected region. When additional imaging is instructed, the imaging analysis device 100 calculates the appropriate imaging position and orientation for additional imaging of the region, thereby supporting the additional imaging.

≪Variation: Three-dimensional point cloud information≫
In the first embodiment described above, the three-dimensional point cloud information is obtained from LiDAR 411. However, the three-dimensional point cloud information is not limited to this and may be generated/obtained using technologies such as artificial intelligence or stereo vision based on image information.

<<Variation Example: 3D Mesh Generation>>
While methods such as the Alpha Shape method and the Poisson method have been described as three-dimensional mesh generation techniques, other methods may also be used. Here, we assume that depth cameras are installed so that they have the same viewpoint as the LiDAR411. Alternatively, depth images can be generated from the three-dimensional point cloud for each LiDAR411 installed, and the three-dimensional mesh can be generated by integrating these depth images using a TSDF (Truncated Signed Distance Function) or the like.

<<Modification: Integration of three-dimensional information>>
While ICP was described as a method for combining multiple 3D point cloud data, other methods may also be used. Alternatively, a 3D mesh may be generated from the 3D point cloud data acquired by multiple LiDAR411s, and then a single 3D mesh may be generated by integrating these meshes. Alternatively, a single 3D mesh may be generated directly by integrating the 3D point cloud data acquired by each LiDAR411.

Furthermore, as another method, the imaging analysis device 100 generates textured three-dimensional meshes based on the three-dimensional point cloud information acquired by each LiDAR 411. The imaging analysis device 100 may also display multiple textured three-dimensional meshes simultaneously when viewing the textured three-dimensional model, making them appear as a single three-dimensional model.

<<Variation: Additional imaging>>
In the embodiment described above, the additional imaging support unit 115 calculates an imaging position and orientation suitable for additional imaging and displays it as a pointer 526 (see Figure 3). The additional imaging support unit 115 may also instruct the robot to perform additional imaging. The additional imaging support unit 115 may transmit the calculated imaging position and orientation to a self-propelled robot or drone equipped with an imaging device and instruct it to perform additional imaging. The robot transmits the additionally captured imaging information to the imaging analysis device 100.

<<Variation Example: Three-Dimensional Model Generator>>
The imaging analysis device may be a three-dimensional model generation device that does not include a comparison unit 113. In such an imaging analysis device, the three-dimensional model generation unit 112 generates a generated three-dimensional model 122 based on three-dimensional point cloud information and images captured from various directions. The three-dimensional model generation unit 112 generates textures based on multiple images, and has high color reproduction accuracy of the object being imaged. By viewing this generated three-dimensional model 122 from various viewpoints, the operator can grasp the condition of the object being imaged, including any damage.

≪Second Embodiment≫
The video analysis device according to the second embodiment measures the water level in real time solely by analyzing video from a surveillance camera. This eliminates the need to install meters for measuring the water level, eliminates the need for maintenance of water level measuring instruments and equipment, and allows for the cancellation of age-related errors in water level measurement through auto-calibration. Furthermore, it offers advantages in terms of cost and aesthetics, as well as the ability to measure the water level in real time.

Figure 6 is a schematic diagram showing the video analysis device 610 according to the second embodiment.
The video analysis device 610 is comprised of a video feature conversion unit 612, a video feature retention unit 613, a three-dimensional data feature conversion unit 615, a three-dimensional data feature retention unit 616, a spatial alignment unit 617, a measurement unit 618, and a display control unit 600.

The three-dimensional data 614 is obtained by pre-measuring an object using a three-dimensional measuring device such as LiDAR. The video data 611 is video data of the same object as the one captured by the LiDAR, and was captured using a two-dimensional camera. The video data 611 is, for example, a video stream with multiple video frames arranged chronologically, but it may also be a still image.

The three-dimensional data feature transformation unit 615 calculates three-dimensional features by performing perspective projection of the input three-dimensional data 614 from multiple viewpoints, assuming two-dimensional projection data. In other words, the three-dimensional data feature transformation unit 615 extracts features from three-dimensional data obtained by measuring an object with a three-dimensional measuring device, assuming two dimensions through perspective projection from multiple viewpoints. The three-dimensional data feature transformation unit 615 extracts boundary information of surfaces detected from normals included in the three-dimensional data of the measured object as the aforementioned features. Here, the object is, for example, a bridge pier, and boundary information of surfaces detected from normals included in the three-dimensional data of the bridge pier is extracted as features and used for water level measurement. Note that the boundary information of surfaces detected from normals included in the three-dimensional data of a structure is edge information of the structure's shape.

The three-dimensional data features calculated by the three-dimensional data feature conversion unit 615 are stored in the three-dimensional data feature storage unit 616. The three-dimensional data 614 only needs to be measured once initially; it is not necessary to continuously measure the object. Furthermore, if three-dimensional information already exists, that information may be used.

The video feature conversion unit 612 calculates video features by converting the features of the input video data 611. In other words, the video feature conversion unit 612 extracts the features of video data in which an object is captured by a camera. The video feature conversion unit 612 extracts edge information from the video data 611 in which the object is captured as features. Here, the object is, for example, a bridge pier. In water level measurement, the video feature conversion unit 612 extracts edge information from the video in which the bridge pier is captured as features. Furthermore, the video feature conversion unit 612 may use three-dimensional data 614 when extracting features from the video data 611 in which the object is captured. This allows the video feature conversion unit 612 to suitably extract the features of the bridge pier. The video features calculated by the video feature conversion unit 612 are stored in the video feature storage unit 613.
Furthermore, the video feature conversion unit 612 may extract features using multiple video frames from different time series. This allows the video feature conversion unit 612 to extract features regardless of the noise in individual frames. Additionally, the video feature conversion unit 612 may extract features based on the position information of the three-dimensional measuring device and the position information of the camera. This allows the video feature conversion unit 612 to extract features appropriately while considering how surfaces that can be suitably measured by the three-dimensional measuring device are captured by the camera.

The spatial alignment unit 617 compares the data in the video feature storage unit 613 and the data in the three-dimensional data feature storage unit 616 to determine the two-dimensional projection data that is closest to the video data 611. In other words, the spatial alignment unit 617 compares the feature quantities of the three-dimensional data stored in the three-dimensional data feature storage unit 616 with the feature quantities of the video data stored in the video feature storage unit to determine the two-dimensional projection data that has the closest field of view to the video data.
For this comparison, since the data in the video feature retention unit 613 may detect not only the edges of structures but also edges that appear on the image due to shadows, etc., it is best to compare only the feature portion of the three-dimensional data feature retention unit 616.
The spatial alignment unit 617 should perform a comparison at a predetermined height or higher based on the three-dimensional data 614. The predetermined height is, for example, the water surface.
Furthermore, for this comparison, it is best to first compare two-dimensional projection data at a general viewpoint or field of view to narrow down the possible viewpoints and fields of view, and then calculate the closest field of view hierarchically by making finer changes to the viewpoint and field of view.

The measurement unit 618 measures the size of an object in the video by comparing the feature quantities of the two-dimensional projection data determined by the spatial alignment unit 617 with the feature quantities of the video data stored by the video feature quantity storage unit.
The display control unit 600 displays the aligned video data and three-dimensional data, and superimposes the water levels indicating caution and danger onto the three-dimensional data. This allows the user to see whether the current water level is something to be cautious about or whether it is dangerous.

As a specific measurement method, the measurement unit 618 compares the waterline feature quantities in the video data with the feature quantities of the three-dimensional data stored in the three-dimensional data feature quantity storage unit 616, thereby converting the waterline information in the video into height information in the three-dimensional data.

Furthermore, the measurement unit 618 uses the feature quantities of the video data at a given time as a reference and compares the feature quantities of the video data at different times to measure the water level based on the difference. This is because, for example, when the water level of a river rises due to flooding, occlusion regions occur where structures such as bridges are obscured by water and are no longer detected as feature quantities in the video. If this is represented as a histogram of feature quantities for each water level, a large difference will occur in the histogram between the reference feature quantity and the feature quantity when the water level increases. The measurement unit 618 measures the increase in water level from the part where this difference occurs and converts it into a water level by adding the height to the reference value. In this case, when determining the water level, the measurement unit 618 may compare the histogram with the reference starting from the highest water level and select a point where the value has changed more than a certain threshold, or a point where the trend is significantly different from the surrounding heights. This makes it possible to calculate river water level information in real time based on video data captured by the surveillance camera.
Furthermore, the measuring unit 618 performs measurements in short time units to measure the water level, but it is not necessary to perform spatial alignment processing each time.

Figure 7 shows an example of video data of a bridge pier.
Bridge piers are installed in rivers, and the shape of the structure and the waterline are visible on the surface of the piers. By calculating the height information of the three-dimensional data of this shape and waterline, and comparing it with the features of reference video data, river water level information can be obtained. To obtain the shape information of this structure, it is preferable to detect boundary line (edge) information between different surfaces from the three-dimensional data based on the status of the normal vectors, which are vectors that represent the surface of the surface, and use this as a feature.

Figure 8 is a flowchart of the measurement visualization process according to the second embodiment.
First, the three-dimensional data feature conversion unit 615 acquires high-precision three-dimensional data 614 (step S40). Then, the three-dimensional data feature conversion unit 615 extracts boundary information of surfaces detected from the normals contained in the three-dimensional data 614 (step S41).
Next, the spatial alignment unit 617 associates the image features with the boundary information of the surfaces included in the three-dimensional data (step S42). The measurement unit 618 measures the difference in occlusion among the associated image features, where feature points appearing in the two-dimensional projection data are hidden by the video data, and creates a height-specific match amount histogram (step S43). By comparing this with the histogram at the reference time, the water level is determined (step S44), and the process shown in Figure 8 is completed.

Figure 9 is a data flow diagram of the video analysis device 610 according to the second embodiment.
During data collection, the LiDAR 710 performs three-dimensional measurement of the object and generates a point cloud 721. The video analysis device 610 then generates a mesh 722 and hands it over to the Raycast unit 730. The Raycast unit 730 corresponds to, for example, the Raycast of the game engine Unity (registered trademark), and performs a process that emits a transparent ray from a specific object and obtains the coordinates of another object that the ray hits.

When the camera is installed, the Raycast unit 730 determines the two-dimensional projection data closest to the image data when the camera captures the target object, based on camera parameters 731 indicating the position and direction of the camera that captures the image, from the objects represented by the mesh 722. This two-dimensional projection data is stored in the virtual normal image feature database 740. The virtual normal image feature database 740 further stores the LiDAR/camera transformation matrix 741 and Raycast resolution data 742.

During operation, the surveillance camera 750 captures the target object and generates an RGB image 751. The video analysis device 610 generates feature data 752 from the RGB image 751 and passes it to the feature matching unit 760 and the feature comparison unit 770.
The feature matching unit 760 compares the feature points of the two-dimensional projection data stored in the virtual normal image feature database 740 with the feature data 752. The feature matching unit 760 then outputs the best-matching normal image feature 761. The feature comparison unit 770 compares the feature data 752, the best-matching normal image feature 761, and the feature quantities at the reference time to generate a height-specific match amount histogram 771. The water level estimation unit 780 estimates the current water level by determining this height-specific match amount histogram 771.

Figure 10 is a diagram showing the specific hardware configuration of the video analysis device 610 according to the second embodiment.
The video analysis device 610 is, for example, a computer and incorporates a feature processing plugin 830 and a water level measurement plugin 840. The video analysis device 610 also stores an image capture information database 881, three-dimensional data 882, an image feature database 883, feature matching processing results 884, and analysis results 885.
The video analysis device 610 is connected to a surveillance camera 810, a video management device 820, a three-dimensional measurement device 850 such as LiDAR, and an initial data generation device 860.

The surveillance camera 810 captures video footage and hands it over to the video management device 820. The video management device 820 stores the video footage captured by the surveillance camera 810 and outputs image data to the feature processing plugin 830 and the water level measurement plugin 840.

The three-dimensional measuring device 850 is, for example, a LiDAR, which measures the object in three dimensions, writes the shooting location ID to the shooting information database 881, and writes the mesh data and shooting location ID to the three-dimensional data 882.
The initial data generation device 860 includes a virtual normal image generation unit 861 and a feature extraction unit 862. The virtual normal image generation unit 861 generates a normal image from the shooting location ID and mesh data. The feature extraction unit 862 generates a feature image and a depth image from the normal image. The initial data generation device 860 stores the normal image, feature image, and depth image in the image feature database 883.

The feature processing plugin 830 comprises a feature extraction unit 831 and a feature matching unit 832. The feature extraction unit 831 extracts features from image data and outputs them to the feature matching unit 832. The feature matching unit 832 receives the features of the image data as input, along with the shooting location ID and camera parameters from the shooting information database 881.

The feature matching unit 832 passes the shooting location ID to the image feature database 883 and reads the feature image corresponding to the shooting location ID. The feature matching unit 832 matches the features of the image data with the feature image to select the normal image ID. The feature matching unit 832 stores the normal image ID and the shooting location ID in the feature matching processing result 884.

The water level measurement plug-in 840 includes a feature extraction unit 841 and a water level measurement unit 842. Image data is input to the feature extraction unit 841 from the video management device 820. The feature extraction unit 841 extracts feature points from the normal image ID and image data and outputs them to the water level measurement unit 842. The water level measurement unit 842 measures the water level from the feature points, the normal image ID, the feature image, and the depth image. The water level measurement unit 842 stores the shooting location ID, shooting time, image data, and water level in the analysis results 885.
The analysis result display device 870 acquires and displays the water level and other information from the analysis results 885. This makes it possible to measure the water level from the video feed of the surveillance camera 810. In this embodiment, three-dimensional data was acquired in advance using LiDAR, but methods that utilize already acquired three-dimensional information may also be used.

≪Third Embodiment≫
The video analysis device according to the third embodiment creates an overhead view of a construction site or similar location from a free viewpoint without distortion. This makes it easy to grasp the progress of construction work.

Figure 11 is a schematic diagram showing the video analysis device 610A according to the third embodiment.
The video analysis device 610A is composed of a video feature conversion unit 612, a video feature retention unit 613, a three-dimensional data feature conversion unit 615, a three-dimensional data feature retention unit 616, a spatial alignment unit 617, and a three-dimensional data generation unit 619.

The three-dimensional data 614 is obtained by measuring objects, such as those at a construction site, using a three-dimensional measuring device such as LiDAR. The video data 611 is video footage of the same objects as those captured by LiDAR, but this footage was taken with a two-dimensional camera.

The three-dimensional data feature transformation unit 615 calculates three-dimensional data features by performing perspective projections of the input three-dimensional data 614 from multiple viewpoints, assuming two-dimensional projected data. The three-dimensional data features calculated by the three-dimensional data feature transformation unit 615 are stored in the three-dimensional data feature storage unit 616. The three-dimensional data 614 is acquired each time the construction progresses.

The video feature conversion unit 612 calculates video features by converting the features of the input video data 611. The video features calculated by the video feature conversion unit 612 are stored in the video feature storage unit 613.

The spatial alignment unit 617 compares the data from the video feature storage unit 613 and the three-dimensional data feature storage unit 616 to determine the two-dimensional projection data that is closest to the video data 611.
The three-dimensional data generation unit 619 generates texture-mapped three-dimensional data by mapping the texture of the video data onto the three-dimensional data based on the two-dimensional projection data determined by the spatial alignment unit 617. In other words, the three-dimensional data generation unit 619 associates video data with mesh information generated from the three-dimensional data corresponding to the two-dimensional projection data determined by the spatial alignment unit 617. By rendering this three-dimensional data with, for example, a game engine such as Unity (registered trademark), it becomes possible to view the object from a free viewpoint.

Figure 12 is a data flow diagram of a modified image analysis device 620.
The video analysis device 620 includes a texture extraction unit 622, a mesh data conversion unit 625, and a viewpoint transformation rendering unit 626, and is connected to multiple cameras 621a, 621b, a long-range LiDAR 623, and a portable LiDAR 624.

Multiple cameras 621a and 621b are positioned at different viewpoints, capturing images of the object from multiple perspectives. The long-range LiDAR 623 measures the object's three-dimensional data from a point at a long distance. The portable LiDAR 624 measures the object's three-dimensional data from a point in close proximity.

The mesh data generation unit 625 integrates the three-dimensional data of the object and converts it into mesh data. The mesh data generated by the mesh data generation unit 625 is output to the texture extraction unit 622.

The texture extraction unit 622 receives video input from multiple cameras 621a and 621b. The texture extraction unit 622 then converts the three-dimensional mesh data into two-dimensional projection data as viewed from each viewpoint of cameras 621a and 621b. It then associates each two-dimensional projection data with each video image and extracts textures from the video to be applied to the mesh data. In this way, the texture extraction unit 622 creates mesh data with the applied textures.

The viewpoint transformation rendering unit 626 is, for example, the game engine Unity®, and renders textured mesh data from an arbitrary viewpoint. This makes it possible to observe an object, such as a construction site, from a desired viewpoint.
In this embodiment, three-dimensional information is captured sequentially using LiDAR, but methods that utilize already acquired three-dimensional information may also be used. Examples of already acquired three-dimensional information include BIM (Building Information Modeling) and CIM (Construction Information Modeling).

≪Fourth Embodiment≫
The video analysis device according to the fourth embodiment measures the length and volume of heavy temporary construction materials on the spot simply by taking a picture with a mobile device such as a smartphone, based on three-dimensional data acquired by LiDAR. The video analysis device grasps the condition of the material bundle from the video captured by the mobile device and performs matching processing with the length analysis information of the cross section. As a result, for example, it is possible to measure a material with a length of 20m from a distance of 20m and classify the length of the material in 10cm increments in real time.

The video analysis device according to the fourth embodiment enables the measurement of both length and volume of cargo on a truck in a single step. This allows for safe and efficient execution of tasks that were previously performed manually, one item at a time, by climbing onto the truck bed.

Figure 13 is a schematic diagram showing the video analysis device 610B according to the fourth embodiment.
The video analysis device 610B is comprised of a video feature conversion unit 612, a video feature retention unit 613, a three-dimensional data feature conversion unit 615, a three-dimensional data feature retention unit 616, a video recognition unit 601, and a measurement unit 602.

The three-dimensional data 614 is obtained by pre-measuring an object using a three-dimensional measuring device such as LiDAR. The video data 611 is an image of the same object as the one captured by the LiDAR, but it was captured using a two-dimensional camera.

The three-dimensional data feature transformation unit 615 calculates three-dimensional data features by performing perspective projections of the input three-dimensional data 614 from multiple viewpoints, assuming two-dimensional projected data. The three-dimensional data features calculated by the three-dimensional data feature transformation unit 615 are stored in the three-dimensional data feature storage unit 616. The three-dimensional data 614 only needs to be measured once initially; it is not necessary to continuously measure the object.

The video feature conversion unit 612 calculates video features by converting the features of the input video data 611. The video features calculated by the video feature conversion unit 612 are stored in the video feature storage unit 613.

The image recognition unit 601 recognizes objects from the image data 611, determines whether it is a bundle of materials, and, if objects are overlapping, recognizes the number of individual pieces or items. Then, the image recognition unit 601 compares the feature data from the image feature storage unit 613 with the feature data from the three-dimensional data feature storage unit 616 to determine the two-dimensional projection data closest to the image data 611, and recognizes the three-dimensional data corresponding to the recognized object.

The measurement unit 602 measures the size of the object based on the range of the three-dimensional data of the object recognized by the image recognition unit 601.

Specifically, this size measurement involves measuring the length, width, and height based on the three-dimensional data of the recognized object (for example, a bundle of materials), calculating the volume from that data, and detecting the number and shape of individual materials contained within the bundle from the video, thereby measuring the length of each individual material.

Figure 14 is a flowchart of the measurement visualization process according to the fourth embodiment.
The user first measures three-dimensional data from the side of the material using LiDAR (step S50). Next, the user takes pictures of the cross-section and outline of the material with a camera (step S51). The measured three-dimensional data and the captured video data are input to the video analysis device 610B.
The video recognition unit 601 of the video analysis device 610B recognizes the bundle of materials from the video data 611 (step S52).

The video recognition unit 601 of the video analysis device 610B performs a matching process between the video data 611 and the three-dimensional data 614 to understand the status of the material bundle (step S53). This makes it possible to recognize the portion of the three-dimensional data 614 that represents the material bundle.
Furthermore, the video recognition unit 601 of the video analysis device 610B analyzes the number of target materials from their cross-sectional shape (step S54). After that, the measurement unit 602 of the video analysis device 610B analyzes the length of the materials in the longitudinal direction (step S55), and the process in Figure 14 is completed.

<<Other variations>>
Although several embodiments and modifications of the present invention have been described above, these embodiments are merely illustrative and do not limit the technical scope of the present invention. The present invention can take on a variety of other embodiments, and furthermore, various modifications such as omissions and substitutions can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention as described herein and elsewhere, as well as in the scope of the invention and its equivalents as described in the claims.

Hardware Configuration
The imaging analysis device 100 and video analysis devices 610, 610A, 610B, and 620 according to the above-described embodiment are implemented by a computer 900 having a configuration such as that shown in Figure 15. Figure 15 is a hardware configuration diagram showing an example of a computer 900 that implements the functions of the imaging analysis device 100 and video analysis devices 610, 610A, 610B, and 620 according to the above-described embodiment. The computer 900 includes a CPU 901, ROM 902, RAM 903, SSD 904, input/output interface 905 (labeled as input/output I/F (Interface) in Figure 15), communication interface 906 (labeled as communication I/F in Figure 15), and media interface 907 (labeled as media I/F in Figure 15). The computer 900 may be equipped with an HDD (Hard Disc Drive) instead of the SSD 904, or it may be equipped with an HDD in addition to the SSD 904.

The CPU 901 operates based on programs stored in the ROM 902 or SSD 904, and is controlled by the control unit 110 shown in Figure 1. The ROM 902 stores boot programs executed by the CPU 901 when the computer 900 starts up, as well as programs related to the computer 900's hardware.

The CPU 901 controls input devices 910, such as a mouse and keyboard, and output devices 911, such as a display and printer, via the input/output interface 905. The CPU 901 acquires data from the input devices 910 and outputs the generated data to the output devices 911 via the input/output interface 905.

The SSD 904 stores programs executed by the CPU 901 and data used by those programs. The communication interface 906 receives data from other devices (e.g., LiDAR 411, 710 and cameras 412, 612a, 612b) via the communication network and outputs it to the CPU 901. It also transmits data generated by the CPU 901 to other devices via the communication network.

The media interface 907 reads programs or data stored in the recording medium 912 and outputs them to the CPU 901 via the RAM 903. The CPU 901 loads the program from the recording medium 912 onto the RAM 903 via the media interface 907 and executes the loaded program. The recording medium 912 can be an optical recording medium such as a DVD (Digital Versatile Disk), a magneto-optical recording medium such as an MO (Magneto Optical Disk), a magnetic recording medium, a conductive memory tape medium, or a semiconductor memory.

For example, when the computer 900 functions as the imaging analysis device 100 or video analysis devices 610, 610A, 610B, and 620 according to the above-described embodiment, the CPU 901 of the computer 900 realizes the functions of the imaging analysis device 100 or video analysis devices 610, 610A, 610B, and 620 by executing the program 128 (see Figure 1) loaded onto the RAM 903. The CPU 901 reads and executes the program from the recording medium 912. Alternatively, the CPU 901 may read the program from another device via a communication network, or it may install and execute the program 128 from the recording medium 912 onto the SSD 904.

100 Imaging and analysis device (video analysis device)
111 Image acquisition unit 112 Three-dimensional model generation unit 113 Comparison unit 114 Display control unit 115 Additional imaging support unit 121 Recognition model 122 Generated three-dimensional model 128 Program 130 Imaging information database 140 Imaging device information database 150 Object information database 160 Reference three-dimensional model database (reference three-dimensional model)
510, 520, 530 Image analysis result screen 610, 610A, 610B Image analysis device (image analysis device)
611 Video data 612 Video feature conversion unit 613 Video feature retention unit 614 Three-dimensional data 615 Three-dimensional data feature conversion unit 616 Three-dimensional data feature retention unit 617 Spatial alignment unit 618 Measurement unit 620 Video analysis device 622 Texture extraction unit 625 Mesh data conversion unit 626 Viewpoint conversion rendering unit 621a Camera 621b Camera 623 Long-range LiDAR
624 Portable LiDAR
710 LiDAR
619 Three-dimensional data generation unit 721 Point cloud 722 Mesh 730 Raycast unit 731 Camera parameters 740 Virtual normal image feature database 741 LiDAR/camera transformation matrix 742 Raycast resolution data 750 Surveillance camera 751 RGB image 752 Feature data 760 Feature matching unit 770 Feature comparison unit 761 Normal image features 771 Height-specific match amount histogram 780 Water level estimation unit 830 Feature processing plugin 840 Water level measurement plugin 881 Shooting information database 882 Three-dimensional data 883 Image feature database 810 Surveillance camera 820 Video management device 850 Three-dimensional measurement device 860 Initial data generation device 861 Virtual normal image generation unit 862 Feature quantity extraction unit 831 Feature extraction unit 832 Feature matching unit 841 Feature extraction unit 842 Water level measurement unit 885 Analysis result 870 Analysis result display device

Claims (13)

A three-dimensional model generation unit generates a three-dimensional model of the object being imaged based on imaging information,
A comparison unit compares a reference three-dimensional model, which is a three-dimensional model of an object to be compared with the generated three-dimensional model and is also a three-dimensional model of an object to be compared with the object to be imaged, with the generated three-dimensional model and calculates the difference.
An imaging analysis device comprising: an additional imaging support unit that outputs a position and orientation suitable for imaging the additional imaging location, which is the part of the object to be imaged in which the difference occurs. The three-dimensional model generation unit,
The imaging analysis apparatus according to claim 1, which generates a new three-dimensional model based on the additional imaging information obtained by imaging the additional imaging location and the imaging information. The aforementioned imaging information is,
Multiple three-dimensional point cloud information and images of the object to be imaged, captured by multiple imaging devices,
The generated three-dimensional model is,
A textured 3D mesh is a 3D mesh with a texture applied to it.
The three-dimensional model generation unit,
Referencing the positions and orientations of the multiple imaging devices,
The imaging and analysis apparatus according to claim 1, which generates the three-dimensional mesh and the texture, and generates a UV map that represents the correspondence between the three-dimensional mesh and the texture. The three-dimensional model generation unit,
The pixels included in the aforementioned texture
The imaging analysis device according to claim 3, which calculates based on images captured by a plurality of imaging devices, according to the distance between a point on the object to be imaged corresponding to a pixel and the imaging device. The three-dimensional model generation unit,
The imaging analysis apparatus according to claim 4, wherein the pixels are calculated based on an image captured by the imaging device with the minimum distance. The three-dimensional model generation unit,
The imaging analysis apparatus according to claim 4, wherein the pixels are calculated based on an image captured by the imaging device, weighted according to the distance. The three-dimensional model generation unit,
The pixels included in the aforementioned texture
The imaging analysis device according to claim 3, which takes the average of images captured by a plurality of imaging devices corresponding to a point in the object to be imaged that corresponds to the pixel in question. The aforementioned texture is
The imaging analysis apparatus according to claim 4 or 7, comprising an image captured by the imaging device and an image composed of pixels calculated by the three-dimensional model generation unit. The aforementioned texture is
The imaging and analysis apparatus according to claim 3, which includes an image corresponding to the UV map. The aforementioned imaging information is,
The three-dimensional point cloud information and image of the object to be imaged are as follows:
The generated three-dimensional model is,
A textured 3D mesh is a 3D mesh with a texture applied to it.
The comparison unit is,
The type of the object to be imaged is detected based on the image or texture.
The imaging analysis apparatus according to claim 1, which compares the reference three-dimensional model and the generated three-dimensional model corresponding to the type of detection result and calculates the difference in at least one of size and shape. The comparison unit is,
The imaging analysis apparatus according to claim 10, which counts the number of objects to be imaged. Computers,
A three-dimensional model generation unit generates a three-dimensional model of the object being imaged based on imaging information,
A comparison unit compares a reference three-dimensional model, which is a three-dimensional model of an object to be compared with the generated three-dimensional model and is also a three-dimensional model of an object to be compared with the object to be imaged, with the generated three-dimensional model and calculates the difference.
A program for functioning as an imaging analysis device, comprising: an additional imaging support unit that outputs a suitable position and orientation for imaging the additional imaging location, which is the part of the object to be imaged where the difference occurs. The imaging and analysis device,
The steps include generating a three-dimensional model, which is a three-dimensional model of the object being imaged, based on the imaging information, and
A step of comparing the generated three-dimensional model with a reference three-dimensional model, which is a three-dimensional model of an object that is to be compared with the object to be imaged, and the generated three-dimensional model, and calculating the difference.
An imaging analysis method comprising the steps of: outputting a suitable position and orientation for imaging the additional imaging location, which is the part of the object to be imaged where the difference occurs.