JP7112469B2 - Image processing device, image processing method, image processing program - Google Patents

Description

The present invention relates to image processing technology for point cloud data.

Point cloud data obtained by a laser scanner or stereo photo measurement is known (see Patent Documents 1 and 2, for example).

International Publication No. WO2011/070927 JP 2012-230594 A

When the point cloud data is displayed in 3D on a display such as a PC, the 3D shape of the object to be measured is displayed by the point cloud of the object. A technique for obtaining a three-dimensional model from point cloud data is one of the main methods of utilizing point cloud data. Since the three-dimensional model expresses the outline of the object as data, it has a high affinity with three-dimensional data handled on CAD software.

When the point cloud data is displayed as an image as it is, the object is displayed in 3D by a set of points. However, at the stage of raw data that has not been converted into a three-dimensional model, it is difficult to grasp the outline of the object, and it is difficult to grasp at a glance which part is the edge of the object (for example, a building). hard. Although some measures have been taken to make it easier to grasp the three-dimensional effect of the 3D display of the point group, such as by changing the color of the points depending on the orientation of the surface, there is still the above problem. Furthermore, in the case of raw data that has not undergone any processing, the above problem becomes apparent.

A specific example will be described. FIG. 2A shows an example of 3D display (perspective view display) of point cloud data of a cubic object. As shown in FIG. 2A, when the image is simply displayed as dots, it is difficult to grasp the perspective and it is difficult to determine which part is the edge of the cube.

FIG. 2(B) shows an example in which a three-dimensional model is created, the cubic edge portion is extracted, and the edge portion is superimposed on the point cloud data and displayed. In this case, it is easy to grasp the stereoscopic effect. However, FIG. 2(B) is a display obtained after processing for three-dimensional modeling. ) cannot be displayed.

In view of such a background, an object of the present invention is to provide a technology that facilitates understanding of the three-dimensional structure of an object from point cloud data displayed in 3D.

The invention according to claim 1 comprises a point cloud data display control unit that displays each point of point cloud data in 3D on a display, and a surface equation related to a surface that constitutes an object from which the point cloud data is obtained. and a normal to the surface based on the equation of the surface ; and a clipping range display control unit for displaying a clipping range display on the display, wherein the calculation of the equation of the surface is performed by obtaining points of the specified portion and fitting the points and a plurality of points around the points. wherein the clipping range display control unit matches the direction of the normal line of the clipping range display with the direction of the normal line of the surface calculated by the surface direction calculation unit. be.

According to a second aspect of the invention, there is provided a point cloud data display control step of displaying each point of the point cloud data in 3D on a display; and a surface direction calculation step of calculating a normal to the surface based on the surface equation, wherein a thick rectangle representing a flat portion cut from the 3D displayed point cloud data The clipping range display control step of displaying a clipping range display having a shape on the display, and the calculation of the equation of the surface are obtained by obtaining a point of the specified portion and fitting to the point and a plurality of points around it. In the clipping range display control step, the direction of the normal line of the clipping range display is matched with the direction of the normal line of the plane calculated in the plane direction calculation step . image processing method.

The invention according to claim 3 is a program to be read and executed by a computer, comprising: a point cloud data display control unit for performing 3D display of each point of point cloud data on a display; and a surface direction calculation unit that calculates a surface equation related to a surface that constitutes the acquired object, and calculates a normal line of the surface based on the surface equation, and cut from the 3D displayed point cloud data A clipping range display having a thick rectangular shape that displays a flat portion to be displayed is operated as a clipping range display control unit that displays on the display, and the calculation of the equation of the surface is performed by acquiring the points of the specified portion , and a surface equation that fits the point and a plurality of points around it, and the clipping range display control unit determines the direction of the normal line of the clipping range display from the above calculated by the plane direction calculation unit. It is an image processing program that matches the direction of the normal line of the surface.

According to the present invention, it is possible to obtain a technique that makes it easy to grasp the three-dimensional structure of an object from point cloud data displayed in 3D.

1 is a block diagram of an embodiment; FIG. FIG. 4 is a conceptual diagram showing an example of point cloud data displayed on a screen; FIG. 4 is a conceptual diagram showing an example of point cloud data displayed on a screen; FIG. 4 is a conceptual diagram showing an example of point cloud data displayed on a screen; It is a drawing substitute photograph of the 3D display of the processed point cloud data. 6 is a 3D display (intensity display) of the point cloud data of FIG. 5 and its partial cross-sectional display. It is a drawing substitute photograph of 3D display of point cloud data. It is a drawing substitute photograph of 3D display of point cloud data. It is a drawing substitute photograph of 3D display of point cloud data. It is a drawing substitute photograph of 3D display of point cloud data. It is a drawing substitute photograph of 3D display of point cloud data. It is a drawing substitute photograph of 3D display of point cloud data.

FIG. 1 shows an image processing device 100 . The image processing device 100 is a device for processing point cloud data. Normally, application software that realizes the functions of the image processing apparatus 100 is installed on a PC (personal computer) instead of dedicated hardware, and the application software is started, whereby the image processing apparatus 100 is realized in software by the PC. be done.

When a PC is used, each functional unit illustrated in FIG. 1 is configured by software. Note that each functional unit shown in FIG. 1 may be configured by a dedicated arithmetic circuit. Also, a functional unit configured in software and a functional unit configured by a dedicated arithmetic circuit may coexist. For example, each functional unit illustrated is configured by an electronic circuit such as a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), and a PLD (Programmable Logic Device) such as an FPGA (Field Programmable Gate Array).

Whether each functional unit is configured by dedicated hardware or configured by software by executing a program in the CPU is determined in consideration of required calculation speed, cost, power consumption, and the like. For example, if a specific functional unit is configured with an FPGA, it is superior in terms of processing speed, but the cost is high. On the other hand, a configuration in which a specific functional unit is realized by executing a program on a CPU can save hardware resources, and is advantageous in terms of cost. However, when the functional unit is implemented by a CPU, the processing speed is inferior to dedicated hardware. It should be noted that configuring the functional units with dedicated hardware and configuring them with software are equivalent from the viewpoint of realizing a specific function, although there are differences as described above.

The image processing apparatus 100 includes a point cloud data acquisition unit 101 , a point cloud data display control unit 102 , a marker display control unit 103 , a specified point cloud extraction unit 104 and a specified point cloud display control unit 105 . The point cloud data acquisition unit 101 acquires point cloud data measured by the three-dimensional laser scanner from the three-dimensional laser scanner via an appropriate communication means. Point cloud data can also be obtained by a method of obtaining three-dimensional coordinates of a large number of characteristic points extracted from a photographed image using the principle of stereo photogrammetry.

A point cloud data display control unit 102 displays the point cloud data on an appropriate image display device such as a PC display. The point cloud data may be displayed as it is as a display of points, or may be displayed after performing various image processing such as changing the display color and removing noise components.

The marker display control unit 103 controls marker display for cutting the 3D-displayed point cloud data on a specific plane. FIG. 3 shows point cloud data 150 of a cubic measurement object (assuming a building) and markers 200, which are an example of the marker display described above.

The marker 200 can be moved in the X direction (specific horizontal direction) by an operator's operation (for example, mouse operation). By moving the marker 200, it is possible to designate a position (position on the X axis) at which the 3D-displayed point cloud data is cut on the YZ plane (Z is the vertical direction, Y is the horizontal direction perpendicular to the X direction).

Aspects in FIG. 3 may be set such that the X direction is the north-south direction (or the east-west direction), the Y direction is the east-west direction (or the north-south direction), and the Z direction is the vertical direction. In this case, it is easy to grasp the relationship between the cut surface and the orientation. Further, when the object of the point cloud data 150 is a building, it is possible to set the directions of the XYZ axes in accordance with the extending directions of the walls of the building. FIG. 3 shows an example of this case.

FIG. 4 shows an example of sub-screens simultaneously displayed on the display separately from FIG. The sub-screen may be displayed on a display separate from the main screen, or may be displayed by switching to the main screen. When the marker 200 in FIG. 3 is moved on the X axis and the YZ plane including the marker 200 overlaps the point cloud data 150, the overlapped portion (point cloud data cut off on the YZ plane) is shown in FIG. is displayed as FIG. 4A shows planar point cloud data 151 obtained by cutting the edge portion on the X axis of the cubic point cloud data 150 in FIG. 3 along the YZ plane. in this case. The front end face on the X axis of the point cloud data 150 is cut off by the YZ plane, and this cut out planar portion is displayed.

When the marker 200 in FIG. 3 is further moved in the positive X-axis direction from the state in FIG. An example of this case is shown in FIG. In this case, since the point cloud inside the cubic object on which the point cloud data 150 is based has not been acquired, a frame-shaped point cloud is obtained as the point cloud of the cross section.

Processing for cutting the point cloud data 150 in FIG. 3 on the YZ plane and extracting the point cloud data 151 and 152 in FIG. 4 is performed as follows. First, the three-dimensional coordinate position of each point forming the point cloud data 150 is known. On the other hand, the coordinates of the YZ plane specified by the marker 200 can be specified by the surface equations in the coordinate system describing the point cloud data 150 . Therefore, it is possible to extract a point group that intersects (or is close to) the YZ plane designated by the marker 200 in the point group data 150 . Strictly speaking, since the point group is distributed discretely, a threshold value is set, and points close to the YZ plane are extracted within the range of this threshold value.

As shown in FIGS. 4A and 4B, the point cloud data displayed in 3D using the marker 200 is cut, and the point cloud existing on the cutting plane is displayed on a separate screen. It becomes easy to grasp the three-dimensional arrangement of data. For example, by moving the marker 200 in the X-axis direction, edge portions of the point cloud data 150 cut on the YZ plane of the point cloud data 150 are displayed one after another. It becomes easier to grasp the three-dimensional shape (edge portion) of the measurement target on which the data 150 is based.

The moving direction of the marker 200 can be set as desired by the operator. Further, for example, by automatically moving the marker 200 and displaying the point cloud display of the cutting planes appearing one after another as a plurality of thumbnail images arranged on the time axis, it is possible to easily grasp the three-dimensional distribution of the point cloud data. UI display is also possible.

In the above example, the YZ plane used for cutting has no thickness, but the YZ plane used for cutting may have thickness. In this case, the point group is cut out in the X-axis direction as well.

The designated point cloud extraction unit 104 extracts point cloud data on a cutting plane obtained by cutting the point cloud data along a specific plane. For example, the designated point cloud extracting unit 104 performs processing for extracting the point cloud data 151 and 152 in FIG. 4 from the 3D-displayed point cloud data 150 in FIG.

The specified point cloud display control unit 105 controls the screen display of the point cloud data extracted by the specified point cloud extraction unit 104 . For example, the display control shown in FIG. 4 is performed by the specified point group display control unit 105 .

As described above, the image processing apparatus 100 includes the point cloud data display control unit 102 for displaying the point cloud data in 3D, and the marker display for designating the cutting plane of the point cloud data displayed in 3D. It is provided with a control unit 103 and a specified point cloud display control unit 105 for displaying a point cloud existing on the cutting plane in the point cloud data on a separate screen.

According to this configuration, since the edge portion of the object to be measured is extracted and displayed, it becomes easy to grasp the three-dimensional distribution state of the point cloud data. For example, with point cloud data for a building, it is easy to grasp the three-dimensional shape of the building without creating a three-dimensional model.

FIG. 5 shows a 3D representation of the processed point cloud data of the object to be measured. The point cloud data in FIG. 5 are processed by coloring the points so that they can be easily viewed as a point cloud image. This processing is performed by superimposing a photographic image taken by a camera at the same time as obtaining the point cloud data and the point cloud data, and coloring the points in the corresponding area based on the photographic image.

The left side of FIG. 6 shows a point cloud image in which the point cloud data on which FIG. 5 is based is displayed with shading corresponding to the intensity of the reflected light, and the right side of FIG. 6 shows the left screen. A point group of a cross-sectional portion cut by a belt-like line is shown as viewed from a direction perpendicular to the cut surface. As shown in FIG. 6, a part of the point cloud data displayed in 3D is cut (see FIG. 6A), and the point cloud of the cut plane is displayed in a state seen from a direction perpendicular to the cut plane (see FIG. 6A). 6(B)) makes it easier to grasp the outline of the object. Also, the width of the line (the line that determines the cutting plane) shown in the point cloud image on the left side of FIG. 6 can be selected. Points can be extracted from regions.

In addition, a “cutting range display”, which is a marker having a function different from that of the marker 200, is also displayed. “Cut-out range display” is a marker that indicates the range of a cut-out portion (cut-off portion) of the 3D-displayed point cloud data. The marker display control unit 103 also controls the display of the “cutting range display”. FIG. 7 shows an example in which a slightly thick flat rectangular "cutting range display" is displayed on the point group display screen. Either one of the display of the marker 200 and the "display of the clipping range" can be displayed, or both can be displayed at the same time. In addition, FIG. 7 is processed so that it can be easily seen as a point cloud image by adding shading to the points. This processing is performed by superimposing a photographic image taken by a camera at the same time as obtaining the point cloud data and the point cloud data, and adding shading to the points in the corresponding area based on the photographic image.

As shown in FIG. 7, the display of "cutting range display" is performed on the screen on which the entire point cloud data is displayed. By doing so, it becomes easier to visually grasp the portion to be cut out. The position and size of the "cut display range" can be specified. For example, when the mouse is left-clicked on the "cut display range" in FIG. 7, the position is moved. You can also change the size by right-clicking the "Clip display range" and moving the mouse. The size change is independently possible in three orthogonal axial directions. These controls are also performed by the marker display control unit 103 .

Also, FIG. 7 shows an example in which the cut portion is displayed in color. By displaying the entire image in monochrome gradation and displaying the cut-out portion in color, it becomes easier to grasp the cut-out portion in three dimensions. FIG. 8 shows a screen displaying only the portion cut out in the "cutting range display" of FIG.

You can also rotate the "Clip Area Display". Since the three-dimensional coordinates of each point in the point cloud data are known, by rotating the "cutting range display", the cutout point changes, and the display of the cutout part (for example, the display in FIG. 8) also changes. .

A mode is also possible in which the "cutting range display" is displayed in correspondence with the direction of the surface of the object. An example of this is shown in FIG. The example of FIG. 11 shows a case where the normal line of the surface of the object and the normal line of the surface of the "clipping range display" are made to match to perform the "clipping range display". There are many cases where point cloud data is handled by focusing on the surface of an object. In such a case, it is convenient to display the "cutting range display" according to the surface of the object.

The process of aligning the orientation (orientation of the normal line) of the "clip area display" with the orientation (orientation of the normal line) of the surface of the object is performed as follows. This processing is performed by the plane direction calculation unit 106 . In this process, first, in the state of FIG. 9, a portion of the object to be displayed in the "cutting range display" that matches the normal line of the surface of the object is designated. In the example of FIG. 9, for example, the right wall portion of the building is designated. This operation is performed, for example, by mouse operation.

Next, a point of the designated portion is obtained, and a plurality of points around the obtained point (designated point) are obtained. For example, points in a rectangular area of 15×15 points centered on the specified point are acquired. Note that the range of points to be obtained and the method of obtaining points (for example, not continuous points but at intervals of 5 points or 10 points, etc.) can be arbitrarily set.

Next, an equation of a surface that fits a point group composed of the specified point and a plurality of points around it is calculated. Then, the normal of the calculated surface is calculated, and this normal is obtained as the normal of the surface of the specified portion of the object. Next, the orientation of the "clip area display" is adjusted according to the normal line of the surface of the specified portion. That is, the orientation of the "display of the clipping range" is adjusted so that the normal of the specified portion calculated by the above method and the normal of the "display of the clipping range" match. This orientation adjustment is performed by the marker display control unit 103 . In this way, for example, as shown in FIG. 11, the "cutting range display" is displayed according to the orientation of the surface of the object.

For example, in the state of FIG. 9, the building is cut obliquely by the “cutting range display”, so the display of FIG. 10, which displays only the cut point group, is a little difficult to grasp visually. On the other hand, in the process of setting the "clipping range display" according to the normal of the surface of the object, as can be seen from the right side of FIG. 11 and the display of FIG. It is possible to display a point cloud that makes it easy to visually grasp an object.

(Inventions disclosed herein)
The present specification discloses the following inventions. A first disclosed invention comprises: a first point cloud data display control unit for performing 3D display of point cloud data; An image processing apparatus comprising: a control unit; and a second point cloud data display control unit for displaying a point cloud existing on the cutting plane in the point cloud data on a separate screen.

In a second disclosed invention, in the first disclosed invention, the marker display can be moved in a direction perpendicular to the cutting plane, and the movement of the cutting plane causes the A new point group cut out by the cut plane is extracted from the point group data.

In the third disclosed invention, in the first or second disclosed invention, points in a specific range in a direction perpendicular to the cutting plane are extracted as the point group existing on the cutting plane. It is characterized by

The fourth disclosed invention is that in any one of the first to third disclosed inventions, the color of the display of the point cloud data cut by the cutting plane is not cut by the cutting plane. It is characterized in that processing is performed to make it different from the color of the display of the group data.

In a fifth disclosed invention, in any one of the first to fourth disclosed inventions, the marker display indicates the range of the cutting plane, and the position and size of the marker display are determined. It is characterized by the fact that one or both can be changed.

A sixth disclosed invention is any one of the first to fifth disclosed inventions, wherein the orientation of the cut surface is set in accordance with the direction of the surface of the point cloud data. Characterized by

A seventh disclosed invention comprises: a first point cloud data display control step of performing 3D display of point cloud data; and a second point cloud data display control step of displaying a point cloud existing on the cut plane in the point cloud data on a separate screen.

An eighth disclosed invention is a program to be read and executed by a computer, comprising: a first point cloud data display control unit for performing 3D display of point cloud data; Operate as a marker display control unit that displays a marker specifying a cutting plane of the group data, and a second point cloud data display control unit that displays a point group existing on the cutting plane in the point cloud data on a separate screen. An image processing program characterized by

Claims (3)

a point cloud data display control unit that performs 3D display of each point of the point cloud data on the display;
a surface direction calculation unit that calculates a surface equation related to a surface that constitutes an object from which the point cloud data is obtained, and calculates a normal line of the surface based on the surface equation;
a clipping range display control unit that displays, on the display, a clipping range display having a thick rectangular shape that displays a flat portion that is clipped from the point cloud data displayed in 3D;
Calculation of the surface equation is performed by obtaining a point of the specified portion and obtaining a surface equation that fits the point and a plurality of points around it,
The cut range display control unit is an image processing device that matches the direction of the normal line of the cut range display with the direction of the normal line of the surface calculated by the surface direction calculation unit. a point cloud data display control step for displaying each point of the point cloud data in 3D on the display;
a plane direction calculation step of calculating a plane equation relating to a plane constituting the object from which the point cloud data is obtained, and calculating a normal line of the plane based on the plane equation;
a clipping range display control step of displaying, on the display, a clipping range display having a thick rectangular shape that displays a flat portion clipped from the point cloud data displayed in 3D, and calculating the surface equation. is performed by obtaining a point of the designated portion, finding a surface equation that fits that point and points around it,
In the image processing method, in the clipping range display control step, the direction of the normal line of the clipping range display is matched with the direction of the normal line of the plane calculated in the plane direction calculation step . A program that is read and executed by a computer,
a point cloud data display control unit for displaying each point of point cloud data in 3D on a display;
a surface direction calculation unit that calculates a surface equation related to a surface that constitutes an object from which the point cloud data is obtained, and calculates a normal line of the surface based on the surface equation;
Operate as a clipping range display control unit that displays a clipping range display having a thick rectangular shape that displays a flat portion that is clipped from the point cloud data displayed in 3D on the display;
Calculation of the surface equation is performed by obtaining a point of the specified portion and obtaining a surface equation that fits the point and a plurality of points around it,
The image processing program, wherein the clipping range display control section matches the direction of the normal line of the clipping range display with the direction of the normal line of the plane calculated by the plane direction calculation section.

Images