JP7290602B2 - LEARNING DEVICE, METHOD AND PROGRAM - Google Patents

Description

The present invention relates to a learning device, method, and program for learning a model for automatically estimating landmarks from images.

As related technology such as positioning technology, there is technology related to landmarks.

In Patent Document 1, images can be classified and searched based on meta information such as the name of a landmark, and the convenience of organizing and searching images taken by the user is enhanced. In Japanese Patent Application Laid-Open No. 2002-100001, when searching for landmarks, landmarks with similar photographing conditions are searched first to reduce the processing load. Patent Literature 3 provides a landmark selection device capable of robustly selecting landmarks in consideration of positional dispersion and luminance dispersion.

Japanese Patent Application Publication No. 2011-010171 Japanese Patent Application Publication No. 2019-052904 Japanese Patent Application Publication No. 2019-053462

However, in the conventional technology as described above, there is a problem that it takes time and effort to manually register meta information and the like in advance in order to use landmarks.

When selecting a landmark, there is an image that judges the superiority of the landmark, but there is a possibility that it will have many similarities or will be almost impossible to find. Landmark learning was only generic with manual annotation. (For manual annotation, there are open data such as Oxford and Paris.) For example, indoors have landmarks for each facility, so general data like this cannot be handled, and each is manually annotated and created. There was a problem that was time-consuming.

In view of the problems of the prior art described above, the present invention utilizes point clouds that are commonly used in positioning and map creation to automatically estimate landmarks from images without the need for manual annotation. An object of the present invention is to provide a learning device, method, and program for learning a model that

In order to achieve the above object, the present invention is a learning device comprising: an image acquisition unit for acquiring a plurality of images of a field; A point cloud construction unit that constructs a point cloud of the field by obtaining the relationship, and in the point cloud, it is determined that the density of the points is high and that the shooting direction range of the image corresponding to the points is wide. a determination unit for setting the area to be marked as a landmark candidate area; assigning an identifier to each landmark candidate area; a learning unit that performs learning using learning data annotated as a certain landmark, and obtains a discrimination model for a landmark candidate area that is determined to have high discrimination performance for an object area in the learning result. do. Moreover, it is characterized by being the method and program corresponding to the said learning apparatus.

According to the present invention, a point cloud is constructed, an area automatically determined to have a high density and a wide range of corresponding shooting directions is set as a landmark candidate area, and an object area formed by the landmark candidate area in an image is determined. A discrimination model is trained using an annotated learning data, and a discrimination model is obtained for a landmark candidate area that is determined to have high discrimination performance for an object area among the results, It is possible to learn a model that automatically estimates landmarks from images without the need for manual annotation.

1 is a configuration diagram of a point cloud processing system according to an embodiment; FIG. 1 is a functional block diagram of a point cloud processing system according to one embodiment; FIG. 4 is a flowchart of the operation of a point cloud processing system according to one embodiment; FIG. 5 is a diagram showing an example of the relationship between point groups and landmarks, which is the basis for enabling the learning device to generate a landmark detection model without manual effort. FIG. 10 is a diagram showing a schematic example of a visual field range r(i,j,k) to be calculated as a two-dimensional cross section; FIG. 10 is a diagram schematically showing mapping in Procedure 2; It is a figure which shows the hardware constitutions in a common computer.

FIG. 1 is a configuration diagram of a point cloud processing system according to one embodiment. The point cloud processing system 100 includes at least one terminal 1 and at least one server that can communicate with each other via a network NW such as the Internet. Have 2. The terminal 1 is a mobile device that can be configured as a mobile device such as a smartphone or an in-vehicle device. It is obtained from In one embodiment, the point cloud processing system 100 includes only the terminal 1 and does not include the server 2, so that the terminal 1 alone may process the point cloud. In one embodiment, the point cloud processing system 100 includes a terminal 1 and a server 2, and the terminal 1 and the server 2 may share the processing and handle the processing related to the point cloud.

FIG. 2 is a functional block diagram of the point cloud processing system 100 according to one embodiment. A point cloud processing system 100 includes a learning device 10 and an estimating device 20 . The learning device 10 includes an image acquisition unit 11, a point cloud construction unit 12, a determination unit 13, a learning unit 14, and a model DB (database) 15. The determination unit 13 further includes a voxel division unit 131, a density evaluation unit 132, and a visual field evaluation unit. 133 and a candidate determination unit 134 are provided. The estimating device 20 includes an image capturing section 21 and an estimating section 22 .

Note that each of the learning device 10 and the estimation device 20 can be configured as the terminal 1 (and the server 2) shown in FIG. The terminal 1 configuring the learning device 10 as a mobile device and the terminal 1 configuring the estimation device 20 as a mobile device may be different terminals.

The terminal 1, which constitutes the learning device 10 as a mobile body, is equipped with at least an image acquisition unit 11, and acquires an image by photographing the surrounding environment of the terminal 1 itself. may be provided in the server 2 so that the processing may be entrusted to the server 2. The terminal 1 that configures the estimating device 20 as a mobile object is equipped with at least an image capturing unit 21, and acquires an image by capturing an image of the surrounding environment of the terminal 1 itself. may be provided in the server 2 so that the processing may be entrusted to the server 2.

FIG. 3 is a flowchart of the operation of the point cloud processing system 100 according to one embodiment. As the overall operation of the point group processing system 100, the learning device 10 builds a model by learning in steps S1 to S6 of FIG. Later, position estimation can be performed in step S7 of FIG. 3 by the estimation device 20 referring to and using the model stored in the model DB 15. FIG.

FIG. 4 is a diagram showing an example of the relationship between point clouds and landmarks, which is the basis for enabling the learning device 10 to generate a landmark detection model without manual effort. The point cloud PG in FIG. 4 shows a part of the field constructed to express the environment of the indoor space from the images taken by the camera moving in the indoor space along the movement trajectory C. be. In Fig. 4, this point group PG is shown in example EX1 from a top view of the field (viewpoint looking at the ground side from the sky), and in example EX2 this point group PG is shown from a side view of the field (perpendicular to the ground). horizontal view). In this point group PG, two locations P1 and P2 are examples of locations suitable for landmarks. Both of these two points P1 and P2 are densely packed with points of the point cloud and have a high density, and the direction in which the points of the point cloud can be seen (that is, the frame in the video from which the points constituting the point cloud are extracted) It has a feature that the camera direction in the image extends over a wide range. Although FIG. 4 includes similar landmark candidates other than P1 and P2, locations P1 and P2 are shown as examples for explanation.

In other words, the objects existing in such places P1 and P2 are visible from a wide range (without being blocked by fixedly arranged objects, etc.), and the number of points in the point cloud is large ( In other words, it is possible to automatically estimate that a landmark is an excellent landmark based on only the quantitative feature that the number of points detected as feature points is larger than that of the image. In the learning device 10 of the present embodiment, it is possible to automatically learn a landmark detection model by using such quantitative features of the point cloud without requiring manual annotation or the like.

Hereinafter, the details of the processing of each functional unit of the learning device 10 and the estimation device 20 will be described while describing the details of each step in FIG.

When the flow of FIG. 3 is started, in step S1, the image acquisition unit 11 acquires an image for building a point cloud in the field in (the point cloud construction unit 12 of) the learning device 10, and then Proceed to step S2. The image acquisition unit 11 is configured by a camera as hardware, and is provided in the terminal 1, which is a mobile object, so that the image in step S1 is acquired by performing shooting while moving in the field with this mobile object. The video (frame image F(t) at each time t=1, 2, 3, .

In step S2, the point group construction unit 12 constructs the point group PG using each frame image F(t) (t=1, 2, 3, . . . ) obtained in step S1, and the point group PG For each point (point in the three-dimensional world coordinates) belonging to , the result of linking the feature points of the corresponding frame image F(t) is output to the determination unit 13 as information of the point group PG, and then step Proceed to S3.

Here, any existing method such as SfM (Structure from Motion) may be used as a method for constructing the point group PG from the frame image F(t) (t=1, 2, 3, . . . ). In SfM, for example, feature points and local feature values are obtained from each image using SIFT features, etc., and the correspondence between feature points between images (correspondence between feature points determined to match the local feature values) is used to calculate triangulation. A point group PG can be constructed by obtaining information on the three-dimensional world coordinates of each feature point according to the principle of surveying. Regarding the linkage between the feature points of the frame image F(t) and the point group PG, each point (point in three-dimensional world coordinates) that constitutes the point group PG corresponds to any feature point of any frame image F(t). By giving (two-dimensional image coordinate point) and information on whether it corresponds to the local feature amount (this information is obtained when constructing the point group PG), linkage can be performed.

In step S3, the voxel dividing unit 131 divides the point group PG obtained in step S2 into voxels, and then the process proceeds to step S4. The voxel division unit 131 can divide the point group PG into voxels by previously dividing the three-dimensional world coordinates defining the point group PG into predetermined voxels (rectangular parallelepiped or cubic lattice). For the sake of explanation, among the results of voxel division of the point group PG by the voxel division unit 131, in the XYZ coordinates of the three-dimensional world coordinates, the voxel V A point group belonging to (i, j, k) is denoted as point group PG(i, j, k).

In step S4, for each voxel V(i,j,k), the density evaluation unit 132 evaluates the density d(i,j,k) of the point group PG(i,j,k), and also evaluates the visual field. After the unit 133 evaluates the viewing range r(i,j,k) of the point cloud P(i,j,k), it proceeds to step S5.

In the density evaluation unit 132, the volume of the voxel V(i,j,k) is defined as vol(i,j,k), and the point group PG(i,j,k) belonging to the voxel V(i,j,k) Let num(i,j,k) be the number of points in k) can be obtained. If the size of each voxel V(i,j,k) is common and the volume vol(i,j,k) is also constant, d(i,j,k)=num(i,j,k) ), the number of points num(i,j,k) may be used as it is as the value of the density d(i,j,k).

In the visual field evaluation unit 133, for example, two or more (N N=N(p) (the number N is generally different for each point p, but hereinafter simply written as N). , consider a cone Cone(p) with a base B(p) bounded by N corresponding camera positions Cp1, Cp2, . ), the voxel point group PG(i,j,k) The viewing range r(i,j,k) may be calculated as the area of the union ∪s(p) of the regions s(p) by all the points p belonging to .

FIG. 5 is a diagram showing a schematic example of the visual field range r(i,j,k) calculated as described above as a two-dimensional cross section. , q, r, a region s(p) is clipped from the unit sphere E by a cone Cone(p) by at least two camera positions Cp1, Cp2 to which point p corresponds, and at least two A region s(q) is cut from the unit sphere E by a cone Cone(q) by two camera positions Cq1,Cq2, and a cone Cone(r) by at least two camera positions Cr1,Cr2 to which a point r corresponds. By cutting the area s(r) from E, in the example of FIG. 5, the visual field range r(i,j, k) is determined.

In the visual field evaluation unit 133, the camera positions Cp1, Cp2, …, CpN (the direction when viewed from the center of point p or voxel V(i,j,k)) varies widely, the viewing range of point p is assumed to be wide, and the viewing range of point p is evaluated. Then, the visual field range r(i, j, k) is calculated so that the wider the overall visual field range by all the points p of the voxel point group PG (i, j, k) is, the larger the value becomes. can be done.

For example, for each surface (six surfaces) of a voxel V(i,j,k) that is simply configured as a rectangular parallelepiped or cube without setting a unit sphere, etc., the voxel point group VG(i,j,k) and the camera positions Cp1, Cp2, . . . , CpN. If it passes, the evaluation value is set to 1, and if such a straight line does not pass, the evaluation value is set to 0, and the sum of the evaluation values for the six surfaces (0 to 6) is simply calculated as the visual field range r (i, j , k) may be calculated.

Information on the N camera positions Cp1, Cp2, . That is, since the feature point of the frame image corresponding to each point p belonging to the point group PG is linked, the information of N frame images corresponding to each point p is linked. As the information of the camera positions when the frame images were shot (this camera position information is also calculated when calculating the point group PG), the information of the camera positions Cp1, Cp2, ..., CpN is also obtained from the point group PG It is possible to hold it by linking it to .

In step S5, the candidate determination unit 134 determines the density d(i,j,k) calculated by the density evaluation unit 132 in step S4 and the visual field calculated by the visual field evaluation unit 133 for each voxel V(i,j,k). Based on the range r(i, j, k), determine whether or not the voxel V(i, j, k) corresponds to the candidate voxel, output the determination result to the learning unit 14, and then step Proceed to S6.

Specifically, the candidate determination unit 134 determines that the density d(i,j,k) is greater than a predetermined density threshold THd and the visual field range r(i,j,k) is a predetermined visual field range threshold value. Those voxels V(i,j,k) that are determined to be greater than THr can be determined as qualifying candidate voxels.
d(i,j,k)>THd and r(i,j,k)>THr

The candidate voxels determined in this way are the point groups PG(i,j,k) of the corresponding voxels V(i,j,k) such as those schematically shown as points P1 and P2 in FIG. It means that it is a candidate that is derived from a landmark having a characteristic. That is, a candidate voxel means that there is a possibility that a landmark exists within the voxel V(i,j,k).

In step S6, the learning unit 14 performs learning using the video (each frame image F(t)) acquired by the image acquisition unit 11 and the information of the candidate voxels determined by the candidate determination unit 134, thereby performing land acquisition from the image. After obtaining a deep learning model for automatically detecting the mark area and storing this deep learning model in the model DB 15, proceed to step S7.

In step S6, the learning unit 14 can specifically learn the model according to procedures 1 to 3 below.

(Procedure 1) Adjacent (connected) voxels (voxels constituting connected components) are grouped into one cluster by applying connected region labeling or the like to the candidate voxels determined by the candidate determination unit 134. In summary, assign a common ID. For the sake of explanation, it is assumed that there are a total of K concatenated candidate voxel groups obtained in this way, and these are Vc1, Vc2, ..., VcK (= {Vck|k = 1, 2, ..., K}). write.

(Procedure 2) Each connection candidate voxel group Vck (k = 1, 2, ..., K) is mapped onto the frame image F (t) (t = 1, 2, 3, ...) by perspective projection, Information on the area R(t,k) occupied by each concatenated candidate voxel group Vck in each frame image F(t) is acquired. (In addition, depending on the combination of t, k, it may not be projected in the first place (R(t, k) is an empty set).)

FIG. 6 is a diagram schematically showing the mapping in procedure 2. A certain connection candidate voxel group Vck existing in the three-dimensional world coordinate system is perspectively projected onto the two-dimensional image coordinates of a certain frame image F(t). It is schematically shown that a region R(t,k) is defined as a region defined by the As already explained, when the point group PG is obtained for the frame image F(t), the camera position (and orientation) corresponding to the extrinsic parameters of the camera is determined in the three-dimensional world coordinate system. Using this information (and known camera intrinsic parameter information), it is possible to perform a perspective projection as shown in FIG.

(Procedure 3) Object detection and instance segmentation by deep learning using frame images F(t) in which at least one region R(t,k) is mapped and projected as training data with automatic annotations. and other learning. Here, the region R(t,k) in the frame image F(t) is a group of K connected candidate voxels Vck (k=1, 2, . . . , K), that is, Among them, it can be used as learning data to which an automatic annotation is given as corresponding to the target area of the k-th landmark. In other words, in the learning data automatically annotated, information about what the target of the landmark is specifically (what object can be recognized by humans) is not explicitly given, but K A landmark identifier indicating which of the k-th landmarks is the landmark candidate is attached as an annotation.

Learning using this automatically annotated learning data can be performed in the following procedure 3-1 to procedure 3-3.

(Procedure 3-1) Each landmark candidate distinguished by identifier k = 1, 2, ..., K (landmark candidate L( k)) to obtain the evaluation results of correctness and accuracy. (That is, part of the learning data is used for training and the remaining part is for verification, and while training is performed using the training data, the learning state is periodically verified using the verification data to obtain evaluation results. ) The accuracy can be evaluated by general indices (recall, precision, F value, etc.), and as an index, IoU (Intersection over Union) can be used to evaluate the accuracy of the estimation of the overlap of regions. Alternatively, mAP (mean Average Precision) may be used as an index for collectively evaluating multiple object detections.

(Procedure 3-2) In the evaluation result of the above procedure 3-1, the landmark candidate L(k) with low accuracy and erroneous estimation is regarded as having low uniqueness, that is, as a landmark It is excluded from the candidates as inappropriate for setting and using.

(Procedure 3-3) The landmark candidate L(k), which was not excluded from the candidates in the above procedure 3-2, is confirmed as a landmark L(k) instead of a candidate. The learned model and the position of the landmark (the position in the three-dimensional world coordinates given as the position of the original candidate voxel group Vck) are linked and stored in the model DB15. (In addition, regarding the landmark candidates excluded from the candidates in procedure 3-2, the learning data is updated as if automatic annotation was not given in the learning data, and the same learning as in procedure 3-1 is performed again. The learning model and landmark positions obtained by

As described above, after completing step S6 in FIG. 3, the process proceeds to step S7. In step S7, the estimating device 10 captures a new image, and by using the model obtained in step S6 and stored in the model DB 15, the camera that captured this image (the camera that constitutes the image capture unit 21 ) is estimated, and the flow of FIG. 3 ends.

Specifically, in step S7, first, the image capturing unit 21 captures an image and outputs this image to the estimating unit 22. FIG. Next, the estimating unit 22 applies object recognition to this image using a trained model stored in the model DB 15, and determines which landmark L(k) (not a candidate) is located in which region in the image. A result of recognizing whether or not the image is captured is obtained, and a position estimation result of the image is obtained from the position and size of the recognized landmark. Here, the position and size of the landmark are estimated as those in the image (position and size in two-dimensional image coordinates), and then the position of the image is estimated as those in three-dimensional world coordinates. Specifically, for example, it may be estimated according to the cases (1) and (2) as follows.

(1) First, when there are three or more landmarks in the captured image, the three-dimensional position of the camera image can be calculated from the positional relationship. In addition, at that time, the accuracy of the three-dimensional position may be improved by also considering the size of the detected two-dimensional image coordinates. (1a) After estimating the three-dimensional position, a method of extracting an image near the estimated camera image position and estimating the position by point cloud matching may be used in order to further improve accuracy. (2) On the other hand, if there are two or less landmarks in the captured image, use the image containing the landmark as a matching candidate image for the point cloud of the normal VPS, and use the normal point cloud matching Three-dimensional position estimation may be performed.

As described above, according to the learning device of the present embodiment, landmarks can be automatically estimated from images using point clouds that are generally used in positioning and map creation, without the need for manual annotation. It is possible to learn a model that Various supplementary explanations are given below.

(A) In step S1 of FIG. 3, the image acquisition unit 11 acquires each frame image of a video captured while moving within the field by the terminal 1 as a single moving object. A group of images to which the form can be applied is not limited to those acquired as videos in this way. In step S1, the image acquisition unit 11 may acquire a group of images of the field to be constructed of the point cloud, captured in various positions and orientations by each camera of a plurality of moving bodies. In addition, a group of images captured by fixed cameras at various positions may be acquired, instead of being limited to mobile cameras. When acquiring in this way, the image acquisition unit 11 is configured as a communication interface for acquiring images captured by various cameras from the network instead of or in addition to being configured as a camera as hardware. can be anything.

(B) FIG. 7 is a diagram showing an example of a hardware configuration in a general computer device 70. As shown in FIG. Each of the terminal 1 and the server 2 constituting the learning device 10 and the estimation device 20 in the point cloud processing system 100 can be implemented as one or more computer devices 70 having such a configuration. When the learning device 10 or the estimation device 20 is realized by two or more computer devices 70, information necessary for processing may be transmitted and received via a network. The computer device 70 includes a CPU (central processing unit) 71 that executes predetermined instructions, and a GPU (graphics processing unit) as a dedicated processor that executes part or all of the execution instructions of the CPU 71 instead of the CPU 71 or in cooperation with the CPU 71. ) 72, RAM 73 as a main storage device that provides a work area to CPU 71 (and GPU 72), ROM 74 as an auxiliary storage device, communication interface 75, display 76, input interface 77 that accepts user input by mouse, keyboard, touch panel, etc. A camera 78 having the image acquiring unit 11 and the image photographing unit 21 as hardware, and a bus BS for exchanging data therebetween are provided. As described above, the hardware configuring the image acquisition unit 11 may be the communication interface 75 .

Each functional unit of the learning device 10 and the estimating device 20 can be realized by the CPU 71 and/or the GPU 72 that reads and executes a predetermined program corresponding to the function of each unit from the ROM 74 . Both the CPU 71 and the GPU 72 are a kind of arithmetic unit (processor). Here, when display-related processing is performed, the display 76 further operates in conjunction, and when communication-related processing relating to data transmission/reception is performed, the communication interface 75 further operates in conjunction. The processing results and the like by the point cloud processing system 100 may be displayed on the display 76 and output.

100…Point cloud processing system, 1…Terminal, 2…Server
10... Learning device, 11... Image acquisition unit, 12... Point group construction unit, 13... Determination unit, 14... Learning unit, 15... Model DB, 131... Voxel division unit, 132... Density evaluation unit, 133... Visual field evaluation unit , 134 ... candidate decision unit
20... estimating device, 21... image capturing unit, 22... estimating unit

Claims (6)

an image acquisition unit that acquires a plurality of images of a field;
a point cloud constructing unit that constructs a point cloud of the field by detecting feature points from each of the images and obtaining a correspondence relationship of the feature points between the images;
a determining unit that sets, as a landmark candidate area, an area determined to have a high density of points in the point group and a wide range of photographing directions of an image corresponding to the points;
assigning an identifier to each landmark candidate area, performing learning using learning data annotated as an object area specified by the identifier in the landmark candidate area assigned the identifier in the image;
a learning unit that obtains a discrimination model for a landmark candidate area that is determined to have high discrimination performance for an object area in the learning result. 2. The learning device according to claim 1, wherein the determining unit divides the point group into predetermined voxels and sets the landmark candidate area for each voxel. 3. The learning device according to claim 2, wherein the learning unit assigns a landmark candidate area identifier to each connected component of voxels set as the landmark candidate area. The learning unit determines the object area specified by the identifier by perspectively projecting the landmark candidate area assigned the identifier in the image from the three-dimensional world coordinates onto the coordinate system of the image. 4. The learning device according to any one of claims 1 to 3. an image acquisition step of acquiring a plurality of images of the field;
a point cloud construction step of constructing a point cloud of the field by detecting feature points from each of the images and determining correspondences of the feature points between the images;
a determination step of setting, as a landmark candidate area, an area in the point cloud determined to have a high density of points and a wide range of shooting directions of an image corresponding to the point;
assigning an identifier to each landmark candidate area, performing learning using learning data annotated as an object area specified by the identifier in the landmark candidate area assigned the identifier in the image;
a learning step of obtaining a discrimination model for a landmark candidate area determined to have high discrimination performance for an object area in the learning result. A program that causes a computer to function as the learning device according to any one of claims 1 to 4.

Images