JP7020322B2 - Building limit judgment device - Google Patents

Description

The present invention relates to a building limit determination device that determines the separation between a building limit area and an obstacle by image processing from a plurality of camera images in front of or behind a vehicle traveling on a rail in the railway field and the image processing field.

Railway equipment requires inspection and maintenance, and Patent Documents 1 and 2 propose a building limit determination device that determines the separation between a building limit area and an obstacle.

Patent Document 1 is a device for determining a building limit by stereo measurement. For stereo measurement, a method called monocular stereo is used to measure using one camera and vehicle speed.

Patent Document 2 is also a device for determining a building limit by stereo measurement. For stereo measurement, images taken synchronously by two or more fixed cameras are used, and the distance between the building limit area and obstacles is measured by recognizing the rail area from the image.

In addition, Non-Patent Document 1 is a research paper of a method called Semantic Segmentation, and describes a technique used for Semantic Segmentation. Semantic Segmentation is a method of estimating what kind (label) the pixel is for each pixel of an image, and the label needs to be decided in advance by a person. Further, Non-Patent Document 2 describes a deep learning method called Fully Convolutional Networks for Semantic Segmentation.

Japanese Unexamined Patent Publication No. 2016-598 Japanese Unexamined Patent Publication No. 2017-83245

M. Thoma, "A Survey of Semantic Segmentation", [online], May 11, 2016 (2016), [Search February 21, 2018], Internet <URL: https://arxiv.org /pdf/1602.06541v2.pdf> J. Long et al., "Fully Convolutional Networks for Semantic Segmentation", [online], March 8, 2015, [Search February 21, 2018], Internet <URL: https: / /arxiv.org/pdf/1411.4038v2.pdf> V. Kolmogorov et al., "Multi-camera Scene Reconstruction via Graph Cuts", ECCV, VOL. 3, 2002, pp. 82-96 P. J. Besl et al., "A Method for Registration of 3-D Shapes", IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 14, NO.2, FEBRUARY 1992, pp. 239-256

Patent Document 1 uses monocular stereo measurement. Although this method requires a simple device configuration, the results tend to be unstable compared to the method of performing stereo measurement with multiple cameras. Therefore, the problem is that erroneous correspondence due to stereo measurement (stereo measurement is triangulation by associating the same part between two images, but the correspondence is made different) is likely to occur. Be done. In addition, there is noise removal as a solution to the mishandling, but the area not used for mapping is defined as the area to be associated, or conversely, the processing to limit the areas to be associated is not performed. That is also a problem.

Since Patent Document 2 is a method of performing stereo measurement with a plurality of cameras, erroneous correspondence is less likely to occur as compared with the monocular stereo method of Patent Document 1. In addition, countermeasures against erroneous measures are taken by performing global optimization with the constraint that the parallax of adjacent pixels (the result of associating and determining how much the pixel position is different) does not fluctuate significantly. However, as in Patent Document 1, in noise removal, an area not used for association may be defined as an area to be associated, or conversely, a process of limiting the areas to be associated may not be performed. It is mentioned as a problem.

Non-Patent Documents 1 and 2 are papers that introduce methods related to Semantic Segmentation, and do not determine building limits in the railway field.

The present invention has been made in view of the above problems, and an object of the present invention is to provide a building limit determination device capable of making a robust and highly accurate determination against noise.

The building limit determination device according to the first invention that solves the above problems is
A plurality of cameras provided on a vehicle traveling on rails to capture the front or rear of the vehicle, and
It has a processing unit that performs image processing of image data captured by the plurality of cameras.
The processing unit
A label teacher data unit that stores teacher image data composed of a plurality of images similar to the image data and teacher label data in which labels are given in advance to each pixel of the plurality of images of the teacher image data.
A label learning unit that preliminarily creates a learning model that associates each pixel of the plurality of images of the teacher image data with the label of the teacher label data by learning by machine learning.
Using the plurality of image data captured at the same time, the corresponding pixel among the plurality of image data is searched as a measurement point from the information of the pixel value of each pixel of the plurality of image data, and the measurement point is reached. A distance data calculation unit that calculates the distance by stereo measurement with the distance as the first distance data,
A rail recognition unit that recognizes the position of the rail from the image data based on the matching data of the rail prepared in advance and calculates the position as the rail position.
A label estimation unit that estimates the label for each pixel of the image data by the learning model and obtains the estimated label as label data.
A noise removing unit that performs noise removal of the first distance data based on the label data for each pixel of the image data and obtains the noise-removed first distance data as the second distance data.
A distance data integration unit that integrates the second distance data of the image data captured at a plurality of points to obtain a three-dimensional environment map and obtains a vehicle inclination, and a distance data integration unit.
A building limit area that calculates a dynamic building limit area for the vehicle based on the rail position at multiple points, the vehicle tilt at multiple points, and the static building limit area given as a fixed value. Calculation unit and
It has a separation calculation unit that calculates the separation distance from the dynamic building limit area to the nearest object based on the dynamic building limit area and the three-dimensional environment map, and determines the separation. It is a feature.

The building limit determination device according to the second invention for solving the above problems is
In the building limit determination device according to the first invention,
When removing noise from the first distance data, the noise removing unit further removes noise by using any of a median filter, a non-local mean filter, and a geometric filter, and the second distance. It is characterized by obtaining data.

The building limit determination device according to the third invention that solves the above problems is
A plurality of cameras provided on a vehicle traveling on rails to capture the front or rear of the vehicle, and
It has a processing unit that performs image processing of image data captured by the plurality of cameras.
The processing unit
A label teacher data unit that stores teacher image data composed of a plurality of images similar to the image data and teacher label data in which labels are given in advance to each pixel of the plurality of images of the teacher image data.
A label learning unit that preliminarily creates a learning model that associates each pixel of the plurality of images of the teacher image data with the label of the teacher label data by learning by machine learning.
A label estimation unit that estimates the label for each pixel of the image data by the learning model and obtains the estimated label as label data.
Using the plurality of image data captured at the same time and the label data for each pixel of the plurality of image data, the information of the pixel value of each pixel of the plurality of image data and the space between the plurality of image data from the label data. A distance data calculation unit that searches for the corresponding pixel as a measurement point and calculates the distance to each measurement point as the first distance data by stereo measurement.
Using the image data and the label data for each pixel of the image data, the position of the rail is recognized from the image data by the matching data of the rail and the label data prepared in advance, and the position is set as the rail position. The rail recognition unit to calculate and
A distance data integration unit that integrates the first distance data of the image data captured at a plurality of points to obtain a three-dimensional environment map and obtains a vehicle inclination, and a distance data integration unit.
A building limit area that calculates a dynamic building limit area for the vehicle based on the rail position at multiple points, the vehicle tilt at multiple points, and the static building limit area given as a fixed value. Calculation unit and
It has a separation calculation unit that calculates the separation distance from the dynamic building limit area to the nearest object based on the dynamic building limit area and the three-dimensional environment map, and determines the separation. It is a feature.

The building limit determination device according to the fourth invention that solves the above problems is
In the building limit determination device according to the third invention,
The processing unit uses any of a median filter, a non-local mean filter, and a geometric filter to remove noise from the first distance data, and the noise-removed first distance data is used as second distance data. Further has a noise removing part to be obtained as
The distance data integration unit is characterized in that the second distance data of the image data captured at a plurality of points is integrated to obtain the three-dimensional environment map and the vehicle inclination.

The building limit determination device according to the fifth invention for solving the above problems is
In the building limit determination device according to any one of the first to fourth inventions.
The label learning unit is a local area centered on each pixel from the plurality of images of the teacher image data by the machine learning using any of SVM (Support Vector Machine), Random Forest, or deep learning and Semantic Segmentation. The image of the range is cut out, the feature amount is extracted from the image of the local range, and the correspondence between the central pixel of the image of the local range and the label of the teacher label data is learned based on the feature amount. Create a learning model and
The label estimation unit cuts out an image of a local range centered on each pixel from the image data by the Semantic Segmentation and the learning model, extracts a feature amount from the image of the local range, and bases the feature amount on the feature amount. The label is estimated for each pixel of the image data by estimating the label for the central pixel of the image in the local range.

The building limit determination device according to the sixth invention for solving the above problems is
In the building limit determination device according to the fifth invention,
When extracting the feature amount, the label learning unit and the label estimation unit perform a color feature amount, a HOG (Histograms of Oriented Gradients) feature amount, a SIFT (Scale-Invariant Feature Transform) feature amount, and a BoVW (Bag of Visual). Words) It is characterized by using either a feature amount or a Texton feature amount, or a combination thereof.

The building limit determination device according to the seventh invention that solves the above problems is
In the building limit determination device according to any one of the first to fourth inventions.
The label learning unit extracts a feature amount from each pixel of the plurality of images of the teacher image data by the machine learning using FCN (Fully Convolutional Networks) and Semantic Segmentation, and based on the feature amount, The learning model is created by learning the correspondence between each pixel of the plurality of images of the teacher image data and the label of the teacher label data.
The label estimation unit extracts a feature amount from each pixel of the image data by the Semantic Segmentation and the learning model, and estimates the label for each pixel of the image data based on the feature amount. It is characterized by.

The building limit determination device according to the eighth invention for solving the above problems is
In the building limit determination device according to any one of the first to seventh inventions.
The label estimation unit is characterized in that an MRF (Markov Random Field) or a CRF (Conditional Random Field) is used when estimating the label.

According to the present invention, it is possible to perform highly accurate noise removal by estimating the label for each pixel, and as a result, in determining the separation between the building limit region and the obstacle, it is possible to perform a robust and highly accurate determination on noise. Can be done.

It is a top view of the schematic structure which shows an example of embodiment of the building limit determination apparatus which concerns on this invention. It is a block diagram explaining an example (Embodiment 1) of the processing unit which constitutes the building limit determination apparatus shown in FIG. (A) is an explanatory diagram for explaining label estimation for a local patch image unit centered on a certain pixel, (b) is an explanatory diagram for explaining the result of label estimation in (a), and (c) is (c). An explanatory diagram for explaining the label estimation of the local patch image unit centering on the pixel next to the pixel in (a), (d) is an explanatory diagram for explaining the result of label estimation in (c), and (e) is. , (C) is an explanatory diagram for explaining the label estimation of the local patch image unit centering on the pixel next to the pixel in (c), and (f) is an explanatory diagram for explaining the result of the label estimation in (e). (A) is a diagram showing a surveillance image captured by the building limit determination device shown in FIG. 1, and (b) is an explanatory diagram illustrating an example of a label for the surveillance image shown in (a). It is a flowchart explaining an example of the building limit determination method in the processing part shown in FIG. It is a block diagram explaining another example (Embodiment 2) of the processing part which constitutes the building limit determination apparatus shown in FIG. It is a flowchart explaining an example of the building limit determination method in the processing part shown in FIG. It is a block diagram explaining another example (Embodiment 3) of the processing part which constitutes the building limit determination apparatus shown in FIG. It is a flowchart explaining an example of the building limit determination method in the processing part shown in FIG. It is a block diagram explaining another example (Embodiment 4) of the processing part which constitutes the building limit determination apparatus shown in FIG. It is a flowchart explaining an example of the building limit determination method in the processing part shown in FIG.

Some of the embodiments of the building limit determination device according to the present invention will be described with reference to FIGS. 1 to 11.

[Example 1]
FIG. 1 is a top view of a schematic configuration showing an example of a building limit determination device of this embodiment. Further, FIG. 2 is a block diagram illustrating an example of a processing unit constituting the building limit determination device shown in FIG. 1. Further, FIG. 3A is an explanatory diagram for explaining label estimation for a local patch image unit centered on a certain pixel, and FIG. 3B is an explanatory diagram for explaining the result of label estimation in FIG. 3A. FIG. 3 (c) is an explanatory diagram illustrating label estimation for a local patch image unit centered on the pixel next to the pixel in FIG. 3 (a), and FIG. 3 (d) is FIG. 3 (c). 3 (e) is an explanatory diagram for explaining the result of label estimation in FIG. 3, and FIG. 3 (e) is an explanatory diagram for explaining label estimation for a local patch image unit centered on the pixel next to the pixel in FIG. 3 (c). (F) is an explanatory diagram illustrating the result of label estimation in FIG. 3 (e). Further, FIG. 4A is a diagram showing a surveillance image captured by the building limit determination device shown in FIG. 1, and FIG. 4B is an example of a label for the surveillance image shown in FIG. 4A. It is explanatory drawing to explain.

The building limit determination device of this embodiment uses image pickup and image processing by a plurality of cameras. Specifically, as shown in FIG. 1, two cameras 22a and 22b provided on a train 21 (vehicle) traveling on a rail 11 in the direction of arrow E, and surveillance images captured by the cameras 22a and 22b. It has a processing unit 30A that performs image processing using (see FIG. 4A) and determines the separation between the building limit area and an obstacle. Reference numeral 12 represents a utility pole.

The cameras 22a and 22b are provided at the head of the train 21 and are arranged toward the front in the longitudinal direction of the rail 11 (in the direction of the arrow E). In this case, the front monitoring images captured by the cameras 22a and 22b are displayed. Used for image processing. Instead of the cameras 22a and 22b, the cameras 23a and 23b provided at the end of the train 21 and arranged toward the rear in the longitudinal direction of the rail 11 may be used. In this case, the rear images captured by the cameras 23a and 23b may be used. The surveillance image of is used for image processing. The cameras 22a, 22b or the cameras 23a, 23b are fixed to the train 21, and can take an image of the surroundings depending on the movement, shaking, and tilt of the train 21. The number of cameras 22a, 22b or the cameras 23a, 23b may be a plurality (two or more).

As described above, when a plurality of cameras 22a and 22b (or cameras 23a and 23b) are used, their relative position-posture relationships are known. Therefore, the distance to the object can be determined by stereo measurement based on the principle of triangulation. It can be obtained directly, and the building limit can be determined not only for an object having a design value (for example, railway equipment) but also for an object without a design value (for example, a natural object). In addition, when image processing is used, it is advantageous in terms of recognizing and discriminating measured railway equipment (for example, rails), the equipment is relatively small and inexpensive, and it is excellent in terms of resolution and imaging speed. It has the advantages that it has the potential for future development and that it is possible for humans to visually inspect the captured images.

Further, in the building limit determination device of this embodiment, the type (label) of each pixel in the image is estimated by Semantic Segmentation. In Semantic Segmentation in this embodiment, as described in Non-Patent Document 1, an image (local patch image) of a small rectangular local range centered on each pixel is extracted, and the label estimation of the local patch image is performed. By doing so, it is possible to acquire the label of each pixel.

Therefore, as shown in FIG. 2, the above-mentioned processing unit 30A includes a label teacher data unit 31, a label learning unit 32A, a monitoring image input unit 41, a distance data calculation unit 42A, a rail recognition unit 43A, and a label. It has an estimation unit 44A, a noise removal unit 45A, a distance data integration unit 46, a building limit area calculation unit 47, a separation calculation unit 48, and a storage unit 49.

In the label teacher data unit 31, the teacher image data D1 composed of a sufficient number of image groups having characteristics similar to the assumed captured image and each pixel of the image group of the teacher image data D1 are manually or somehow labeled. The teacher label data D2 given the above is stored.

The above-mentioned labels need to be determined in advance, and in railway equipment, signs, rails, and the like exemplified in Table 1 below are used as labels.

In the label learning unit 32A, the teacher image data D1 and the teacher label data D2 in the label teacher data unit 31 are used, and learning is performed by supervised learning of a machine learning algorithm, and each pixel of the image group of the teacher image data D1 is used. A learning model D3A associated with the label of the teacher label data D2 is created in advance, and the created learning model D3A is stored in the storage unit 49.

Specifically, by machine learning described later and Semantic Segmentation described above, a local patch image centered on each pixel of the image group of the teacher image data D1 is cut out, and a feature amount is extracted from the cut out local patch image and extracted. Based on the feature amount, learn the correspondence between the central pixel of the local patch image and the label of the teacher label data D2 attached to each pixel in advance, and define a function to output the label for the input of the feature amount. As a result, a learning model D3A that associates each pixel of the image group of the teacher image data D1 with the label of the teacher label data D2 is created.

In the label learning unit 32A, the feature amount is extracted by using the color of each pixel as it is, the HOG (Histograms of Oriented Gradients) feature amount using the histogram of the gradient of the pixel value, and the scale and orientation of the object to be reflected in the image. BoVW (Bag of Visual Words) features that vector-quantify the extent to which SIFT (Scale-Invariant Feature Transform) features, HOG features, and SIFT features are included, and express the features in a histogram. Either the amount, the Specton features extracted by CNNs (Convolutional Neural Networks) called convolutional neural networks, or a combination of these is used.

In addition, the machine learning algorithm is a classifier using supervised learning, SVM (Support Vector Machine) capable of non-linear discrimination by the kernel method, and Random that can perform highly generalized classification using multiple decision trees. Use either Forest, deep learning with high expressiveness and generalization ability.

The surveillance image input unit 41 inputs the front (or rear) surveillance image data D11 captured by the front plurality of cameras 22a, 22b (or the rear rear plurality of cameras 23a, 23b) to the storage unit 49. .. At this time, the monitoring image data D11 may be input in association with the kilometer information (or the kilometer position) indicating the position of the train 21 on the rail 11.

The distance data calculation unit 42A performs stereo support (searching for corresponding pixels among a plurality of images as measurement points) using a plurality of (two or more) monitoring image data D11 captured at the same time. , Stereo measurement is performed by performing triangular measurement of the measurement points searched by this stereo correspondence, and by this, the distance data D12A (first distance data) to the object including the obstacle is calculated and calculated. The distance data D12A is stored in the storage unit 49.

That is, in the distance data calculation unit 42A, the information of the pixel value of each pixel of the plurality of surveillance image data D11 captured at the same time (for example, color information, brightness information, etc.) is used between the surveillance image data D11. The corresponding pixel is searched as a measurement point, and triangular measurement is performed for the distance to each searched measurement point. In these processes, a method called block matching and a method called graph cut (non-patent documents). 3) is used.

Specifically, using block matching, the area similar to the pixels around the pixel of interest of the surveillance image data D11 of one of the cameras 22a, 22b (or the cameras 23a, 23b) is monitored by the other camera. The distance data D12A is calculated by searching from the image data D11 and performing stereo measurement using the principle of triangular measurement with the pixel of interest as the measurement point. When there is no characteristic texture around the pixel of interest on the surveillance image data D11, a penalty term is used so that the estimated distance data of the pixel of interest does not differ significantly from the estimated distance data of the periphery. Optimizes the distance data D12A. By using this block matching and graph cut, it is possible to calculate the distance data D12A in consideration of smoothness from the monitoring image data D11 regardless of edge detection, binarization processing, and block detection.

The rail recognition unit 43A recognizes the position of the rail 11 from the monitoring image data D11 by using the monitoring image data D11 and the matching data of the rail 11 prepared in advance, calculates the position as the rail position D13, and calculates the position as the rail position D13. The position D13 is stored in the storage unit 49.

Specifically, as matching data, template image data of the rail 11 and reference distribution data (reference rail brightness distribution data) of the brightness value of the rail 11 are prepared in advance. Then, in the monitoring image data D11, the area of the arbitrary horizontal line (line in the sleeper direction) is matched by raster scanning using the template image data of the rail 11 and the reference rail brightness distribution data, and the most matching position is performed. Is calculated as the rail position D13. In this way, the position of the rail 11 in the monitoring image data D11 is recognized, and the rail position D13 can be accurately detected from the monitoring image data D11.

The label estimation unit 44A estimates the label for each pixel of the monitoring image data D11 by the learning model D3A, obtains the estimated label as the label data D14, and stores the obtained label data D14 in the storage unit 49.

Specifically, the above-mentioned Semantic Segmentation and the learning model D3A in the storage unit 49 are used to cut out a local patch image centered on each pixel from the monitoring image data D11, and extract a feature amount from the cut out local patch image. Based on the extracted feature amount, the label for the central pixel of the local patch image is estimated, the probability (probability) of each label for the pixel is obtained, and the label with high probability in each pixel is labeled as the label for each pixel. We are looking for data D14.

Here, an example of label estimation for each local patch image using Semantic Segmentation will be described with reference to FIGS. 3 (a) to 3 (f).

As shown in FIG. 3A, the label estimation unit 44A cuts out a 3 × 3 pixel region centered on a certain pixel PX1 in the image as a local patch image, performs label estimation of this local patch image, and performs pixel estimation. The probability of each label of PX1 is obtained, and as shown in FIG. 3B, a label having a high probability is acquired as a label estimation result of pixel PX1. Then, as shown in FIGS. 3 (c) to 3 (f), label estimation is sequentially performed for each local patch image centered on each pixel, such as pixel PX2 next to pixel PX1 and pixel PX3 next to pixel PX2. By performing this, the probability of each label for each pixel is obtained, and the label having a high probability is acquired as the label estimation result of each pixel, the label of the entire image can be estimated. Here, for the sake of explanation, the size of the local patch is set to the area of 3 × 3 pixels, but the size of the local patch may be changed according to the situation.

Further, a specific example of label estimation for the monitored image will be described with reference to FIGS. 4 (a) and 4 (b).

As shown in FIG. 1, when two cameras 22a and 22b are used to take an image of the front of the rail 11 in the longitudinal direction (direction of arrow E), as shown in FIG. 4A, the front monitoring image is displayed. It is imaged. In this surveillance image, as an example, the front rail 11, the utility pole 12, the ground 13, and the sky 14 are imaged. When label estimation is performed by the label estimation unit 44A using the labels exemplified in Table 1 for such a monitoring image, as shown in FIG. 4B, the rail 11 is a “rail” and the utility pole 12 is a “rail”. Labels of "telephone pole", "ground" on the ground 13, and "sky" on the sky 14 will be obtained.

In such a label estimation unit 44A, for extracting the feature amount, any one of the color feature amount, the HOG feature amount, the SIFT feature amount, the BoVW feature amount, and the Texton feature amount, or these, as in the label learning unit 32A. Use the combination of. Furthermore, as post-processing to improve the accuracy of label estimation, as described in Non-Patent Document 1, methods such as MRF (Markov Random Field) and CRF (Conditional Random Field) for checking the consistency with nearby pixels. May be used.

That is, the label estimation unit 44A performs extraction of the feature amount of the local patch image unit, label estimation by the learning model, and post-processing, and the label estimation of each pixel can be performed with high accuracy by these processes.

The noise removing unit 45A removes noise from the distance data D12A based on the label data D14 of each pixel of the monitoring image data D11, and stores the noise-removed distance data D12A as the distance data D15 (second distance data). Stored at 49.

Specifically, noise is removed from the distance data D12A of each pixel by using the label data D14 of each pixel of the plurality of monitoring image data D11 captured at the same time. For example, when a certain pixel is labeled as "sky" as the label data D14, the distance data D12A is removed as noise because the sky does not become an obstacle. Then, the distance data D15 after noise removal will be used for the building limit determination.

Here, it is assumed that noise other than the static obstacles shown in Table 1 is removed, but dynamic obstacles are not removed, and the measurement target is made in label units or noise. You can also decide whether to do it. Further, in the case of noise removal, in addition to the noise removal based on the label data D14 described above, the median filter, the non-local mean filter, and the lumpiness of the distance data, which are general noise removal methods in image processing, are evaluated. Noise removal may also be performed by any of the geometric filters that remove the isolated points. Further, as the post-processing for improving the accuracy of noise removal, the above-mentioned methods such as MRF and CRF may be used.

The distance data integration unit 46 uses all of the distance data D15 (total distance data D15) calculated from the surveillance image data D11 captured at different points, and uses a method called an ICP (Iterative Closest Point) algorithm (Non-Patent Document 4). ) Is used to integrate the distance, obtain the three-dimensional environment map data D16 and the vehicle position / attitude data D17 at each imaging point, and store the three-dimensional environment map data D16 and the vehicle position / attitude data D17 in the storage unit 49. There is.

Specifically, the surveillance image data D11 captured at a certain point is used as a reference, a plurality of points of the reference surveillance image data D11 are extracted, the three-dimensional coordinates of the plurality of points are used as initial values, and then the three-dimensional coordinates of the plurality of points are used as initial values. In the monitored image data D11 of the integrated target captured at different points, the points corresponding to the above-mentioned plurality of points are obtained, and the corresponding points are aligned in the two monitored image data D11 using the ICP algorithm. The two surveillance image data D11 are integrated. By doing this for all the surveillance image data D11, the all-distance data D15 will be integrated, and the three-dimensional environment map data D16 will be obtained.

That is, in the distance data integration unit 46, in order to determine the building limit in the entire determination target section, the total distance data D15 calculated from the monitoring image data D11 captured at different points is integrated, thereby three-dimensionally. The three-dimensional environment map data D16, which is a map of the above, is obtained. With the three-dimensional environment map data D16 obtained in this way, it is possible to grasp an object (for example, a natural object) having no design value.

Further, when the distance data D15 is integrated (aligned) using the ICP algorithm, the relative position / orientation of the distance data D15 is obtained. Therefore, by knowing this position / orientation, the train for each imaging is performed. The vehicle position / attitude data D17 (traveling amount, speed, vehicle tilt) of 21 is also obtained. This vehicle inclination is a very important factor in determining the building limit area.

The ICP algorithm is a method of finding a solution by iterative calculation, and the result depends on the initial value. Therefore, it is necessary to set a correct initial value, but here, by utilizing the fact that the vehicle 21 moves only on the rail 11, it becomes easy to set a good initial value.

In the building limit area calculation unit 47, the rail position D13 at a plurality of imaging points, the vehicle position / attitude data D17 at a plurality of imaging points (vehicle tilt at a plurality of imaging points), and static values given in advance as fixed values are set. Building limit area data D18 is calculated using a specific building limit area, and the building limit area data D18 is stored in the storage unit 49. That is, it is possible to calculate the dynamic building limit area data D18 for the train 21.

The separation calculation unit 48 calculates the separation distance between these data using the three-dimensional environment map data D16 and the building limit area data D18, and stores the separation distance as the separation data D19 in the storage unit 49. For example, when there is an obstacle in the three-dimensional environment map data D16, the separation distance from the building limit area data D18 considering the vehicle inclination to the corresponding obstacle is calculated. By calculating the separation data D19 to the obstacle in this way, the separation determination between the building limit region and the obstacle can be performed, and the separation determination robust to noise becomes possible.

In the storage unit 49, the above-mentioned learning model D3A, monitoring image data D11, distance data D12A, rail position D13, label data D14, distance data D15, three-dimensional environment map data D16, vehicle position / attitude data D17, building limit area data D18. , The separation data D19 is stored.

Next, regarding the building limit determination device of the present embodiment having the above-described configuration, the procedure of the building limit determination method will be described with reference to FIG. Here, FIG. 5 is a flowchart illustrating an example of a building limit determination method in the processing unit shown in FIG. 2. Since the learning model D3A is required for label estimation, it is assumed that the learning model D3A is prepared in advance.

(Step S1)
The surveillance image input unit 41 inputs front (or rear) surveillance image data D11 captured by a plurality of front cameras 22a and 22b (or a plurality of rear cameras 23a and 23b).

(Step S2)
The distance data calculation unit 42A uses the input monitoring image data D11 to calculate the distance data D12A of the object by the block matching and graph cut methods described above together with the stereo measurement.

(Step S3)
The rail recognition unit 43A uses the input monitoring image data D11, performs rail recognition by matching using the matching data described above, and calculates the rail position D13.

(Step S4A)
The label estimation unit 44A uses the input monitoring image data D11 and the learning model D3A, and acquires the label data D14 of each pixel by the label estimation by the Semantic Segmentation described above.

(Step S5)
The noise removing unit 45A removes noise from the distance data D12A based on the label data D14 of each pixel of the monitoring image data D11, and acquires it as the distance data D15.

(Step S6)
The processing unit 30A proceeds to step S8 when the imaging by the front plurality of cameras 22a and 22b (or the rearward plurality of cameras 23a and 23b) is completed, and proceeds to step S7 when the imaging is not completed. move on.

(Step S7)
If the imaging is not completed, the front (or rear) surveillance image data D11 newly captured by the front multiple cameras 22a, 22b (or the rear multiple cameras 23a, 23b) is input (monitoring). Image input unit 41) returns to step S2. That is, steps S2 to S7 are repeatedly executed until the imaging is completed, and the distance data D12A, the rail position D13, the label data D14, and the distance data D15 are obtained for the monitoring image data D11 imaged from different points.

(Step S8)
The distance data integration unit 46 integrates the total distance data D15 calculated from each monitoring image data D11 using the ICP algorithm described above to obtain the three-dimensional environment map data D16 and the vehicle position / attitude data D17. ..

(Step S9)
The building limit area calculation unit 47 calculates the building limit area data D18 based on the rail position D13, the vehicle position / attitude data D17 (vehicle inclination), and the static building limit area.

(Step S10)
The separation calculation unit 48 calculates the separation distance from the building limit area data D18 at each point to the nearest obstacle (for example, the utility pole 12) as the separation data D19. Then, based on the calculated separation data D19, the separation determination between the building limit area and the obstacle is performed.

As described above, the building limit determination device of the present embodiment can remove noise based on the label of the label data D14 of the pixel for the distance data D12A of each pixel initially calculated, so that it is a simple outlier. It is possible to remove noise with higher accuracy than the noise removal method such as removing noise. As a result, in the determination of the separation between the building limit area and the obstacle, it is possible to make a robust and highly accurate determination against noise.

In addition, by selecting the size, feature amount, and classifier of the local patch image used for label estimation according to the image, it is possible to perform processing specialized for that image, and label estimation is more robust than learning all. It can be carried out.

[Example 2]
The building limit determination device of this embodiment will be described with reference to FIGS. 6 and 7. Here, FIG. 6 is a block diagram illustrating another example of the processing unit constituting the building limit determination device shown in FIG. 1. Further, FIG. 7 is a flowchart illustrating an example of a building limit determination method in the processing unit shown in FIG.

The building limit determination device of this embodiment basically has the same configuration as the building limit determination device described in the first embodiment, but has a partial configuration of the processing unit 30B of the building limit determination device of this embodiment. However, it is different from the processing unit 30A of the building limit determination device of the first embodiment. Therefore, in the present embodiment, the same reference numerals are given to the configurations equivalent to those shown in the first embodiment, and the description overlapping with the first embodiment will be omitted.

The building limit determination device of this embodiment uses the processing unit 30B shown in FIG. 6 instead of the processing unit 30A shown in FIGS. 1 and 2. As shown in FIG. 6, the processing unit 30B includes a label teacher data unit 31, a label learning unit 32B, a monitoring image input unit 41, a distance data calculation unit 42A, a rail recognition unit 43A, and a label estimation unit 44B. It has a noise removing unit 45A, a distance data integration unit 46, a building limit area calculation unit 47, a separation calculation unit 48, and a storage unit 49.

That is, the processing unit 30B in the present embodiment is different from the processing unit 30A in the first embodiment in the label learning unit 32B and the label estimation unit 44B. On the other hand, in this embodiment, the label teacher data unit 31, the monitoring image input unit 41, the distance data calculation unit 42A, the rail recognition unit 43A, the noise removal unit 45A, the distance data integration unit 46, the building limit area calculation unit 47, and the separation calculation unit. The structure of the 48 and the storage unit 49 is the same as that of the first embodiment.

In the processing unit 30B, the label learning unit 32B uses the teacher image data D1 and the teacher label data D2 in the label teacher data unit 31 and learns by the supervised learning of the machine learning algorithm, and the image of the teacher image data D1. A learning model D3B that associates each pixel of the group with the label of the teacher label data D2 is created, and the created learning model D3B is stored in the storage unit 49.

In the label learning unit 32B, as described in Non-Patent Documents 1 and 2, information (for example, color information) of each pixel of the image is directly input to the machine learning algorithm, and the label is directly estimated for each pixel. A deep learning method called FCN (Fully Convolutional Networks) is used.

Specifically, the feature amount is extracted from each pixel of the image group of the teacher image data D1 by the above-mentioned FCN and Semantic Segmentation, and based on the extracted feature amount, each pixel of the image group of the teacher image data D1 is used. By learning the correspondence with the label of the teacher label data D2 attached to each pixel in advance and defining a function to output the label in response to the input of the feature amount, each pixel of the image group of the teacher image data D1 and the teacher A learning model D3B that associates with the label of the label data D2 is created. That is, unlike the label learning unit 32A in the first embodiment, the local patch image is not cut out, and the feature amount is also extracted by the FCN, which is called end-to-end learning. ing.

Further, the label estimation unit 44B estimates the label for each pixel of the monitoring image data D11 by the learning model D3B, obtains the estimated label as the label data D14, and stores the obtained label data D14 in the storage unit 49. ..

Specifically, the feature amount is extracted from each pixel of the monitoring image data D11 by the above-mentioned Semantic Segmentation and the learning model D3B in the storage unit 49, and the label is estimated for each pixel based on the extracted feature amount. , The probability (probability) of each label for each pixel is obtained, and the label data D14 is obtained with the label having a high probability as the label for each pixel. Further, as the post-processing for improving the accuracy of the label estimation, MRF or CRF may be used as in the label estimation unit 44A in the first embodiment.

As described above, the label estimation unit 44B estimates the label of each pixel in the same manner as the label estimation unit 44A in the first embodiment, but uses the learning model D3B created by the FCN and Semantic Segmentation. Therefore, unlike the label estimation unit 44A in the first embodiment, the local patch image is not cut out, and the feature amount is also extracted by the FCN. Therefore, it is possible to prevent a situation in which correct processing cannot be performed in the middle of the Semantic Segmentation of the multi-stage processing as in the first embodiment (the size of the cutout of the local patch image is not appropriate, the extraction of the feature amount is not appropriate, etc.).

In the storage unit 49, as in the first embodiment, the monitoring image data D11, the distance data D12A, the rail position D13, the label data D14, the distance data D15, the three-dimensional environment map data D16, the vehicle position / attitude data D17, and the building limit. The area data D18 and the separation data D19 are stored, but the learning model D3B is stored instead of the learning model D3A in the first embodiment.

Next, regarding the building limit determination device of the present embodiment having the above-described configuration, the procedure of the building limit determination method will be described with reference to FIG. 7. Also here, since the learning model D3B is required for label estimation, it is assumed that the learning model D3B is prepared in advance.

(Steps S1 to S3)
In the building limit determination method in the present embodiment, steps S1 to S3 are the same as steps S1 to S3 in the first embodiment.

(Step S4B)
The label estimation unit 44B uses the input monitoring image data D11 and the learning model D3B, and acquires the label data D14 of each pixel by the label estimation by the Semantic Segmentation described above.

(Steps S5 to S10)
In the building limit determination method in the present embodiment, steps S5 to S10 are the same as steps S5 to S10 in the first embodiment.

That is, the building limit determination device of this embodiment is also simple because noise can be removed from the initially calculated distance data D12A of each pixel based on the label of the label data D14 of the pixel, as in the first embodiment. It is possible to remove noise with higher accuracy than the noise removal method such as removing outliers. As a result, in the determination of the separation between the building limit area and the obstacle, it is possible to make a robust and highly accurate determination against noise.

Further, in this embodiment, it is not necessary to explicitly select the size, feature amount, and classifier of the local patch image in the label estimation, and high-precision label estimation is performed by end-to-end learning by the above FCN. It can be performed.

[Example 3]
The building limit determination device of this embodiment will be described with reference to FIGS. 8 and 9. Here, FIG. 8 is a block diagram illustrating another example of the processing unit constituting the building limit determination device shown in FIG. 1. Further, FIG. 9 is a flowchart illustrating an example of a building limit determination method in the processing unit shown in FIG.

The building limit determination device of this embodiment has basically the same configuration as the building limit determination device described in the first embodiment, but has a partial configuration of the processing unit 30C of the building limit determination device of this embodiment. However, it is different from the processing unit 30A of the building limit determination device of the first embodiment. Therefore, also in this embodiment, the same reference numerals are given to the configurations equivalent to those shown in the first embodiment, and the description overlapping with the first embodiment is omitted.

The building limit determination device of this embodiment uses the processing unit 30C shown in FIG. 8 instead of the processing unit 30A shown in FIGS. 1 and 2. As shown in FIG. 8, the processing unit 30C includes a label teacher data unit 31, a label learning unit 32A, a monitoring image input unit 41, a distance data calculation unit 42B, a rail recognition unit 43B, and a label estimation unit 44A. It has a noise removing unit 45B, a distance data integration unit 46, a building limit area calculation unit 47, a separation calculation unit 48, and a storage unit 49.

That is, the processing unit 30C in this embodiment is different from the processing unit 30A in the first embodiment in the distance data calculation unit 42B, the rail recognition unit 43B, and the noise removal unit 45B. In the first embodiment, noise is removed from the distance data D12A of the pixel based on the label of the label data D14 of each pixel, whereas in this embodiment, the label data D14 of each pixel is removed. This is because the distance data D12B of the pixel is calculated and the rail position D13 is calculated based on the label.

Further, as can be seen from the comparison between FIG. 5 described above and FIG. 9 described later, in the first embodiment, noise removal (step S5) is performed after the distance data calculation (step S2) and the rail recognition (step S3). ), The label estimation (step S4A) is performed, whereas in this embodiment, the label estimation (step S12A) is performed before the distance data calculation (step S13) and the rail recognition (step S14). The processing procedure of this embodiment is also different from that of the first embodiment.

On the other hand, regarding the label teacher data unit 31, the label learning unit 32A, the monitoring image input unit 41, the label estimation unit 44A, the distance data integration unit 46, the building limit area calculation unit 47, the separation calculation unit 48, and the storage unit 49 in this embodiment. Is equivalent to the configuration in Example 1.

In the processing unit 30C, the distance data calculation unit 42B uses the label data D14 in addition to the plurality of surveillance image data D11 captured at the same time to perform stereo correspondence, and the measurement points searched by this stereo correspondence are performed. Stereo measurement is performed by performing triangular measurement, thereby calculating distance data D12B (first distance data) to an object including an obstacle, and storing the calculated distance data D12B in the storage unit 49. is doing.

That is, in the distance data calculation unit 42B, in addition to the pixel value information of each pixel of the plurality of monitoring image data D11, the label information of the label data D14 is also used to generate the corresponding pixels among the monitoring image data D11. It is searched as a measurement point, and triangular survey is performed for the distance to each searched measurement point.

Specifically, when supporting stereo, not only the pixel value information (for example, color information, brightness information, etc.) but also the label information (decision value of label type or likelihood of each label) is applied to each pixel. ) Is also given. For example, in the case of only color information, it becomes RGB three-dimensional information, but by having label information, it becomes higher dimensional information. More specifically, using the labels illustrated in Table 1, it is possible to obtain a total of 30 dimensions of RGB 3 dimensions + 27 types of label information and 27 dimensions. By providing such high-dimensional information, it is possible to reduce erroneous correspondence in the case of stereo correspondence.

Further, in a certain pixel, when the likelihood of a label of the type to be noise is higher than the likelihood of another label, the distance data D12B is deleted at the time of calculation of the distance data D12B of the pixel. The low noise distance data D12B can be calculated at the stage of the distance data calculation unit 42B.

As described above, the distance data calculation unit 42B is characterized in that not only the pixel value information but also the label information is used for stereo measurement, and the label information is used for noise removal.

Further, the rail recognition unit 43B also uses the monitoring image data D11 and the label data D14 in the same manner as the distance data calculation unit 42B, and the monitoring image data D11 is based on the matching data and the label data D14 of the rail 11 prepared in advance. The position of the rail 11 is recognized from the above, the position is calculated as the rail position D13, and the rail position D13 is stored in the storage unit 49.

Specifically, in the monitoring image data D11, matching is performed by raster scanning using matching data (rail template image data and reference rail luminance distribution data) for an arbitrary horizontal line area. At the time of matching, label information is given to each pixel in the area of an arbitrary horizontal line. Then, the position that most matches the matching data and has a high likelihood of the "rail" as the label information is calculated as the rail position D13. In this way, the position of the rail 11 in the monitoring image data D11 is recognized, and the rail position D13 can be accurately detected from the monitoring image data D11.

Further, in the noise removing unit 45B, the noise of the distance data D12B is removed by any of a median filter, a non-local mean filter, and a geometric filter, and the noise-removed distance data D12B is used as the distance data D15 (second distance data). It is stored in the storage unit 49. As described above, the noise removing unit 45B uses a general noise removing method in image processing. However, since the noise removal using the label information is performed by the distance data calculation unit 42B described above, the noise removal unit 45B is not an indispensable configuration, and if this configuration is not provided, the distance data integration unit 46 will perform the noise removal. Instead of the distance data D15, the distance data D12B may be used to calculate the three-dimensional environment map data D16 and the vehicle position / attitude data D17.

In the storage unit 49, as in the first embodiment, the learning model D3A, the monitoring image data D11, the rail position D13, the label data D14, the distance data D15, the three-dimensional environment map data D16, the vehicle position / attitude data D17, and the building limit. The area data D18 and the separation data D19 are stored, but the distance data D12B is stored instead of the distance data D12A in the first embodiment.

Next, the building limit determination device of the present embodiment having the above-described configuration will be described with reference to FIG. 9 for the procedure of the building limit determination method. Also here, since the learning model D3A is required for label estimation, it is assumed that the learning model D3A is prepared in advance.

(Step S11)
In the building limit determination method in the present embodiment, step S11 is the same as step S1 in the first embodiment.

(Step S12A)
The label estimation unit 44A uses the input monitoring image data D11 and the learning model D3A, and acquires the label data D14 of each pixel by the label estimation by the Semantic Segmentation described above.

(Step S13)
The distance data calculation unit 42B uses the input monitoring image data D11 and the label data D14 of each pixel of the monitoring image data D11, and calculates the distance data D12B of the object by using the label information together with the stereo measurement.

(Step S14)
The rail recognition unit 43B uses the input monitoring image data D11 and the label data D14 of each pixel of the monitoring image data D11, and performs rail recognition by matching and label information using the matching data described above. The rail position D13 is calculated.

(Step S15)
The noise removing unit 45B removes noise from the distance data D12B by a general noise removing method in image processing, and acquires it as the distance data D15. However, as described above, the procedure of step S15 by the noise removing unit 45B may not be necessary.

(Step S16)
The processing unit 30C proceeds to step S18 when the imaging by the front plurality of cameras 22a and 22b (or the rearward plurality of cameras 23a and 23b) is completed, and proceeds to step S17 when the imaging is not completed. move on.

(Step S17)
If the imaging is not completed, the front (or rear) surveillance image data D11 newly captured by the front multiple cameras 22a, 22b (or the rear multiple cameras 23a, 23b) is input (monitoring). Image input unit 41) returns to step S12A. That is, steps S12A to S17 are repeatedly executed until the imaging is completed, and the distance data D12B, the rail position D13, the label data D14, and the distance data D15 are obtained for the monitoring image data D11 imaged from different points.

(Steps S18 to S20)
In the building limit determination method in the present embodiment, steps S18 to S20 are the same as steps S8 to S10 in the first embodiment.

As described above, the building limit determination device of this embodiment takes into account not only the pixel value information but also the label information of the pixel when measuring the stereo, and thus supports stereo, so that noise is reduced. Robust distance data D12B can be calculated. As a result, in the determination of the separation between the building limit area and the obstacle, it is possible to make a robust and highly accurate determination against noise.

In addition, by selecting the size, feature amount, and classifier of the local patch image used for label estimation according to the image, it is possible to perform processing specialized for that image, and label estimation is more robust than learning all. It can be carried out.

[Example 4]
The building limit determination device of this embodiment will be described with reference to FIGS. 10 and 11. Here, FIG. 10 is a block diagram illustrating another example of the processing unit constituting the building limit determination device shown in FIG. 1. Further, FIG. 11 is a flowchart illustrating an example of a building limit determination method in the processing unit shown in FIG.

The building limit determination device of this embodiment basically has the same configuration as the building limit determination device described in the third embodiment, but has a partial configuration of the processing unit 30D of the building limit determination device of this embodiment. However, it is different from the processing unit 30C of the building limit determination device of the third embodiment, and the different configuration is the same as the partial configuration of the processing unit 30B of the building limit determination device of the second embodiment. There is. Therefore, in this embodiment, the same reference numerals are given to those having the same configurations as those shown in Examples 1 to 3, and the description overlapping with Examples 1 to 3 will be omitted.

The building limit determination device of this embodiment uses the processing unit 30D shown in FIG. 10 instead of the processing unit 30A shown in FIGS. 1 and 2. As shown in FIG. 10, the processing unit 30D includes a label teacher data unit 31, a label learning unit 32B, a monitoring image input unit 41, a distance data calculation unit 42B, a rail recognition unit 43B, and a label estimation unit 44B. It has a noise removing unit 45B, a distance data integration unit 46, a building limit area calculation unit 47, a separation calculation unit 48, and a storage unit 49.

That is, the processing unit 30D in the present embodiment is different from the processing unit 30C in the third embodiment in the label learning unit 32B and the label estimation unit 44B, but these are the label learning in the processing unit 30B in the second embodiment. It has the same configuration as the unit 32B and the label estimation unit 44B. Further, in this embodiment, the label teacher data unit 31, the monitoring image input unit 41, the distance data calculation unit 42B, the rail recognition unit 43B, the noise removal unit 45B, the distance data integration unit 46, the building limit area calculation unit 47, and the separation calculation unit The structure of the 48 and the storage unit 49 is the same as that of the third embodiment.

As for the storage unit 49, as in the third embodiment, the monitoring image data D11, the distance data D12B, the rail position D13, the label data D14, the distance data D15, the three-dimensional environment map data D16, the vehicle position / attitude data D17, and the building. The limit area data D18 and the separation data D19 are stored, but the learning model D3B is stored instead of the learning model D3A in the third embodiment.

In this embodiment, unlike the label learning unit 32A and the label estimation unit 44A in the first and third embodiments, the local patch image is not cut out and the second embodiment is configured as described above. Similar to the label learning unit 32B and the label estimation unit 44B in the above, the FCN described above extracts the feature amount.

Next, the building limit determination device of the present embodiment having the above-described configuration will be described with reference to FIG. 11 for the procedure of the building limit determination method. Also here, since the learning model D3B is required for label estimation, it is assumed that the learning model D3B is prepared in advance.

(Step S11)
In the building limit determination method in the present embodiment, step S11 is the same as step S11 in the third embodiment.

(Step S12B)
The label estimation unit 44B uses the input monitoring image data D11 and the learning model D3B, and acquires the label data D14 of each pixel by the label estimation by the Semantic Segmentation described above.

(Steps S13 to S20)
In the building limit determination method in the present embodiment, steps S13 to S20 are the same as steps S13 to S20 in the third embodiment.

That is, the building limit determination device of this embodiment also supports stereo by taking into account not only the pixel value information but also the label information of the pixel at the time of stereo measurement, as in the third embodiment. , It is possible to calculate the distance data D12B that is robust against noise. As a result, in the determination of the separation between the building limit area and the obstacle, it is possible to make a robust and highly accurate determination against noise.

Further, in this embodiment, it is not necessary to explicitly select the size, feature amount, and classifier of the local patch image in the label estimation, and high-precision label estimation is performed by end-to-end learning by the above FCN. It can be performed.

INDUSTRIAL APPLICABILITY The present invention is suitable for determining the distance between a building limit area and an obstacle for a vehicle traveling on a rail.

11 Rail 12 Electric pole 13 Ground 14 Sky 21 Train 22a, 22b, 23a, 23b Cameras 30A, 30B, 30C, 30D Processing unit 31 Label teacher data unit 32A, 32B Label learning unit 41 Monitoring image input unit 42A, 42B Distance data calculation unit 43A, 43B Rail recognition unit 44A, 44B Label estimation unit 45A, 45B Noise removal unit 46 Distance data integration unit 47 Building limit area calculation unit 48 Separation calculation unit 49 Storage unit

Claims (8)

A plurality of cameras provided on a vehicle traveling on rails to capture the front or rear of the vehicle, and
It has a processing unit that performs image processing of image data captured by the plurality of cameras.
The processing unit
A label teacher data unit that stores teacher image data composed of a plurality of images similar to the image data and teacher label data in which labels are given in advance to each pixel of the plurality of images of the teacher image data.
A label learning unit that preliminarily creates a learning model that associates each pixel of the plurality of images of the teacher image data with the label of the teacher label data by learning by machine learning.
Using the plurality of image data captured at the same time, the corresponding pixel among the plurality of image data is searched as a measurement point from the information of the pixel value of each pixel of the plurality of image data, and the measurement point is reached. A distance data calculation unit that calculates the distance by stereo measurement with the distance as the first distance data,
A rail recognition unit that recognizes the position of the rail from the image data based on the matching data of the rail prepared in advance and calculates the position as the rail position.
A label estimation unit that estimates the label for each pixel of the image data by the learning model and obtains the estimated label as label data.
A noise removing unit that performs noise removal of the first distance data based on the label data for each pixel of the image data and obtains the noise-removed first distance data as the second distance data.
A distance data integration unit that integrates the second distance data of the image data captured at a plurality of points to obtain a three-dimensional environment map and obtains a vehicle inclination, and a distance data integration unit.
A building limit area that calculates a dynamic building limit area for the vehicle based on the rail position at multiple points, the vehicle tilt at multiple points, and the static building limit area given as a fixed value. Calculation unit and
It has a separation calculation unit that calculates the separation distance from the dynamic building limit area to the nearest object based on the dynamic building limit area and the three-dimensional environment map, and determines the separation. A characteristic building limit judgment device. In the building limit determination device according to claim 1,
When removing noise from the first distance data, the noise removing unit further removes noise by using any of a median filter, a non-local mean filter, and a geometric filter, and the second distance. A building limit determination device characterized by obtaining data. A plurality of cameras provided on a vehicle traveling on rails to capture the front or rear of the vehicle, and
It has a processing unit that performs image processing of image data captured by the plurality of cameras.
The processing unit
A label teacher data unit that stores teacher image data composed of a plurality of images similar to the image data and teacher label data in which labels are given in advance to each pixel of the plurality of images of the teacher image data.
A label learning unit that preliminarily creates a learning model that associates each pixel of the plurality of images of the teacher image data with the label of the teacher label data by learning by machine learning.
A label estimation unit that estimates the label for each pixel of the image data by the learning model and obtains the estimated label as label data.
Using the plurality of image data captured at the same time and the label data for each pixel of the plurality of image data, the information of the pixel value of each pixel of the plurality of image data and the space between the plurality of image data from the label data. A distance data calculation unit that searches for the corresponding pixel as a measurement point and calculates the distance to each measurement point as the first distance data by stereo measurement.
Using the image data and the label data for each pixel of the image data, the position of the rail is recognized from the image data by the matching data of the rail and the label data prepared in advance, and the position is set as the rail position. The rail recognition unit to calculate and
A distance data integration unit that integrates the first distance data of the image data captured at a plurality of points to obtain a three-dimensional environment map and obtains a vehicle inclination, and a distance data integration unit.
A building limit area that calculates a dynamic building limit area for the vehicle based on the rail position at multiple points, the vehicle tilt at multiple points, and the static building limit area given as a fixed value. Calculation unit and
It has a separation calculation unit that calculates the separation distance from the dynamic building limit area to the nearest object based on the dynamic building limit area and the three-dimensional environment map, and determines the separation. A characteristic building limit judgment device. In the building limit determination device according to claim 3,
The processing unit uses any of a median filter, a non-local mean filter, and a geometric filter to remove noise from the first distance data, and the noise-removed first distance data is used as second distance data. Further has a noise removing part to be obtained as
The distance data integration unit integrates the second distance data of the image data captured at a plurality of points to obtain the three-dimensional environment map and the vehicle inclination. Device. In the building limit determination device according to any one of claims 1 to 4.
The label learning unit is a local area centered on each pixel from the plurality of images of the teacher image data by the machine learning using any of SVM (Support Vector Machine), Random Forest, or deep learning and Semantic Segmentation. The image of the range is cut out, the feature amount is extracted from the image of the local range, and the correspondence between the central pixel of the image of the local range and the label of the teacher label data is learned based on the feature amount. Create a learning model and
The label estimation unit cuts out an image of a local range centered on each pixel from the image data by the Semantic Segmentation and the learning model, extracts a feature amount from the image of the local range, and bases the feature amount on the feature amount. The building limit determination device is characterized in that the label is estimated for the pixel at the center of the image in the local range, and the label is estimated for each pixel of the image data. In the building limit determination device according to claim 5.
When extracting the feature amount, the label learning unit and the label estimation unit perform a color feature amount, a HOG (Histograms of Oriented Gradients) feature amount, a SIFT (Scale-Invariant Feature Transform) feature amount, and a BoVW (Bag of Visual). Words) A building limit determination device characterized by using either a feature amount or a Texton feature amount, or a combination thereof. In the building limit determination device according to any one of claims 1 to 4.
The label learning unit extracts a feature amount from each pixel of the plurality of images of the teacher image data by the machine learning using FCN (Fully Convolutional Networks) and Semantic Segmentation, and based on the feature amount, The learning model is created by learning the correspondence between each pixel of the plurality of images of the teacher image data and the label of the teacher label data.
The label estimation unit extracts a feature amount from each pixel of the image data by the Semantic Segmentation and the learning model, and estimates the label for each pixel of the image data based on the feature amount. A building limit judgment device characterized by. In the building limit determination device according to any one of claims 1 to 7.
The label estimation unit is a building limit determination device characterized in that an MRF (Markov Random Field) or a CRF (Conditional Random Field) is used when estimating the label.

Images