JP7467299B2 - Location management system, location identification device, and location identification method - Google Patents

Description

Embodiments of the present invention relate to a location management system, a location identification device, and a location identification method.

In recent years, for example, to realize dynamic maps for use in MaaS (Mobility as a Service), it is necessary to measure and manage accurate position information of moving objects such as vehicles. When obtaining vehicle position information, it is possible to obtain more accurate position information by using the Global Navigation Satellite System (GNSS) rather than the Global Positioning System (GPS).

However, even when GNSS is used, there are problems, such as large errors in location information in areas with many tall buildings. Therefore, to realize advanced autonomous driving, it is necessary to reduce position measurement errors, and as a solution, for example, it is necessary to install high-precision positioning devices in vehicles.

JP 2018-84533 A JP 2018-32433 A JP 2020-30362 A JP 2018-81252 A

However, the problem with the high-precision positioning devices mentioned above is that they are expensive.

The problem that the present invention aims to solve is to provide a location management system, a location identification device, and a location identification method that can obtain highly accurate location information of moving objects at low cost.

The position management system of the embodiment includes a fixed camera that captures an image of a specific area in which a moving object moves, a storage unit that stores positional correspondence information between an image captured by the fixed camera and an overhead image of the specific area captured from above and associated with highly accurate position information, an acquisition unit that acquires the captured image from the fixed camera, a detection unit that detects a moving object captured in the captured image, and a control unit that refers to the correspondence information and identifies the position information of the moving object.

FIG. 1 is a schematic diagram for explaining the main differences between the embodiment and the comparative example. FIG. 2 is a diagram showing the overall configuration of the location management system according to the embodiment. FIG. 3 is a diagram for explaining an outline of the process performed by the PC according to the embodiment. FIG. 4 is a diagram for explaining the correspondence between the stationary camera image and the satellite image in the embodiment. FIG. 5 is a diagram showing a conversion table according to the embodiment. FIG. 6 is a flowchart showing a process performed by the PC according to the embodiment. FIG. 7 is a flow chart showing the details of the process in step S3 of FIG. FIG. 8 is a flowchart showing a process performed by the location management server according to the embodiment. FIG. 9 is a flowchart showing the details of the process in step S43 of FIG. FIG. 10 is a flowchart illustrating another process performed by the location management server according to the embodiment. FIG. 11 is a diagram for explaining another example of calculation of the center coordinates of the vehicle in the embodiment. FIG. 12 is a flowchart showing a process of calculating the position information of the point P3 by the PC according to the embodiment.

Embodiments of the position management system, position identification device, and position identification method of the present invention will be described below with reference to the drawings. In the following, a vehicle will be used as an example of a moving object.

First, referring to FIG. 1, a general difference between the embodiment and a comparative example (prior art) will be described.
1 is a schematic diagram for explaining the main differences between the embodiment and a comparative example (conventional technology). In the comparative example (conventional technology), a dynamic map is created using position information from a high-precision positioning device mounted on a vehicle. In this case, highly accurate position information of the vehicle can be obtained in all areas in which the vehicle travels, but there is a problem that the high-precision positioning device is expensive.

In contrast, in this embodiment, we will assume that a dynamic map is created using position information from the proposed model described below. In the proposed model, roads are classified into two types of areas, and the vehicle position measurement accuracy is classified into different levels. One type is an area that requires highly accurate position measurement, such as an intersection, and is the target of the proposed model. The other type is an area that does not require highly accurate position measurement, such as a straight road, and is not the target of the proposed model, and uses position information from the GPS sensor of each vehicle. With this method, highly accurate vehicle position information can be obtained at low cost in high-accuracy position measurement areas. This will be described in detail below.

2 is a diagram showing the overall configuration of a location management system S according to an embodiment of the present invention. The location management system S includes a location management server 1, a PC (Personal Computer) 2, a fixed camera 3, and a map management server 4 (management server).
The fixed camera 3 captures an image of a predetermined area (e.g., a road intersection) in which a vehicle 5 (hereinafter also referred to as "vehicle" without a reference number) moves. More specifically, the fixed camera 3 is installed, for example, at a high position in a major location on the road, with an angle of view overlooking the road from above. The range of the field of view of this fixed camera 3 is a high-precision position measurement area. Furthermore, the fixed cameras 3 are installed at various points on the road, and one PC 2 is installed for each fixed camera 3.

Here, an overview of the process performed by the PC 2 according to the embodiment will be described with reference to FIG.
3A and 3B are diagrams for explaining an outline of the process performed by the PC 2 according to the embodiment of the present invention, and FIG 3A is an example of an image obtained by the fixed camera 3 (hereinafter, also referred to as a fixed camera image).

As shown in FIG. 3(a), PC2 detects vehicles from images captured by a fixed camera. In FIG. 3(b), the detected vehicles are surrounded by a rectangle. PC2 then identifies location information for the detected vehicles. FIG. 3(c) shows diagrammatically how the vehicle location information is identified based on the center (intersection of diagonals) of the rectangle surrounding the vehicle (details will be described later).

Returning to FIG. 2, the PC 2 includes a processing unit 21, a memory unit 22, an input unit 23, a display unit 24, and a communication unit 25.

The storage unit 22 is a storage device such as a hard disk drive (HDD) or a solid state drive (SSD), and stores various information. For example, the storage unit 22 stores a location database (DB) 221 (correspondence information), a conversion table 222 (correspondence information), and first vehicle information 223.

The location DB 221 is a DB that defines the correspondence between fixed camera images and satellite images (or aerial images, an example of an overhead image). Here, the correspondence between fixed camera images and satellite images in the embodiment will be explained with reference to FIG. 4.

Figure 4 is a diagram for explaining the correspondence between fixed camera images and satellite images in an embodiment. In the position DB 221, for the movable range of the vehicle 5, each pixel in the fixed camera image shown in Figure 4 (a) is associated with each pixel in the satellite image shown in Figure 4 (b). Therefore, by referring to this position DB 221, it is possible to identify pixels in the satellite image that correspond to pixels in the fixed camera image. In addition, this position DB 221 may be realized by a projective transformation matrix generated based on the coordinates of multiple pairs of corresponding points between the fixed camera image and the satellite image.

Returning to FIG. 2, the conversion table 222 is a table in which the coordinates of pixels in a satellite image are associated with the location information of the satellite image. Here, the conversion table 222 of the embodiment will be described with reference to FIG. 5.

Figure 5 is a diagram showing a conversion table 222 according to an embodiment. In the conversion table 222, the coordinates (X coordinate, Y coordinate) of a pixel of a satellite image correspond to highly accurate position information (latitude, longitude, altitude) of the satellite image.

2, the first vehicle information 223 is information about a vehicle detected based on an image captured by a fixed camera. The first vehicle information 223 includes, for example, the following information (1) to (8).
(1) Date and time of shooting (2) Frame number of the fixed camera image (3) Coordinates of the representative points (e.g., upper left and lower right) of the rectangle surrounding the vehicle (4) Coordinates of the center of the rectangle (5) Vehicle type (passenger car/truck/bus, etc.)
(6) Color information of the central region (e.g., RGB (Red, Green, Blue) histogram)
(7) Tracking ID (Identifier) (when tracking a vehicle over time)
(8) Location information (latitude, longitude, altitude)

Of (1) to (8), (1) and (2) are obtained from images taken with a stationary camera. (3) is obtained by processing in the detection unit 212. (4) to (8) are obtained by processing in the control unit 213.

The processing unit 21 controls the overall operation of the PC 2 and realizes various functions of the PC 2. The processing unit 21 includes, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory). The CPU controls the overall operation of the PC 2. The ROM is a storage medium for storing various programs and data. The RAM is a storage medium for temporarily storing various programs and rewriting various data. The CPU executes programs stored in the ROM, the storage unit 22, etc., using the RAM as a work area. The processing unit 21 includes, as functional units, an acquisition unit 211, a detection unit 212, and a control unit 213.

The acquisition unit 211 acquires various information from an external device (e.g., the fixed camera 3, the location management server 1, etc.). The acquisition unit 211 acquires, for example, captured images from the fixed camera 3.

The detection unit 212 detects vehicles captured in stationary camera images based on techniques such as DNN (Deep Neural Network). The detection unit 212 also generates the information (3) described above.

The control unit 213 executes various information processing. For example, the control unit 213 refers to the position DB 221 and the conversion table 222, and identifies the vehicle's position information based on the center position of a rectangle surrounding the vehicle captured in the stationary camera image and the corresponding position in the satellite image. The control unit 213 also generates the above-mentioned information (4) to (8) as appropriate.

The input unit 23 accepts user operations on the PC 2. The input unit 23 is, for example, an input device such as a keyboard or a mouse.

The display unit 24 displays various information. The display unit 24 is, for example, a liquid crystal display (LCD (Liquid Crystal Display)) or an organic EL (Electro-Luminescence) display device.

The communication unit 25 is a communication interface for communicating with external devices (e.g., the fixed camera 3, the location management server 1, etc.).

Next, we will explain the location management server 1. The location management server 1 is a computer device, and includes a processing unit 11, a storage unit 12, an input unit 13, a display unit 14, and a communication unit 15.

The memory unit 12 is a storage device such as an HDD or SSD, and stores various information. For example, the memory unit 12 stores first vehicle information 121, second vehicle information 122, and correspondence information 123.

The first vehicle information 121 is the same as the first vehicle information 223 in PC2.

The second vehicle information 122 is similar to the second vehicle information 51 of the vehicle 5. The second vehicle information 51 includes, for example, the following information (11) to (13).
(11) GPS date and time (12) GPS location information (latitude, longitude, altitude)
(13) Vehicle type (passenger car/truck/bus, etc.)

The correspondence information 123 is information that associates the first vehicle information 121 with the second vehicle information 122 for the same vehicle 5.

The processing unit 11 controls the overall operation of the location management server 1 and realizes the various functions of the location management server 1. The processing unit 11 includes, for example, a CPU, a ROM, and a RAM. The CPU controls the overall operation of the location management server 1. The ROM is a storage medium that stores various programs and data. The RAM is a storage medium for temporarily storing various programs and rewriting various data. The CPU executes programs stored in the ROM, the storage unit 12, etc., using the RAM as a work area. The processing unit 11 includes, as functional units, an acquisition unit 111, a response processing unit 112, and a control unit 113.

The acquisition unit 111 acquires various information from an external device (e.g., PC 2, map management server 4, etc.). For example, the acquisition unit 111 acquires first vehicle information 223 from PC 2 and stores it as first vehicle information 121 in the storage unit 12. The acquisition unit 111 also acquires second vehicle information 51 from vehicle 5 and stores it as second vehicle information 122 in the storage unit 12.

The correspondence processing unit 112 matches the first vehicle information 121 with the second vehicle information 122 for the same vehicle 5, and stores the processing result in the memory unit 12 as correspondence information 123.

The control unit 113 executes various information processes. For example, the control unit 113 transmits information such as "(8) Position information (latitude, longitude, altitude)" in the first vehicle information 121 to the map management server 4. This enables the map management server 4 to update the dynamic map 41 using the highly accurate "(8) Position information (latitude, longitude, altitude)" as auxiliary information.

The control unit 113 also transmits information such as "(8) location information (latitude, longitude, altitude)" in the first vehicle information 121 to the corresponding vehicle 5. This allows the vehicle 5 to utilize the highly accurate "(8) location information (latitude, longitude, altitude)".

The input unit 13 accepts user operations on the location management server 1. The input unit 13 is, for example, an input device such as a keyboard or a mouse.

The display unit 14 displays various information. The display unit 14 is, for example, a liquid crystal display (LCD) or an organic electroluminescence display device.

The communication unit 15 is a communication interface for communicating with an external device (e.g., PC 2, map management server 4, etc.).

Next, the process performed by the PC 2 according to the embodiment will be described with reference to FIG.
FIG. 6 is a flowchart showing the process performed by the PC 2 according to the embodiment.
First, in step S<b>1 , the acquisition unit 211 acquires a fixed camera image from the fixed camera 3 .

Next, in step S2, the detection unit 212 detects vehicles (moving objects) captured in the stationary camera images based on techniques such as DNN.

Next, in step S3, the control unit 213 identifies the position information of the vehicle (moving object). Here, the details of the process of step S3 in FIG. 6 will be described with reference to FIG. 7.

FIG. 7 is a flow chart showing the details of the process in step S3 of FIG.
In step S31, the control unit 213 refers to the position DB 221 and performs coordinate conversion of the fixed camera image into a satellite image (FIG. 4).

Next, in step S32, the control unit 213 calculates the center coordinates of the vehicle (moving object) in the satellite image (FIG. 3(c)). Next, in step S33, the control unit 213 refers to the conversion table 222 and converts the center coordinates of the vehicle in the satellite image into position information (latitude, longitude, altitude).

Returning to FIG. 6, after step S3, in step S4, the control unit 213 generates the first vehicle information 223. Next, in step S5, the control unit 213 transmits the first vehicle information 223 to the location management server 1.

Next, the process performed by the location management server 1 according to the embodiment will be described with reference to FIG.
FIG. 8 is a flowchart showing the process performed by the location management server 1 according to the embodiment.
First, in step S41 , the acquisition unit 111 acquires the first vehicle information 223 from the PC 2 and stores it as the first vehicle information 121 in the storage unit 12 .

Next, in step S42, the acquisition unit 111 acquires second vehicle information 51 from the vehicle 5 and stores it as second vehicle information 122 in the memory unit 12.

Next, in step S43, the correspondence processing unit 112 associates the first vehicle information 121 with the second vehicle information 122 for the same vehicle 5, and stores the processing result in the storage unit 12 as the correspondence information 123. Here, the details of the processing of step S43 in FIG. 8 will be described with reference to FIG. 9.

FIG. 9 is a flowchart showing the details of the process in step S43 of FIG.
In step S431, the response processing unit 112 uses a plurality of pieces of first vehicle information 121 in chronological order to generate first related information (vehicle travel direction/speed/existence range on the road, etc.) as auxiliary information.

Next, in step S432, the response processing unit 112 uses multiple pieces of second vehicle information 122 in chronological order to generate second related information (vehicle's direction of travel/speed/existence range on the road, etc.) as auxiliary information.

Next, in step S433, the correspondence processing unit 112 uses the first vehicle information 121, the first related information, the second vehicle information 122, and the second related information to associate the first vehicle information 121 with the second vehicle information 122.

The first related information and the second related information may or may not be used. For example, when the accuracy of the second vehicle information 122 based on GPS is poor or when multiple vehicles are close to each other on a road, the accuracy of matching the first vehicle information 121 and the second vehicle information 122 can be improved by generating and using the first related information and the second related information as auxiliary information.

Returning to FIG. 8, after step S43, in step S44, the control unit 113 transmits information such as "(8) location information (latitude, longitude, altitude)" in the first vehicle information 121 to the map management server 4.

Next, in step S45, the control unit 113 transmits information such as "(8) location information (latitude, longitude, altitude)" in the first vehicle information 121 to the corresponding vehicle 5.

Next, another process performed by the location management server 1 according to the embodiment will be described with reference to FIG.
10 is a flowchart showing another process by the location management server 1 according to the embodiment. There may be a case where the number of vehicles recognized based on the first vehicle information 121 differs from the number of vehicles recognized based on the second vehicle information 122. Here, it is assumed that the number of vehicles recognized based on the second vehicle information 122 is correct.

In such cases, the accuracy of vehicle detection based on stationary camera images may decrease due to adverse conditions such as nighttime or bad weather. Specifically, the number of vehicle rectangles detected from stationary camera images may be too few or too many compared to the number of vehicles traveling on the road.

When the number of vehicle rectangles is small, the conditions for vehicle detection are too strict. When the number of vehicle rectangles is large, the conditions for vehicle detection are too lenient. Therefore, the process shown in Figure 10 is performed.

First, in step S51, the control unit 113 uses the second vehicle information 122 to calculate the second vehicle number (N2), which is the number of vehicles.

Next, in step S52, the control unit 113 uses the first vehicle information 121 to calculate the first vehicle number (N1), which is the number of vehicles.

Next, in step S53, the control unit 113 determines whether (N2-N1) is greater than a first threshold value (e.g., about 3 to 5), and if Yes, proceeds to step S54, and if No, proceeds to step S55.

In step S54, the control unit 113 switches dictionaries (switches between day and night models, switches between weather models, etc.) and adjusts parameters in the DNN used by the detection unit 212 of PC2 to detect vehicles using images from fixed cameras so that N1 increases. After step S54, the process returns to step S52.

In step S55, the control unit 113 determines whether (N1-N2) is greater than a second threshold value (e.g., approximately 3 to 5), and if Yes, proceeds to step S56, and if No, ends the process.

In step S56, the control unit 113 switches dictionaries and adjusts parameters in the DNN used by the detection unit 212 of PC2 to detect vehicles using images from fixed cameras so as to reduce N1. After step S56, the process returns to step S52. In this way, by switching dictionaries and adjusting parameters as necessary, the numbers N1 and N2 can be brought closer together, and vehicle position information can be appropriately identified.

Next, another example of calculation of the center coordinates of the vehicle in the embodiment will be described with reference to FIG.
FIG. 11 is a diagram for explaining another example of calculation of the center coordinates of the vehicle in the embodiment.
11, a line L1 is a rectangle surrounding the vehicle 5 in the image captured by the fixed camera and its diagonal line, and a point P1 is the intersection of the diagonal lines.

Line L2 is a rectangular parallelepiped that surrounds vehicle 5 when vehicle 5 is viewed as a three-dimensional object, and is a diagonal line on its top and bottom surfaces. Point P2 is the intersection of the diagonals on the top surface. Point P3 is the intersection of the diagonals on the bottom surface.

The position information of point P3 is what we want to obtain as the position information of the center of vehicle 5. Also, as can be seen by comparing points P3 and P1, the two are close to each other, so even if the position information of point P1 is used as the position information of the center of vehicle 5, the error is small. However, in order to obtain more accurate information, the position information of point P3 can be obtained, for example, using the process in FIG. 12.

FIG. 12 is a flowchart showing a process of calculating the position information of the point P3 by the PC2 according to the embodiment.
First, in step S61, the control unit 213 calculates a vehicle area (circumscribing rectangle) from an input image (image from a fixed camera). Next, in step S62, the control unit 213 determines the vehicle type (passenger car/truck/bus, etc.) based on the input image.

Next, in step S63, the control unit 213 estimates the vehicle's posture from the circumscribing rectangle and the vehicle type. Next, in step S64, the control unit 213 represents the vehicle in a rectangular wire frame based on the vehicle's posture, etc.

Next, in step S65, the control unit 213 extracts the top and bottom surfaces of the wire frame. Next, in step S66, the control unit 213 calculates the center of the vehicle area on the ground contact surface (bottom surface). More specifically, the control unit 213 refers to the position DB 221 and the conversion table 222, and identifies the vehicle's position information based on the center position of the bottom surface in the wire frame and the corresponding position in the satellite image. In this way, the position information of point P3 can be obtained.

In this way, the position management system S of this embodiment can obtain highly accurate position information of moving objects (vehicles) at low cost based on images from fixed cameras and satellite images associated with highly accurate position information.

In addition, by basing the center position of a rectangle surrounding a moving object captured in a fixed camera image on its corresponding position in a satellite image, the position information of the moving object can be identified through simple processing.

In addition, by using the wireframe representation technique described above, the accuracy of the location information can be further improved.

In addition, by transmitting the vehicle's position information to the map management server 4, the map management server 4 can use this highly accurate position information as auxiliary information to appropriately update the dynamic map 41.

In addition, by transmitting vehicle position information to the corresponding vehicle 5, the vehicle 5 can effectively utilize this highly accurate position information for autonomous driving (automatic driving, etc.), etc.

The programs executed by the location management server 1 and PC 2 in the embodiment are modular in structure including each of the functional units described above, and in actual hardware, the CPU (processor) reads the programs from the ROM and executes them, loading each of the functional units described above into the main storage device and generating them on the main storage device.

The program is provided in the form of an installable or executable file recorded on a computer-readable recording medium such as a CD-ROM, flexible disk, CD-R, or DVD.

The program may also be configured to be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. The program may also be configured to be provided or distributed via a network such as the Internet. The program may also be configured to be provided by being pre-installed in a ROM or the like.

Although the above describes an embodiment of the present invention, the above embodiment is merely an example and is not intended to limit the scope of the invention. The above embodiment can be implemented in various forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. The above embodiment is included in the scope and gist of the invention, and is included in the scope of the invention and its equivalents described in the claims.

For example, the moving object is not limited to a vehicle, but may be other moving objects such as moving bodies in a factory (transport vehicles, mobile robots, etc.).

Furthermore, the functions of the location management server 1 and the PC 2 may be realized by one computer device, or may be realized by three or more computer devices.

If there is distortion in the image from the fixed camera, the image may be corrected so that the white lines of the crosswalk in the image become straight. If there is distortion in the satellite image, the image may be corrected in the same way.

The second vehicle information may also be generated using other methods, such as GNSS, instead of GPS.

Furthermore, multiple fixed cameras 3 may be installed in one shooting area, and multiple fixed camera images may be used.

1...location management server, 2...PC, 3...fixed camera, 4...map management server, 5...vehicle, 11...processing unit, 12...storage unit, 13...input unit, 14...display unit, 15...communication unit, 21...processing unit, 22...storage unit, 23...input unit, 24...display unit, 25...communication unit, 41...dynamic map, 51...second vehicle information, 111...acquisition unit, 112...correspondence processing unit, 113...control unit, 121...first vehicle information, 122...second vehicle information, 123...correspondence information, 211...acquisition unit, 212...detection unit, 213...control unit, 221...location DB, 222...conversion table, 223...first vehicle information, S...location management system

Claims (7)

A fixed camera that captures an image of a predetermined area in which a moving object moves;
A storage unit that stores positional correspondence information between an image captured by the fixed camera and an overhead image captured from above of the predetermined area, the overhead image being associated with highly accurate positional information;
An acquisition unit that acquires a photographed image from the fixed camera;
A detection unit that detects a moving object captured in the captured image;
a control unit that refers to the correspondence information and identifies position information of the moving object;
A location management system comprising: the control unit refers to the correspondence relationship information, and identifies position information of the moving object based on a position in the overhead image that corresponds to a center position of a rectangle surrounding the moving object captured in the captured image.
The location management system of claim 1 . the moving object is a vehicle;
The control unit is
Identifying the type of the vehicle based on the captured image;
Estimating a posture of the vehicle based on a rectangle surrounding the vehicle and the vehicle type shown in the captured image;
expressing the vehicle as a rectangular parallelepiped wire frame based on the attitude;
referring to the correspondence information, and specifying position information of the moving object based on a position in the overhead image that corresponds to a center position of a bottom surface in the wire frame;
The location management system of claim 1 . The control unit transmits the position information of the moving object to a management server of a dynamic map.
The location management system of claim 1 . The control unit associates the position information of the moving object with second position information based on the GPS or GNSS acquired from the moving object, and transmits the position information of the moving object to the moving object.
The location management system of claim 1 . a storage unit that stores positional correspondence information between an image captured by a fixed camera that captures a predetermined area in which a moving object moves and an overhead image captured from above of the predetermined area, the overhead image being associated with highly accurate positional information;
An acquisition unit that acquires a photographed image from the fixed camera;
A detection unit that detects a moving object captured in the captured image;
A control unit that refers to the correspondence relationship information and identifies position information of the moving object;
A location determination device comprising: An acquisition step of acquiring a captured image from a fixed camera that captures an image of a predetermined area in which a moving object moves;
a detection step of detecting a moving object captured in the captured image;
a control step of identifying position information of the moving object by referring to position correspondence information between the image captured by the fixed camera and an overhead image of the predetermined area captured from above and associated with highly accurate position information;
A location determination method comprising:

Images