JP7610904B2 - Parking assistance method and parking assistance device - Google Patents

Description

This disclosure relates to a parking assistance method and a parking assistance device.

In recent years, there has been a demand for parking assistance functions that allow the vehicle to park automatically without the driver having to operate the steering wheel or the accelerator/brake (hereinafter referred to as automatic parking).

For example, in Patent Document 1, when a driver drives a vehicle, the vehicle generates automatic driving information for automatic driving of the vehicle based on images of the surrounding environment. Furthermore, when performing automatic driving, the vehicle calculates vehicle position information based on the images of the surrounding environment and the automatic driving information, and performs automatic parking.

JP 2017-138664 A

However, there are problems with the technology described in Patent Document 1. In the technology described in Patent Document 1, automatic parking is performed based on information from when the driver drives and parks the vehicle. In this case, the environment around the vehicle may be different when the driver drives and parks the vehicle and when automatic parking is performed thereafter. For example, it is not uncommon for an object (such as a bicycle) that was not present when the driver drove and parked the vehicle to be present when the vehicle is subsequently parked automatically.

In this case, the accuracy of the vehicle's self-location estimation during automatic parking may decrease, and as a result, the accuracy of automatic parking itself may also decrease. On the other hand, the change in the environment around the vehicle when parking may be temporary. For example, if the above-mentioned object is a bicycle, it is quite possible that it will be gone or its position will have changed the next time the vehicle is parked. Therefore, even if the surrounding environment is updated to that understood during automatic parking, the accuracy of the vehicle's self-location estimation may still decrease.

This disclosure has been made in consideration of the above problems, and its main objective is to provide a parking assistance method and a parking assistance device in a parking assistance system that can estimate the vehicle's self-position with high accuracy even when the surrounding environment changes when performing automatic parking.

The parking assistance method according to the present disclosure is a parking assistance method used in a parking assistance system that estimates the vehicle's own position from map information present around the vehicle and performs parking assistance based on the own position. The map information present around the vehicle is acquired during a first parking and a second parking performed after the first parking. When the first map information in the first parking and the second map information in the second parking are different, difference information showing the difference between the first map information and the second map information is displayed.

The parking assistance device according to the present disclosure is a parking assistance device used in a parking assistance system that estimates the vehicle's own position from map information present around the vehicle and performs parking assistance based on the own position. The parking assistance device includes an acquisition unit that acquires the map information present around the vehicle during a first parking and a second parking performed after the first parking, and a display unit that displays difference information indicating the difference between the first map information and the second map information when the first map information in the first parking and the second map information in the second parking are different.

The parking assistance method and parking assistance device disclosed herein can estimate the vehicle's self-position with high accuracy when performing automatic parking, even if changes occur in the surrounding environment.

FIG. 1 is a block diagram showing an example of the configuration of a parking assistance system according to the first embodiment. FIG. 2 is a flowchart illustrating an example of a procedure of information processing according to the first embodiment. 3(a) to (c) are diagrams showing an example of map information generated in an information processing device of embodiment 1, where (a) is a diagram showing an example of map information recorded in a memory unit, (b) is a diagram showing an example of map information generated in an automatic driving mode, and (c) is a diagram showing an example of a state in which the self-position is estimated from the map information in (a) and the map information in (b). FIG. 4 is a diagram showing an example of a display form of difference information according to the first embodiment. FIG. 5 is a diagram showing an example of a display form of difference information according to the first embodiment. FIG. 6 is a diagram showing an example of a display form of difference information according to the first embodiment. FIG. 7 is a diagram showing an example of a display form of difference information according to a modified example.

(Embodiment 1)
A parking assistance system according to a first embodiment will be described with reference to FIGS.

The parking assistance system according to the first embodiment is a system that provides parking assistance to realize automatic parking, which performs parking of a vehicle based on prerecorded route information.
Specifically, the vehicle drives in either a teacher driving mode or an automatic driving mode according to the user's instructions. The teacher driving mode is a parking mode performed as a preliminary step to automatic parking, and is a mode in which the vehicle is parked by the user's operation.
In the supervised driving mode, while the vehicle is parked in a parking space by the user's operation, the parking assistance device creates map information of the surroundings of the vehicle to be used for estimating the vehicle's self-position based on information obtained from the sensor device or the imaging device 50, and sequentially records the created map information until parking is completed. Furthermore, the parking assistance device records information on the route traveled until parking of the vehicle is completed together with creating the map information.
The automatic driving mode is a mode in which the vehicle is parked automatically without user operation. In the automatic driving mode, map information of the surroundings is generated from images captured by the imaging device 50, as in the teacher driving mode. Next, the surrounding map information recorded in the teacher driving mode is collated, and the vehicle's own position is estimated from the position information of the recorded map information, and the vehicle is parked according to the route information recorded based on the estimated position. Next, each component constituting the parking assistance system according to the first embodiment will be described.

(Configuration of parking assistance system 10)
First, the configuration of a parking assistance system 10 according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the parking assistance system 10 according to the first embodiment.

The parking assistance system 10 is for performing automatic parking of a vehicle as described above, and includes an information processing device 20, an information display device 30, an automatic driving control device 40, an imaging device 50, and a drive device 60. The parking assistance system 10 may be mounted inside the vehicle, or may be connected to the vehicle via a separate wireless network and be provided in whole or in part outside the vehicle. The configurations of the information processing device 20 and the information display device 30 are described in detail below.

(Configuration of information processing device 20)
The information processing device 20 includes a processing unit 210 and a storage unit 220, and generates map information consisting of image feature point information such as various three-dimensional structures and road surface textures around the vehicle for use in estimating the vehicle's own position, or OGM information from image features of three-dimensional structures, from captured images of the surroundings of the vehicle captured by the imaging devices 50 installed on the front, rear, left and right sides of the vehicle. Furthermore, the map information newly generated from the captured images in which the captured positions around the vehicle are updated as the vehicle travels is associated with the amount of movement (distance and direction) of the vehicle estimated from the steering angle sensor and wheel speed pulse sensor, which are on-board sensors, and stored as one piece of map information. Furthermore, the information processing device 20 generates difference information indicating the difference between the first map information generated in the first parking and recorded in the storage unit 220 of the information processing device 20 and the second map information newly generated in the second parking. Furthermore, the information processing device 20 updates the stored map information as necessary.
The first parking may be performed by driving the vehicle by the driver, and the second parking may be performed by automatic driving. Alternatively, the first parking and the second parking may be performed by automatic driving by repeating parking in the automatic driving mode.

(Processing Unit 210)
A more detailed configuration of the processing unit 210 will be described below. The processing unit 210 performs information processing for generating the above-mentioned difference information. The processing unit 210 includes an image acquisition unit 211, a map information generation unit 212, a difference information generation unit 213, an output control unit 214, and a self-position estimation unit 215. Each unit constituting the processing unit 210 will be described below.

(Image Acquisition Unit 211)
The image acquisition unit 211 acquires an image of the surroundings of the vehicle captured by the imaging device 50 mounted on the vehicle. The captured image is digital image data in which a pixel value is specified for each pixel. Usually, the imaging device 50 mounted on the vehicle is composed of four imaging devices 50 installed on the front, rear, left and right of the vehicle. Note that the image acquisition unit 211 may acquire an image of the surroundings of the vehicle captured by an imaging device 50 such as an infrastructure camera, not limited to the imaging device 50 mounted on the vehicle.

(Map information generating unit 212)
The map information generating unit 212 generates map information using the captured image acquired by the image acquiring unit 211. The map information is information used to calculate the vehicle's own position. The vehicle's own position is estimated by matching the feature amount in the map information stored in advance with the feature amount in the map information generated based on the captured image. In the first embodiment, the map information is described by taking as an example a case where the map information is information held in an OGM (Occupied Grid Map) format in which the presence or absence of a three-dimensional obstacle or the presence or absence of a three-dimensional obstacle is set to be unknown for each section (also called a grid) divided into a grid defined by a certain size (for example, 5 cm or 10 cm).

In addition, in response to information from the information display device 30, the map information generation unit 212 updates the map information recorded in the storage unit 220 by reflecting the difference information in the map information recorded in the storage unit 220.

(Difference information generation unit 213)
The difference information generating unit 213 generates difference information using the second map information generated by the map information generating unit 212 and the first map information recorded in advance. In the first embodiment, the difference information is The difference information is information for notifying the user of the difference when a difference exists. As an example, the difference information is information for notifying the user of the difference when a difference exists. This information indicates a set of sections in which the total number of sets of sections in which differences exist is equal to or greater than a predetermined value when compared with the second map information newly generated in the driving mode.

(Output control unit 214)
The output control unit 214 outputs the difference information generated by the difference information generating unit 213 to the information display device 30 .

(Self-position estimation unit 215)
The self-position estimation unit 215 collates the map information recorded in the storage unit 220 and estimates the vehicle's own position from the position information of the map information.

(Memory unit 220)
The storage unit 220 is a recording medium for storing map information. The map information recorded in the storage unit 220 is initially generated in the teacher driving mode. Thereafter, when an instruction is received from the user, the map information stored in the storage unit 220 is updated based on the map information generated in the automatic driving mode.

Examples of the recording medium include tapes such as magnetic tapes and cassette tapes, magnetic disks such as floppy disks (registered trademark) and hard disks, disks including optical disks such as CD-ROMs (Compact Disc Read-Only Memory), MO disks (Magneto-Optical discs), MDs (Mini Discs), DVDs (Digital Versatile Discs (registered trademark)), CD-Rs (CD Recordable), and Blu-ray Discs (registered trademark), cards such as IC cards (including memory cards) and optical cards, semiconductor memories such as mask ROMs, EPROMs (Erasable Programmable Read-Only Memory), EEPROMs (Electrically Erasable and Programmable Read-Only Memory (registered trademark)), and flash ROMs, and logic circuits such as PLDs (Programmable logic devices) and FPGAs (Field Programmable Gate Arrays).

Note that the memory unit 220 does not necessarily need to be provided inside the information processing device 20, and may be provided outside the information processing device 20 as necessary.

(Configuration of information display device 30)
Next, the information display device 30 constituting the parking assistance system 10 will be described. The information display device 30 displays the difference information generated by the information processing device 20 in a form that can be viewed by the user. The information display device 30 may be directly connected to the information processing device 20 or may be connected via a network. For example, an in-vehicle display device that functions as an HMI for a car navigation system or a parking assistance system can be cited as an example of a device that is directly connected to the information processing device 20. For example, a portable information terminal such as a smartphone or a tablet can be cited as an example of a device that is connected to the information processing device 20 via a network.

The information display device 30 includes a difference information acquisition unit 310, a difference information processing unit 320, a display unit 330, an input unit 340, and an instruction unit 350.

(Difference information acquisition unit 310)
The difference information acquisition unit 310 acquires difference information generated and output by the information processing device 20 .

(Difference information processing unit 320)
The difference information processing unit 320 performs processing to display the difference information acquired by the difference information acquisition unit 310 on the display unit 330 in a form visible to the user. In this embodiment, display in a form visible to the user specifically means display using an image. Furthermore, when the difference information processing unit 320 receives an instruction from the input unit 340 to update the map information recorded in the memory unit 220, the difference information processing unit 320 instructs the instruction unit 350 to reflect, in the map information recorded in the memory unit 220, the section sets in which the total number of sets of sections in which differences exist is equal to or greater than a predetermined value.

(Display unit 330)
The display unit 330 is a display device that notifies the user of information by displaying an image. The notified information is information for notifying the user of the difference information acquired by the difference information acquisition unit 310 and an inquiry as to whether or not to update the map information recorded in the storage unit 220. Specific examples of the display unit 330 include display devices such as a liquid crystal display or an organic electroluminescence display.

(Input unit 340)
The input unit 340 is an input device for receiving instructions from a user. Examples of the input unit 340 include a touch panel that is integrated with the display unit 330, or a microphone that receives voice input.

(Instruction Unit 350)
The instruction unit 350 instructs the information processing device 20 to update the map information recorded in the memory unit 220 by reflecting the information of the section set that has been instructed to be updated by the user, which is among the difference information of the map information, in the map information recorded in the memory unit 220.

(Configuration of the automatic driving control device 40)
The automatic driving control device 40 includes an automatic driving information registration unit 410 and an automatic driving control unit 420 .

(Autonomous driving information registration unit 410)
The automatic driving information registration unit 410 operates in the teacher driving mode. In the teacher driving mode, the automatic driving information registration unit 410 records information about the route traveled until the vehicle is parked.

(Automatic driving control unit 420)
The automatic driving control unit 420 operates in the automatic driving mode. The automatic driving control unit 420 controls the drive unit 60 to automatically drive the vehicle based on the image of the surroundings of the vehicle captured by the imaging device 50 in the automatic driving mode and the self-position estimated by the self-position estimation unit 215, and in accordance with the route information recorded in the automatic driving information registration unit 410.

(Configuration of imaging device 50)
The installation position and the angle of view of the imaging device 50 are adjusted in advance so that the imaging device 50 can capture images of the surroundings of the vehicle. In the first embodiment, the vehicle is provided with a plurality of imaging devices 50 each having a different imaging direction.

(Configuration of driving device 60)
The driving device 60 controls the driving section to automatically drive the vehicle. More specifically, the driving device 60 controls the acceleration amount, braking amount, steering angle, etc. of the vehicle so that the vehicle automatically travels on the route at a specified speed based on the vehicle's own position and the route information recorded in the teacher driving mode.

(Procedure of Information Processing in Parking Assistance System 10)
Next, the procedure of information processing executed in the parking assistance system 10 will be described below with reference to Fig. 2. Fig. 2 is a flowchart showing the procedure of information processing executed in the parking assistance system 10.

First, the image acquisition unit 211 acquires an image of the surroundings of the vehicle captured by the imaging device 50 while the vehicle is traveling (step S101).

Next, the map information generating unit 212 generates map information using the captured image acquired in step S101 and estimates the vehicle's own position (step S102). A specific example of the generated map information is described below.

(a) to (c) of FIG. 3 are diagrams showing an example of map information generated by the map information generating unit 212. As shown in FIG. 3(a) to (c), in the map information in the OGM format, the space around the vehicle where parking is performed is divided into sections of a certain size, and the presence or absence of a three-dimensional obstacle or the presence or absence of a three-dimensional obstacle is set for each section. The sections with diagonal lines represent sections where a three-dimensional obstacle exists, and the white sections represent sections where a three-dimensional obstacle does not exist. FIG. 3(a) is a diagram showing an example as a partial area of the map information recorded in the memory unit 220, FIG. 3(b) is a diagram showing an example of map information generated from an image captured by the vehicle at a certain point during driving in the automatic driving mode, and FIG. 3(c) is a diagram showing an example of a state in which the self-position 110 is estimated with respect to the map information in FIG. 3(a) from the map information in FIG. 3(a) and the map information in FIG. 3(b). In addition, the section surrounded by a thick frame in Figure 3 (a) and Figure 3 (c) is the parking position 100. In addition, the size of the section in Figure 3 is shown as a schematic large size, but in reality, the size of the section is further subdivided and the map information is represented by finer sections.

In the automatic driving mode, the map information generating unit 212 generates map information based on the captured image acquired by the image acquiring unit 211, and calculates the position of the section in which the vehicle exists, which is the self-position in the generated map information. The map information generating unit 212 reads out the map information (FIG. 3(b)) generated in the automatic driving mode and the map information (FIG. 3(a)) recorded in the storage unit 220, performs matching, searches for the position where the total value of the matching error is the smallest, and determines the self-position of the vehicle. The result of the self-position calculated by the matching process is shown in FIG. 3(c). The vehicle is driven to follow the route based on the self-position 110 estimated by the self-position estimating unit 215 and the route information stored in the automatic driving information registering unit 410, and automatic parking is performed.
In addition, the matching process may involve calculating the amount of movement of the vehicle from its previously calculated self-position to the position where the current captured image is to be acquired using on-board sensors (e.g., a steering angle sensor and a wheel speed pulse sensor, etc.), and searching the surrounding area based on the position obtained by adding the amount of movement to the previous self-position, thereby reducing the amount of processing required for the search and reducing errors in calculating the self-position.
The number of captured images used for matching is not limited to one, and map information obtained from four captured images taken of the front, rear, left, and right sides of the vehicle may be matched. This makes it possible to obtain the vehicle's own position robustly against differences in position accuracy, differences in the illuminance environment around the vehicle, and changes in the placement of three-dimensional obstacles.
In the matching process, the estimation result of the self-location is calculated not necessarily as an integer multiple of the section location, but as a decimal point location, thereby making it possible to increase the location resolution.

The difference information generating unit 213 compares the second map information generated by the map information generating unit 212 with the first map information previously recorded in the storage unit 220 to generate difference information (step S103). Specifically, the first map information previously recorded and used in step S102 is compared with the second map information generated in the automatic driving mode, and a set of sections in which the total number of sets of sections in which a difference exists is equal to or greater than a predetermined value is detected as an area in which the difference is considered to be a difference in map information rather than within the range of error, and difference information is generated. In other words, information is generated that indicates the change in the environment around the vehicle between when the map information recorded in the storage unit 220 was generated during parking and when the vehicle is traveling.
Furthermore, in order to display the difference information on the information display device, the difference information generating unit 213 transmits to the output control unit 214 the difference information in association with the captured image from which the difference information was detected.

The output control unit 214 outputs the difference information generated by the difference information generation unit 213 and the captured image associated with the difference information to the information display device 30 (step S104). The processing up to this step is executed in the information processing device 20. The processing in the information display device 30 will be described below with continued reference to FIG. 2. The processing in the information processing device 20 and the processing in the information display device 30 may be processed continuously as one unit, or may be processed with a time interval therebetween.

The difference information acquisition unit 310 of the information display device 30 acquires the difference information output in step S104 (step S105).

The difference information processing unit 320 generates an image for displaying the difference information acquired by the difference information acquisition unit 310 in a form that is visible to the user (step S106).

The difference information processing unit 320 displays the image generated by the difference information processing unit 320 on the display unit 330, and notifies the user whether or not to update the map information recorded in the storage unit 220 (step S107). For example, the difference information processing unit 320 controls the display unit 330 to output an image indicating whether or not to update the map information recorded in the storage unit 220 by reflecting, in the map information recorded in the storage unit 220, information indicating a set of sections in which the total number of sets of sections in which differences exist is equal to or greater than a predetermined value, together with the image generated by the difference information processing unit 320.

The input unit 340 receives an instruction from the user as to whether or not to update the map information recorded in the memory unit 220 (step S109). This makes it possible to determine whether or not to reflect changes in the environment around the vehicle in the map information based on the user's judgment. If an instruction to update the map information recorded in the memory unit 220 is received from the user (step S109: Yes), the process proceeds to step S110. If an instruction to update the map information recorded in the memory unit 220 is not received from the user (step S109: No), the operation of the parking assistance system 10 is terminated.

The instruction unit 350 instructs the information processing device 20 to update the map information recorded in the memory unit 220, and ends the operation of the parking assistance system 10 (step S110).

In this way, parking assistance system 10 allows the user to determine whether the change in the captured image around the vehicle is temporary, and determines whether to reflect the change in the environment around the vehicle in the map information based on the user's determination. Therefore, parking assistance system 10 can maintain high accuracy in estimating the vehicle's self-position even if the surrounding environment changes when performing automatic parking, by creating map information that accurately reflects the change in the environment around the vehicle each time parking is performed.

Note that it is preferable to notify the user whether or not to update the map information stored in the memory unit 220 immediately after parking ends and the vehicle comes to a stop.

In addition, although an example has been described in which only one piece of map information is stored in the memory unit 220, multiple pieces of map information may be stored in the memory unit 220.

Furthermore, in the third parking performed after the second parking, map information present around the vehicle may be acquired, and if the second map information generated in the second parking and recorded in the memory unit 220 of the information processing device 20 differs from the third map information newly generated in the third parking, difference information showing the difference between the second map information and the third map information may be displayed.

(Display form 1)
Next, the display form when the difference information is displayed to the user on the display unit 330 will be specifically described below with reference to Figures 4 to 6. As described above, the difference information in the first embodiment is information that indicates a set of sections where the total number of sets of sections where there is a difference is equal to or greater than a predetermined value when comparing the first map information recorded in advance in the storage unit 220 with the second map information newly generated in the autonomous driving mode.

FIG. 4 is a diagram showing where a set of sections in which the total number of sets of sections in which differences exist is equal to or greater than a predetermined value (hereinafter, referred to as a set of sections that are the difference) is located with respect to the vehicle's driving route. In this display form, the position of the set of sections that are the difference (also referred to as a difference position) is specified by a relative position with respect to the vehicle's driving route or the parking position 100, and the difference position is displayed surrounded by a thick frame. Furthermore, the position of the set of sections that are the difference is specified by a relative position with respect to the vehicle's driving route or the parking position 100, and the difference position is displayed as a newly added object using text information. As for the vehicle's driving route, the route traveled from the parking start point to the parking end point is displayed using an arrow line. By displaying the difference information to the user in this form, the user can specifically know the position where the environment around the vehicle has changed.
In Fig. 4(a), the travel route, a solid line surrounded by a thick frame representing the difference information position, and dotted lines representing the parking start position and the parking end position are shown in an aerial photograph of the three-dimensional obstacle taken from above. However, since it is usually difficult to create an aerial photograph from an image captured by the imaging device 50 mounted on the vehicle, the image displayed on the information display device may be composed of only the travel route, the solid line surrounded by a thick frame, and the dotted line, as shown in Fig. 4(b).

In response to the above-mentioned notification, the user may select whether or not to update using, for example, the input unit 340, which is an input function of a touch panel that is integral with the display unit 330.

(Display form 2)
Fig. 5 is a diagram showing where a set of sections in which the total number of sets of sections in which differences exist is equal to or greater than a predetermined value is located relative to the vehicle's driving route. In addition to the display format of Fig. 4, the sections in which the total number of sets of sections in which differences exist is equal to or greater than a predetermined value are displayed in a color different from other sections. By displaying difference information to the user in this format, the user can specifically know the location where the environment around the vehicle has changed.
In Fig. 5(a), in an aerial photograph showing a three-dimensional obstacle viewed from above, the travel route, the filled-in areas using a different color from other sections to indicate the difference information positions, the solid lines surrounded by thick frames to indicate the difference information positions, and the dotted lines to indicate the parking start position and the parking end position are shown. However, since it is usually difficult to create an aerial photograph from an image captured by the imaging device 50 mounted on a vehicle, the image displayed on the information display device may be composed only of the travel route, the filled-in areas using a different color from other sections to indicate the difference information positions, the solid lines surrounded by thick frames, and the dotted lines, as shown in Fig. 5(b).

(Display form 3)
FIG. 6 is a diagram showing a section set in which the total number of sections having differences is equal to or greater than a predetermined value, superimposed on a captured image of the periphery of a vehicle. As an example, a range including a predetermined area in which the section set is located is cut out from the captured image and displayed on the captured image. The section set that is the difference is displayed surrounded by a thick frame. In addition, the section set that is the difference is displayed by text information as a newly added object. By displaying the difference information to the user in this form, the position where the environment around the vehicle has changed can be specifically known. The captured image may be a composite image that is composited based on captured images captured by a plurality of imaging devices 50 mounted on the vehicle. In order to make it easier for the user to recognize the difference information, the display image of FIG. 4 or FIG. 5 may be displayed side by side in addition to the display image of FIG. 6.
Although the display forms 1 to 3 have been described taking the case where a new object is added as an example, the difference information can be displayed using a similar display form even if an object has disappeared.

(Processing when multiple pieces of map information are represented as difference information)
4 to 6, the case where there is only one piece of map information represented as difference information has been described as an example, but there may be multiple pieces of map information represented as difference information. That is, there may be multiple added objects, or multiple lost objects. Also, an added object and a lost object may exist simultaneously. This allows changes in the environment around the vehicle to be reflected in the map information in more detail. When there are multiple pieces of difference information, after repeating steps S109 and S110 in the flowchart of FIG. 2, the user is asked whether or not there has been an update for all of the difference information, and the process ends.

When there are multiple pieces of map information represented as difference information, they may be notified to the user individually or collectively. When notifying the user individually, each piece of difference information is processed by notifying one of the added or disappeared objects and repeating the operation of accepting instructions from the user in sequence. In this case, the user can examine each piece of map information represented as difference information one by one and decide whether or not to reflect it. When notifying the user collectively, instructions from the user for the added or disappeared object are accepted at once and each piece of difference information is processed.

(Modification)
In the above-mentioned parking assistance system 10, the map information is stored in the OGM format, in which the area around the parking route is divided into a grid of a certain size, and the presence or absence of a three-dimensional obstacle is set for each section, or the presence or absence of a three-dimensional obstacle is unknown. However, a modified example will be described below in which the map information is stored in another format.

As a modified example, the map information is information indicating the coordinate positions of the feature points extracted from the captured image. In this case, the difference information is information for emphasizing and displaying coordinate positions where there are no corresponding feature points in a comparison between the first map information generated in advance and recorded in the storage unit 220 of the information processing device 20 and the second map information newly generated in the automatic driving mode. Each feature point of the feature points has a coordinate position expressed as (Xi, Yi) or (Xi, Yi, Zi) in a coordinate space in which the origin is set at an arbitrary point with respect to the environment around the parking and driving and the coordinate axes of a two-dimensional plane or three-dimensional space are set. Furthermore, in addition to the coordinate information of the feature points, it is also possible to associate the coordinates of the vehicle position at the time of extracting the feature points to form map information.

More specifically, the difference information is information for emphasizing and displaying the coordinate positions of a group of feature points that are present in the first map information generated in advance and recorded in the information processing device 20 but are not present in the newly generated second map information. Alternatively, the difference information is information for emphasizing and displaying the coordinate positions of a group of feature points that are not present in the first map information generated in advance and recorded in the information processing device 20 but are present only in the newly generated second map information.
The former is a case where an object that was not present when the map information recorded in the information processing device 20 was generated is added, and the latter is a case where an object that was present when the map information recorded in the information processing device 20 was generated is disappeared. In the former case, in response to an instruction from the user, the map information recorded in the storage unit 220 is updated by adding the coordinate positions of the feature points that are present only in the newly generated map information to the map information recorded in the storage unit 220. Also, in the latter case, in response to an instruction from the user, the map information recorded in the storage unit 220 is updated by deleting the coordinate positions of the feature points that are not present in the newly generated map information.

(Display form 4)
Next, a display form when displaying difference information to a user in the modified example will be specifically described below with reference to FIG. 7. FIG. 7 is a diagram showing coordinate positions where no corresponding feature points exist in a captured image of the periphery of a vehicle. The coordinate positions where no corresponding feature points exist are displayed surrounded by a thick frame. Furthermore, the user is notified of whether or not to update the map information recorded in the storage unit 220. In response to the above-mentioned notification, the user may select whether or not to update using, for example, the input unit 340, which is an input function in a touch panel provided integrally with the display unit 330. By displaying difference information to the user in this form, the user can specifically know the position where the environment around the vehicle has changed.

In the above-mentioned embodiment, the program for executing the information processing has a modular configuration including each of the above-mentioned multiple functional units, and the actual hardware is, for example, a CPU (processor circuit) that reads the information processing program from a ROM or HDD and executes it, whereby each of the above-mentioned multiple functional units is loaded onto a RAM (main memory) and each of the above-mentioned multiple functional units is generated on the RAM (main memory). It is also possible to realize some or all of the above-mentioned multiple functional units using dedicated hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array).

Although the above describes an embodiment, the above embodiment is presented as an example and is not intended to limit the scope of the present disclosure. The above embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. The above embodiment is included in the scope or gist of the present disclosure, and is included in the scope of the invention and its equivalents described in the claims.

The parking assistance system disclosed herein can be used, for example, for automatic parking assistance.

REFERENCE SIGNS LIST 10 Parking assistance system 20 Information processing device 210 Processing section 211 Image acquisition section 212 Map information generation section 213 Difference information generation section 214 Output control section 215 Self-position estimation section 220 Storage section 30 Information display device 310 Difference information acquisition section 320 Difference information processing section 330 Display section 340 Input section 350 Instruction section 40 Automatic driving control device 410 Automatic driving information registration section 420 Automatic driving control section 50 Imaging device 60 Drive device

Claims (13)

A parking assistance method for use in a parking assistance system that estimates a vehicle's own position from map information present around the vehicle and performs parking assistance based on the vehicle's own position, comprising:
In a first parking and a second parking performed after the first parking, the map information existing around the vehicle is acquired;
When the first map information in the first parking area and the second map information in the second parking area are different, difference information indicating a difference between the first map information and the second map information is displayed;
the map information is information stored in a format in which the presence or absence of a three-dimensional obstacle is set for each section divided into a grid in a captured image of the periphery of the vehicle,
The difference information is information indicating a set of sections in which a total number of sets of sections in which differences exist is equal to or greater than a predetermined value in a comparison between the first map information and the second map information.
Parking assistance methods. the difference information is information displayed by a frame line,
A difference position indicating a difference position between the first map information and the second map information based on a relative position with respect to a driving route or a parking position of the vehicle is indicated by the frame line.
The parking assistance method according to claim 1 . the difference information is information displayed by character information,
A difference position indicating a difference position between the first map information and the second map information based on a relative position with respect to a driving route or a parking position of the vehicle is indicated by the character information .
3. A parking assistance method according to claim 1 or 2 . an image of a range including the difference position is cut out from a captured image of the periphery of the vehicle and displayed;
4. The parking assistance method according to claim 2 or 3 . The timing for displaying the difference information is after the second parking is completed.
A parking assistance method according to any one of claims 1 to 4 . The timing for displaying the difference information is during driving in the second parking position.
A parking assistance method according to any one of claims 1 to 5 . The first parking is performed by a driver driving the vehicle,
The second parking is performed by automatic driving.
A parking assistance method according to any one of claims 1 to 6 . The first parking and the second parking are performed by automatic driving.
A parking assistance method according to any one of claims 1 to 6 . In a third parking performed after the second parking, the map information existing around the vehicle is acquired;
When the second map information in the second parking area and the third map information in the third parking area are different, difference information indicating a difference between the second map information and the third map information is displayed.
A parking assistance method according to any one of claims 1 to 8 . notifying a user of the vehicle of whether or not a difference between the first map information and the second map information should be reflected in the map information;
reflecting a difference between the first map information and the second map information in the map information when a user of the vehicle selects to reflect the difference;
A parking assistance method according to any one of claims 1 to 9 . notifying a user of the vehicle of whether or not a difference between the first map information and the second map information is to be reflected in the map information;
When a user of the vehicle has not selected to reflect the difference between the first map information and the second map information, the difference is not reflected in the map information.
A parking assistance method according to any one of claims 1 to 10 . the display device for displaying the difference information is a portable information terminal and is connected to the information processing device via a network;
12. A parking assistance method according to claim 10 or 11 . A parking assistance device used in a parking assistance system that estimates a vehicle's own position from map information present around the vehicle and performs parking assistance based on the vehicle's own position,
an acquisition unit that acquires the map information present around the vehicle during a first parking and a second parking performed after the first parking;
a display unit that displays difference information indicating a difference between the first map information and the second map information when the first map information in the first parking area and the second map information in the second parking area are different;
Equipped with
the map information is information stored in a format in which the presence or absence of a three-dimensional obstacle is set for each section divided into a grid in a captured image of the surroundings of the vehicle,
The difference information is information indicating a set of sections in which a total number of sets of sections in which differences exist is equal to or greater than a predetermined value in a comparison between the first map information and the second map information.
Parking assistance device.

Images