JP7578084B2 - Parking Assistance System - Google Patents

Description

The present invention relates to a vehicle parking assistance system.

Conventionally, a parking assistance system is known in which an image of a parking lot taken by a camera mounted on the vehicle is registered in advance, and the next time the vehicle is parked in that parking lot, the system automatically parks the vehicle in the registered parking lot by determining the relative position of the vehicle and the parking lot based on the feature points of the image of the surroundings of the vehicle taken by the camera at that time and the feature points of the registered parking lot image (see, for example, Patent Document 1).

JP 2020-117128 A

When the above system receives a parking assistance request from the driver, it performs a matching process between the image of the vehicle's surroundings captured by the camera at that time and a registered parking lot image, and starts parking assistance control if it can detect a parking space in the registered parking lot. Therefore, if the current position of the vehicle when the parking assistance request is received is significantly away from the position where the vehicle was parked when the parking lot was registered in the system, specifically, the position where the vehicle was parked near the entrance to the parking lot and the registration operation was performed (hereinafter referred to as the vehicle position at the time of registration), it is not possible to detect the parking space in the parking lot and it is not possible to start parking assistance control.

In such a situation, in order for the parking assistance system to start parking assistance control, the driver needs to align the vehicle's current position with the vehicle's position at the time of registration. However, the system is not able to detect parking spaces in registered parking lots, and is unable to properly inform the driver of where to move the vehicle to align it with the vehicle's position at the time of registration, which means there is room for improvement in convenience.

The present invention has been made to solve the above problems. That is, one of the objects of the present invention is to provide a parking assistance system that can improve convenience when a registered parking lot cannot be detected.

The parking assistance system (10) of the present invention comprises:
an imaging device (45) mounted on the vehicle (100) for capturing an image (51P) of the surroundings of the vehicle (100) and a parking lot image (60) of the parking lot (62) for automatically parking the vehicle (100) in the parking lot;
a control device (90) that registers the parking lot image (62) captured by the imaging device (45) as parking lot information for the parking lot, and performs parking assistance control to automatically park the vehicle (100) in the parking lot (62) based on the parking lot information when a parking assistance request is received from a driver; and
and a display device (50) capable of displaying an image captured by the imaging device (45).

The control device (90)
The surrounding image (51P) captured by the imaging device (45) is compared with the parking lot image (60) registered as the parking lot information, and the parking assistance control is started when the parking lot (62) is detected by the comparison process.
If the parking lot (62) cannot be detected by the matching process, the parking lot image (60) registered as the parking lot information and the current surrounding image (51P) captured by the imaging device (45) are displayed on the display device (50).

According to the above configuration, when a registered parking lot (62) cannot be detected, a registered parking lot image (60) and an image (51P) of the surroundings of the vehicle (100) are displayed on the display device (50). This makes it possible to inform the driver that the current position of the vehicle (100) is far from the vehicle position (P) at the time of registration in a situation where parking assistance control cannot be started, thereby effectively improving convenience.

In another embodiment of the present invention,
The vehicle further includes a position information acquisition device (30) for acquiring position information of the vehicle (100),
The control device (90)
The parking lot image (60) captured by the imaging device (45) is associated with location information acquired by the location information acquisition device (30) when the image is captured, and registered as the parking lot information of the parking lot (62);
If the parking lot (62) cannot be detected by the matching process, a determination is made as to whether the current position of the vehicle (100) is within a predetermined distance range from the parking lot (62) based on the current position information of the vehicle (100) acquired by the position information acquisition device (30) and the position information of the parking lot (62) registered as the parking lot information;
When it is determined that the current position of the vehicle (100) is within a predetermined distance range from the parking lot (62), the parking lot image (60) registered as the parking lot information and the current surrounding image (51P) captured by the imaging device (45) are displayed on the display device (50), whereas when it is determined that the current position of the vehicle (100) is not within a predetermined distance range from the parking lot (62), a message image indicating that the parking assistance control cannot be executed is displayed on the display device (50).

According to this aspect, different images are displayed on the display device (50) when the vehicle (100) is within a predetermined distance range from the registered parking lot (62) and when the vehicle (100) is farther away than the predetermined distance range from the registered parking lot (62). This makes it possible to present the driver with appropriate information according to the distance between the current position of the vehicle (100) and the registered parking lot (62), making it possible to more reliably improve convenience.

In another embodiment of the present invention,
The control device (90)
The parking lot images (60) of a plurality of parking lots (62) captured by the imaging device (45) are configured to be associated with location information acquired by the location information acquisition device (30) when the images are captured, and can be registered as parking lot information of each parking lot,
If none of the multiple parking lots (62) can be detected by the matching process, the parking lot (62) closest to the current position of the vehicle (100) is identified among the multiple parking lots (62) based on the current position information of the vehicle (100) acquired by the position information acquisition device (30) and the position information of the multiple parking lots (62) registered as the parking lot information, and if it is determined that the current position of the vehicle (100) is within a predetermined distance range from the identified parking lot (62), a parking lot image (60) of the identified parking lot (62) and the current surrounding image (51P) captured by the imaging device (45) are displayed on the display device (50).

According to this embodiment, the parking lot image (60) of the parking lot (62) that is located closest to the current position of the vehicle (100) among the multiple parking lots (62) is displayed on the display device (50). This makes it possible to effectively present to the driver the parking lot (62) that is most suitable for the current position of the vehicle (100) even when multiple parking lots (62) are registered.

In the above explanation, to aid in understanding the invention, the symbols used in the embodiments are enclosed in parentheses with respect to the constituent elements of the invention corresponding to the embodiments, but each constituent element of the invention is not limited to the embodiment defined by the symbols.

1 is a diagram showing a parking assistance system according to an embodiment of the present invention and a vehicle to which the parking assistance system is applied; FIG. 2 is a diagram showing the arrangement and detection range of a sonar sensor. FIG. 2 is a diagram showing the arrangement and imaging range of a camera sensor. FIG. 1 is a diagram showing an example of a parking lot. FIG. 2 shows the front and rear ranges. FIG. 2 is a diagram showing a left range and a right range. FIG. 13 is a diagram showing feature points. FIG. FIG. 2 is a diagram for explaining the operation of the parking assistance system shown in FIG. 1; 2 is a diagram for explaining the operation of the parking assistance system shown in FIG. 1; 2 is a diagram for explaining the operation of the parking assistance system shown in FIG. 1; FIG. 13 is a diagram showing entry feature points. 2 is a diagram for explaining the operation of the parking assistance system shown in FIG. 1; 2 is a diagram for explaining the operation of the parking assistance system shown in FIG. 1; 2 is a diagram for explaining the operation of the parking assistance system shown in FIG. 1; 2 is a diagram for explaining the operation of the parking assistance system shown in FIG. 1; FIG. 2 is a flowchart showing a display control routine executed by the ECU shown in FIG. 1 . FIG.

The parking assistance system according to this embodiment will be described below with reference to the drawings. Identical components are given the same reference numerals, and their names and functions are also the same. Therefore, detailed descriptions thereof will not be repeated. Note that hereinafter, the parking assistance system may also be referred to as the "embodiment device."

[Overall configuration]
FIG. 1 shows a parking assistance system 10 according to this embodiment and a vehicle 100 to which the parking assistance system 10 is applied.

As shown in FIG. 1, the parking assistance system 10 includes an ECU 90 (control device). ECU is an abbreviation for Electronic Control Unit. The ECU 90 includes a microcomputer as its main component. The microcomputer includes a CPU, ROM, RAM, non-volatile memory, and an interface. The CPU is configured to realize various functions by executing instructions, programs, or routines stored in the ROM.

The vehicle 100 is equipped with a drive device 11, a steering device 12, and a braking device 13. The drive device 11 generates a driving force that is transmitted to the drive wheels of the vehicle 100. Examples of the drive device 11 include an electric motor and an engine. The vehicle 100 may be any of an engine vehicle, a hybrid vehicle (HEV), a plug-in hybrid vehicle (PHEV), a fuel cell vehicle (FCEV), and an electric vehicle (BEV). The steering device 12 is, for example, an electric power steering device, and applies a steering force to the wheels of the vehicle 100. The braking device 13 is, for example, a disc brake device, and applies a braking force to the wheels of the vehicle 100.

The drive device 11, steering device 12, and braking device 13 are electrically connected to the ECU 90. The ECU 90 controls the operation of the drive device 11 to control the drive force applied to the drive wheels of the vehicle 100. The ECU 90 also controls the operation of the steering device 12 to control the steering torque applied to the steered wheels of the vehicle 100. The ECU 90 also controls the operation of the braking device 13 to control the braking force applied to the wheels of the vehicle 100.

The ECU 90 is connected to a parking assistance switch 15, a vehicle state acquisition device 20, a position information acquisition device 30, a surrounding recognition device 40, and a display 50 (display device), etc.

The parking assist switch 15 is a switch that is turned ON/OFF by the driver. When the driver turns the parking assist switch 15 ON, the ECU 90 accepts a parking assist request.

The vehicle state acquisition device 20 is a set of sensors that acquire the state of the vehicle 100. Specifically, the vehicle state acquisition device 20 includes an accelerator sensor 21, a brake sensor 22, a steering angle sensor 23, a steering torque sensor 24, a vehicle speed sensor 25, etc.

The accelerator sensor 21 detects the amount of operation of an accelerator pedal (not shown) of the vehicle 100. The brake sensor 22 detects the amount of operation of a brake pedal (not shown) of the vehicle 100. The steering angle sensor 23 detects the steering angle θst of a steering wheel (or steering shaft) (not shown) of the vehicle 100. The steering torque sensor 24 detects the steering torque TQst acting on the steering shaft (not shown) of the vehicle 100 due to operation of the steering wheel. The vehicle speed sensor 25 detects the traveling speed (vehicle speed) SPD of the vehicle 100. Various state information of the vehicle 100 acquired by the vehicle state acquisition device 20 is transmitted to the ECU 90.

The position information acquisition device 30 acquires current position information of the vehicle 100. For example, a GPS (Global Positioning System) or a GNSS (Global Navigation Satellite System) can be used as the position information acquisition device 30. The current position information of the vehicle 100 acquired by the position information acquisition device 30 is transmitted to the ECU 90.

The surrounding recognition device 40 is a set of sensors that recognizes targets around the vehicle 100. Specifically, the surrounding recognition device 40 includes a sonar sensor 41 and a camera sensor 44 (image capture device). Target information about the targets around the vehicle 100 acquired by the surrounding recognition device 40 is transmitted to the ECU 90.

Figure 2 shows the arrangement and detection range of the sonar sensor 41. The sonar sensor 41 is equipped with a first clearance sonar 411 to a twelfth clearance sonar 422. In Figure 2, the direction indicated by the symbol Dx is the fore-and-aft direction of the vehicle 100, and the direction indicated by the symbol Dw is the width direction of the vehicle 100.

As shown in FIG. 2, the first clearance sonar 411 is attached to the vehicle 100 so as to radiate sound waves from the front left end of the vehicle 100 to the left front. The second clearance sonar 412 is attached to the vehicle 100 so as to radiate sound waves forward from the left front end of the vehicle 100. The third clearance sonar 413 is attached to the vehicle 100 so as to radiate sound waves from the front right end of the vehicle 100 to the right front. The fourth clearance sonar 414 is attached to the vehicle 100 so as to radiate sound waves forward from the right front end of the vehicle 100.

The fifth clearance sonar 415 is attached to the vehicle 100 so as to radiate sound waves from the rear left end of the vehicle 100 to the left rear. The sixth clearance sonar 416 is attached to the vehicle 100 so as to radiate sound waves from the rear left end of the vehicle 100 to the rear. The seventh clearance sonar 417 is attached to the vehicle 100 so as to radiate sound waves from the rear right end of the vehicle 100 to the right rear. The eighth clearance sonar 418 is attached to the vehicle 100 so as to radiate sound waves from the rear right end of the vehicle 100 to the rear.

The ninth clearance sonar 419 is attached to the vehicle 100 so as to radiate sound waves to the left from the front left side of the vehicle 100. The tenth clearance sonar 420 is attached to the vehicle 100 so as to radiate sound waves to the left from the rear left side of the vehicle 100. The eleventh clearance sonar 421 is attached to the vehicle 100 so as to radiate sound waves to the right from the front right side of the vehicle 100. The twelfth clearance sonar 422 is attached to the vehicle 100 so as to radiate sound waves to the right from the rear right side of the vehicle 100.

The first clearance sonar 411 to the twelfth clearance sonar 422 receive sound waves reflected by objects. The sonar sensor 41 is electrically connected to the ECU 90. The sonar sensor 41 transmits information to the ECU 90 regarding the sound waves emitted by the first clearance sonar 411 to the twelfth clearance sonar 422 and the sound waves received by the first clearance sonar 411 to the twelfth clearance sonar 422. The ECU 90 acquires information regarding objects present around the vehicle 100 as object information OBJ based on the information received from the sonar sensor 41 (hereinafter referred to as "sonar information SON").

Figure 3 shows the arrangement and imaging range of the camera sensor 44. The camera sensor 44 captures images of the surroundings of the vehicle 100 and obtains an image of the surroundings of the vehicle 100 by processing the captured image data. The camera sensor 44 is, for example, a stereo camera or a monocular camera, and a digital camera having an imaging element such as a CMOS or CCD can be used.

The camera sensor 44 includes a front camera 44a, a rear camera 44b, a left side camera 44c, and a right side camera 44d. When there is no need to distinguish between the multiple cameras 44a to 44d, they are simply referred to as "camera 45."

The front camera 44a is provided at approximately the center of the front of the vehicle 100 in the vehicle width direction. The optical axis of the front camera 44a faces forward of the vehicle 100. The rear camera 44b is provided at the rear of the vehicle 100. The optical axis of the rear camera 44b faces rearward of the vehicle 100. The left side camera 44c is provided on the left side of the vehicle 100. The optical axis of the left side camera 44c faces leftward of the vehicle 100. The right side camera 44d is provided on the right side of the vehicle 100. The optical axis of the right side camera 44d faces rightward of the vehicle 100.

The angle of view of the camera 45 is wide. Therefore, the imaging range of the camera 45 includes the "ranges to the right, left, below, and above" based on the optical axis. The entire periphery of the vehicle 100 is included in the imaging range of the four cameras 44a to 44d. The camera 45 acquires image information (image data) by imaging the area around the vehicle 100 that corresponds to the imaging range every time a predetermined time has elapsed. The camera 45 transmits the acquired image data to the ECU 90.

Specifically, the front camera 44a captures the surrounding area in front of the vehicle 100 and transmits the generated image data (hereinafter referred to as "front image data IMG1") to the ECU 90. The rear camera 44b captures the surrounding area behind the vehicle 100 and transmits the generated image data (hereinafter referred to as "rear image data IMG2") to the ECU 90. The left side camera 44c captures the surrounding area on the left side of the vehicle 100 and transmits the generated image data (hereinafter referred to as "left side image data IMG3") to the ECU 90. The right side camera 44d captures the surrounding area on the right side of the vehicle 100 and transmits the generated image data (hereinafter referred to as "right side image data IMG4") to the ECU 90. In the following, the front image data IMG1, rear image data IMG2, left side image data IMG3, and right side image data IMG4 are collectively referred to as "image data IMG" as necessary.

When a predetermined condition is met, the parking assistance system 10 acquires a feature point F based on the image data IMG. The feature point F is a predetermined range of the image captured by the camera 45 where the brightness changes significantly.

For example, when the camera 45 captures an image of the parking lot 62 shown in FIG. 4, the corners of the concrete cover member 63B, the corners of the ground 63 made of grass 63L, and the boundary between the ground 63 made of cover member 63B and the ground 63 made of grass 63L are acquired as feature points F.

The ground 63 of the parking lot 62 shown in FIG. 4 is made up of a ground 63 made of concrete 63C and a ground 63 made of grass 63L. In addition, at the entrance 62ent of the parking lot 62, multiple concrete cover members 63B that cover the side gutter are installed side by side. Therefore, the ground 63 at the entrance 62ent of the parking lot 62 is made up of the surface of the cover members 63B.

Based on the forward image data IMG1, the parking assistance system 10 acquires a feature point F (hereinafter, "forward feature point F1") on the ground 63 in a predetermined range 71 (see FIG. 5) in front of the vehicle 100. Based on the rear image data IMG2, the parking assistance system 10 acquires a feature point F (hereinafter, "rear feature point F2") on the ground 63 in a predetermined range 72 (see FIG. 5) in the rear of the vehicle 100. Based on the left side image data IMG3, the parking assistance system 10 acquires a feature point F (hereinafter, "left side feature point F3") on the ground 63 in a predetermined range 73 (see FIG. 6) on the left side of the vehicle 100. Based on the right side image data IMG4, the parking assistance system 10 acquires a feature point F (hereinafter, "right side feature point F4") on the ground 63 in a predetermined range 74 (see FIG. 6) on the right side of the vehicle 100.

As shown in FIG. 5, the predetermined range 71 is a range defined by lines L711, L712, L713, and L714. Line L711 is a line L711 that extends in the vehicle width direction Dy from the front camera 44a a predetermined distance Dset away from the front of the vehicle 100. Line L712 is a line L712 that extends in the vehicle width direction Dy through the front camera 44a. Line L713 is a line that extends in the vehicle front-rear direction Dx from the front camera 44a to the left of the vehicle 100 a predetermined distance Dset away. Line L714 is a line that extends in the vehicle front-rear direction Dx from the front camera 44a to the right of the vehicle 100 a predetermined distance Dset away. Hereinafter, the predetermined range 71 is referred to as the "forward range 71".

The front range 71 is divided into eight ranges 71D, for example, divided into four equal parts in the vehicle width direction Dy and two equal parts in the vehicle front-rear direction Dx. That is, the front range 71 is divided into a plurality of ranges 71D of equal area. Hereinafter, these ranges 71D are referred to as "front divided ranges 71D." Furthermore, of the front divided ranges 71D, the two front divided ranges 71D set at the left end in the vehicle width direction Dy are referred to as "left end divided range 71D3," the two front divided ranges 71D set at the right end in the vehicle width direction Dy are referred to as "right end divided range 71D4," and the four front divided ranges 71D set in the middle in the vehicle width direction Dy are referred to as "middle divided range 71D5."

As shown in FIG. 5, the predetermined range 72 is a range defined by lines L721, L722, L723, and L724. Line L721 is a line that passes through the rear camera 44b and extends in the vehicle width direction Dy. Line L722 is a line that extends in the vehicle width direction Dy from the rear camera 44b to the rear of the vehicle 100 at a predetermined distance Dset. Line L723 is a line that extends in the vehicle front-rear direction Dx from the rear camera 44b to the left of the vehicle 100 at a predetermined distance Dset. Line L724 is a line that extends in the vehicle front-rear direction Dx from the rear camera 44b to the right of the vehicle 100 at a predetermined distance Dset. Hereinafter, the predetermined range 72 is referred to as the "rear range 72".

The rear range 72 is divided into eight ranges 72D, which are divided into four equal parts in the vehicle width direction Dy and two equal parts in the vehicle front-rear direction. That is, the rear range 72 is divided into a plurality of ranges 72D of equal area. Hereinafter, these ranges 72D are referred to as "rear divided ranges 72D." In addition, of the rear divided ranges 72D, the two rear divided ranges 72D set at the left end in the vehicle width direction Dy are referred to as "left end divided range 72D3," the two rear divided ranges 72D set at the right end in the vehicle width direction Dy are referred to as "right end divided range 72D4," and the four rear divided ranges 72D set in the middle in the vehicle width direction Dy are referred to as "middle divided range 72D5."

As shown in FIG. 6, the predetermined range 73 is a range defined by lines L731, L732, L734, and L733. Line L731 is a line that extends in the vehicle width direction Dy from the left side camera 44c forward of the vehicle 100 at a predetermined distance Dset. Line L732 is a line that extends in the vehicle width direction Dy from the left side camera 44c rearward of the vehicle 100 at a predetermined distance Dset. Line L733 is a line that extends in the vehicle front-rear direction Dx from the left side camera 44c to the left of the vehicle 100 at a predetermined distance Dset. Line L734 is a line that extends in the vehicle front-rear direction Dx through the left side camera 44c. Hereinafter, the predetermined range 73 is referred to as the "left side range 73".

The left side area 73 is divided into eight areas 73D, which are divided into four equal parts in the vehicle longitudinal direction Dx and two equal parts in the vehicle width direction Dy. That is, the left side area 73 is divided into a plurality of areas 73D of equal area. Hereinafter, these areas 73D are referred to as "left side divided area 73D." In addition, among the left side divided area 73D, the two left side divided area 73D set at the front end in the vehicle longitudinal direction Dx are referred to as "front end divided area 73D1," the two left side divided area 73D set at the rear end in the vehicle longitudinal direction Dx are referred to as "rear end divided area 73D2," and the four left side divided area 73D set in the middle in the vehicle longitudinal direction Dx are referred to as "middle divided area 73D5."

As shown in FIG. 6, the predetermined range 74 is a range defined by lines L741, L742, L743, and L744. Line L741 is a line extending in the vehicle width direction Dy from the right side camera 44d forward of the vehicle 100 at a predetermined distance Dset. Line L742 is a line extending in the vehicle width direction Dy from the right side camera 44d rearward of the vehicle 100 at a predetermined distance Dset. Line L743 is a line extending in the vehicle front-rear direction Dx through the right side camera 44d. Line L744 is a line extending in the vehicle front-rear direction Dx from the right side camera 44d to the right of the vehicle 100 at a predetermined distance Dset. Hereinafter, the predetermined range 74 is referred to as the "right range 74".

The right side area 74 is divided into eight areas 74D, which are divided into four equal parts in the vehicle longitudinal direction Dx and two equal parts in the vehicle width direction Dy. That is, the right side area 74 is divided into a plurality of areas 74D of equal area. Hereinafter, these areas 74D are referred to as "right side divided area 74D." In addition, among the right side divided area 74D, the two right side divided area 74D set at the front end in the vehicle longitudinal direction Dx are referred to as "front end divided area 74D1," the two right side divided area 74D set at the rear end in the vehicle longitudinal direction Dx are referred to as "rear end divided area 74D2," and the four right side divided area 74D set in the middle in the vehicle longitudinal direction Dx are referred to as "middle divided area 74D5."

When the image of the range corresponding to the characteristic point F captured by the camera 45 is converted into a planar image, the converted image of the range corresponding to the characteristic point F is an image of a square range 75 with a side of a predetermined length Lset, as shown in FIG. 7. When a predetermined condition is met, the parking assistance system 10 divides the characteristic point F into 25 equal square ranges 75D and acquires the luminance LUM of each range 75D. Then, the parking assistance system 10 subtracts the average value LUMave of the acquired luminance LUM from each luminance LUM to obtain a value ΔLUM (=LUM-LUMave), and acquires the tendency of the high and low luminance LUM at the characteristic point F as shading information CT based on these values ΔLUM. In other words, when a predetermined condition is met, the parking assistance system 10 acquires the shading pattern in the image of the characteristic point F captured by the camera 45 as shading information CT.

Referring again to FIG. 1, the display 50 is electrically connected to the ECU 90. The display 50 is disposed in a location of the vehicle 100 that is visible to the driver. In this embodiment, the display 50 is, for example, a display of a navigation device.

The ECU 90 can display various images on the display 50. In this embodiment, the ECU 90 can display at least a plan view image 51P, a parking space line image 52, a setting button image 53, a registration start button image 54, a registration button image 55, a parking start button image 56, a move button image 57, and a parking lot image 60 on the display 50.

The planar view image 51P is an image that includes a vehicle planar view image and a vehicle surroundings image. The vehicle planar view image is an image that displays the vehicle 100 when the vehicle 100 is viewed from above in the vertical direction. The vehicle surroundings image is an image that displays the surroundings of the vehicle 100 when the surroundings of the vehicle 100 are viewed from above in the vertical direction. These vehicle planar view image and vehicle surroundings image are generated by the ECU 90 based on the image data IMG.

The parking space line image 52 is an image that displays a parking space. A parking space is a space (or an area or region) in which the vehicle 100 is parked by the parking assistance control described below. As shown in FIG. 8, the parking space 61 is set within the parking lot 62.

The setting button image 53 is an image corresponding to a setting button that is touched by the driver to set (confirm) the parking space 61 in which the vehicle 100 is to be parked in the parking assistance control described below.

The registration start button image 54 is an image corresponding to the registration start button that is touched by the driver to start the parking and maneuvering process during registration of the parking assistance control described below.

The parking start button image 56 is an image corresponding to the parking start button that is touched by the driver to start the automatic parking and driving process described below when parking the vehicle 100 in a parking space 61 registered in the parking assistance system 10.

The registration button image 55 includes a first registration button image 551, a second registration button image 552, and a third registration button image 553.

The parking assistance system 10 can register (i.e., store) multiple parking information Ipark that differs depending on the parking assistance control described below. The parking information Ipark is information about a parking lot that is used by the parking assistance system 10 to automatically park the vehicle 100 in the parking lot. In this embodiment, the parking assistance system 10 can register up to three parking information Ipark. Hereinafter, these parking information Ipark are referred to as "first parking information Ipark_1", "second parking information Ipark_2", and "third parking information Ipark_3", respectively. Note that the number of parking information Ipark that can be registered is not limited to three, and may be two, or four or more.

The first registration button image 551 to the third registration button image 553 are images that are touched by the driver to register the parking lot information Ipark acquired by the parking assistance control described below in the parking assistance system 10. When the first registration button image 551 is touched by the driver, the parking lot information Ipark is registered in the parking assistance system 10 as the first parking lot information Ipark_1. When the second registration button image 552 is touched by the driver, the parking lot information Ipark is registered in the parking assistance system 10 as the second parking lot information Ipark_2. When the third registration button image 553 is touched by the driver, the parking lot information Ipark is registered in the parking assistance system 10 as the third parking lot information Ipark_3.

The parking lot image 60 includes a first parking lot image, a second parking lot image, and a third parking lot image. The first parking lot image is an image of the parking lot captured by the camera 45 and registered, and is a parking lot image corresponding to the first parking lot information Ipark_1. The second parking lot image is an image of the parking lot captured by the camera 45 and registered, and is a parking lot image corresponding to the second parking lot information Ipark_2. The third parking lot image is an image of the parking lot captured by the camera 45 and registered, and is a parking lot image corresponding to the third parking lot information Ipark_3.

[Overview of parking assistance system operation]
Next, an overview of the operation of the parking assistance system 10 will be described. The parking assistance system 10 is configured to be able to execute parking assistance control. The parking assistance control is a control for parking the vehicle 100 in a parking space without the driver needing to operate the accelerator pedal, the brake pedal, or the steering wheel.

There are parking lots where parking spaces are divided by lines such as white lines (hereinafter referred to as "dividing lines"). When automatically parking the vehicle 100 in such a parking lot, the vehicle can be driven automatically and parked in the parking space using the dividing lines captured by the camera 45 as a landmark.

On the other hand, there are parking lots, such as private parking lots, that do not have demarcation lines that define parking spaces. When automatically parking the vehicle 100 in such a parking lot, there are no demarcation lines to serve as landmarks. The parking assistance control performed by the parking assistance system 10 includes control to automatically park the vehicle 100 in the parking lot and to register parking lot information related to the parking lot, and control to automatically park the vehicle 100 in the parking lot whose parking lot information is registered.

[Parking a vehicle in an unregistered parking lot and registering it]
The parking assistance system 10 automatically parks the vehicle 100 in an unregistered parking lot 62 and also performs a process of registering the unregistered parking lot. Note that the contents of the process are simply reversed between when the unregistered parking lot 62 is on the right side of the vehicle 100 and when it is on the left side of the vehicle 100. For this reason, the following will describe the case where the unregistered parking lot 62 is on the left side of the vehicle 100, and will omit a description of the case where the unregistered parking lot 62 is on the right side of the vehicle 100.

When the vehicle 100 stops near the entrance 62ent of an unregistered parking lot 62 and the parking assist switch 15 (see FIG. 1) is turned ON, as shown in FIG. 10, the parking assist system 10 displays a planar image 51P, a setting button image 53, and a movement button image 57 (571-574) on the display 50 as shown in FIG. 9(A). In addition, the parking assist system 10 sets the area in which the vehicle 100 can be parked within the parking lot 62 based on the image data IMG as a parking space 61 (see FIG. 8), and displays a parking space line image 52 representing the set parking space 61 superimposed on the planar image 51P.

Here, the movement button image 57 includes an upward movement button image 571, a downward movement button image 572, a left movement button image 573, and a right movement button image 574. The upward movement button image 571 is an image operated by the driver to move the parking space line image 52 upward on the display 50. The downward movement button image 572 is an image operated by the driver to move the parking space line image 52 downward on the display 50. The left movement button image 573 is an image operated by the driver to move the parking space line image 52 left on the display 50. The right movement button image 574 is an image operated by the driver to move the parking space line image 52 right on the display 50.

Until the driver touches the setting button image 53, the driver can move the parking space line image 52 on the display 50 as desired by touching the movement button image 57. By moving the parking space line image 52 on the display 50, the driver can change the position of the parking space 61 in which the vehicle 100 is to be parked.

When the setting button image 53 is touched by the driver, the parking assistance system 10 acquires and stores as a parking lot image 60 an image of the parking lot 62 (particularly an image of the entrance 62ent of the parking lot 62) captured by the camera 45 when the vehicle 100 is stopped near the entrance 62ent of the parking lot 62. The parking assistance system 10 associates the parking lot image 60 with the position information of the vehicle 100 acquired by the position information acquisition device 30 and stores it.

When the driver touches the set button image 53, the parking assistance system 10 erases the set button image 53 and the move button image 57 from the display 50, and displays the registration start button image 54, as shown in FIG. 9(B). When the driver touches the set button image 53, the parking assistance system 10 sets the parking space 61 at the position corresponding to the parking space line image 52 displayed on the display 50 as the parking space 61set to be registered.

Furthermore, when the driver touches the setting button image 53, the parking assistance system 10 sets a target driving route Rtgt along which the vehicle 100 will travel in order to park the vehicle 100 in the registration target parking space 61set. For example, as shown in FIG. 10, when the vehicle 100 stops so that an unregistered parking space 62 is located to the left of the vehicle 100, the parking assistance system 10 sets the target driving route Rtgt shown in FIG. 11.

Furthermore, when the driver touches the setting button image 53 while the vehicle 100 is stopped in the state shown in FIG. 10, the parking assistance system 10 acquires two left side feature points F3 for each of the four intermediate divided ranges 73D5 of the left side range 73, acquires one left side feature point F3 for each of the two front end divided ranges 73D1, and acquires one left side feature point F3 for each of the two rear end divided ranges 73D2, as shown in FIG. 12 and FIG. 13, and stores these feature points F3 as the entrance feature point Fent of the unregistered parking lot 62. The entrance feature point Fent is the feature point F of the entrance 62ent of the parking lot 62.

After acquiring the entrance characteristic point Fent, the parking assistance system 10 stores the coordinates XY of the acquired entrance characteristic point Fent in the coordinate system Cpre as the entrance coordinates XYent, as shown in FIG. 12, and stores the shading information CT of the acquired entrance characteristic point Fent as the entrance shading information CTent. Here, the coordinate system Cpre refers to a coordinate system whose origin is a predetermined position Ppre in the parking space 61set to be registered.

When the registration start button image 54 is touched by the driver, the parking assistance system 10 performs a registration time parking driving process that drives the vehicle 100 along the target driving route Rtgt to the registration target parking space 61set. The registration time parking driving process is a process that controls the operation of the drive device 11, the steering device 12, and the brake device 13 so that the vehicle 100 drives along the target driving route Rtgt based on the image data IMG, object information OBJ, steering angle θst, steering torque TQst, vehicle speed SPD, etc.

For example, as shown in FIG. 10, when the vehicle 100 stops with an unregistered parking lot 62 located to the left of the vehicle 100, the parking assistance system 10 starts the registered parking and driving process, and first moves the vehicle 100 forward while turning right, and stops it, as shown in FIG. 14. Next, the parking assistance system 10 moves the vehicle 100 backward while turning left, as shown in FIG. 15.

The parking assistance system 10 acquires the characteristic point F at least once before parking of the vehicle 100 in the parking lot 62 is completed, and acquires the coordinates XY of the characteristic point F in the coordinate system Cpre and the shading information CT of the characteristic point F. In particular, after the vehicle 100 starts traveling in order to park the vehicle 100 in the parking lot 62, the parking assistance system 10 acquires the characteristic point F at least once when the vehicle 100 is predicted to travel straight without turning thereafter, and acquires the coordinates XY of the characteristic point F in the coordinate system Cpre and the shading information CT of the characteristic point F.

In this embodiment, the parking assistance system 10 acquires a rear feature point F2 while the vehicle 100 is being reversed at a point in time when the vehicle 100 is predicted to continue driving straight without turning thereafter (see FIG. 16). Specifically, the parking assistance system 10 acquires at least one rear feature point F2 present in each of the rear divided ranges 72D as an in-park intermediate feature point Fmid. The parking assistance system 10 also stores the coordinates XY in the coordinate system Cpre of the acquired in-park intermediate feature point Fmid as the in-park intermediate coordinates XYmid, and stores the shading information of the acquired in-park intermediate feature point Fmid as in-park intermediate shading information CTmid.

As shown in FIG. 17, when the entire vehicle 100 is within the registration target parking space 61set, the parking assistance system 10 stops the vehicle 100 and ends the registration time parking and driving process. This completes parking of the vehicle 100 in the parking lot 62.

When parking of the vehicle 100 in the parking lot 61 is completed, the parking assistance system 10 acquires at least one or more front feature points F1 present in each of the front divided ranges 71D, acquires at least one or more left feature points F3 present in each of the left divided ranges 73D, acquires at least one or more right feature points F4 present in each of the right divided ranges 74D, and stores these feature points F1, F3, and F4 as final feature points Ffin within the lot. The parking assistance system 10 also stores the coordinates XY of the acquired final feature points Ffin in the coordinate system Creg as final coordinates XYfin within the lot, and stores the shading information CT of the acquired final feature points Ffin as final shading information CTfin within the lot. The parking assistance system 10 also stores the coordinates XY of the parking space 61set to be registered in the coordinate system Creg as the registration space coordinates XYarea.

When parking of the vehicle 100 in the parking lot 62 is completed, the parking assistance system 10 displays the first registered button image 551 to the third registered button image 553 on the display 50, as shown in FIG. 9(C).

When the driver touches the first registration button image 551, the parking assistance system 10 registers the acquired entrance characteristic point Fent, the intermediate feature point Fmid, the final feature point Ffin, the registered section coordinates XYarea, and the parking lot image 60 as the first parking lot information Ipark_1. When the driver touches the second registration button image 552, the parking assistance system 10 registers the acquired entrance characteristic point Fent, the intermediate feature point Fmid, the final feature point Ffin, the registered section coordinates XYarea, and the parking lot image 60 as the second parking lot information Ipark_2. When the driver touches the third registration button image 553, the parking assistance system 10 registers the acquired entrance characteristic point Fent, the intermediate feature point Fmid, the final feature point Ffin, the registered section coordinates XYarea, and the parking lot image 60 as the third parking lot information Ipark_3.

[Parking a vehicle in a registered parking lot]
When the vehicle speed SPD of the vehicle 100 becomes equal to or lower than a predetermined vehicle speed SPDth (for example, equal to or lower than 15 km/h), the parking assistance system 10 performs a search process to search the camera image CMR for image portions having shading patterns that match the shading patterns of the registered entrance characteristic points Fent. The parking assistance system 10 performs this search process using the left side image data IMG3 and the right side image data IMG4. Here, the registered entrance characteristic points Fent are the entrance characteristic points Fent whose shading information is stored in the parking assistance system 10 by the above-mentioned registration process. The camera image CMR is an image of the current surroundings of the vehicle 100 captured by the camera 45.

When image portions having shading patterns that match the shading patterns of the multiple entrance feature points Fent are extracted from the camera image CMR, the parking assistance system 10 performs a comparison process (matching process) between the positional relationship between the multiple extracted image portions and the positional relationship between the multiple entrance feature points Fent having shading patterns that match the shading patterns of the multiple image portions.

When the positional relationship between the extracted image portions coincides with the positional relationship between the entrance feature points Fent that have a shading pattern that matches the shading pattern of the extracted image portions, the parking assistance system 10 determines that it has been able to detect a parking space 61 in a registered parking lot 62. A registered parking lot 62 is a parking lot whose parking lot information Ipark has been stored in the parking assistance system 10 by the above-mentioned registration process.

More specifically, if the positional relationship between multiple image portions extracted based on the left side image data IMG3 matches the positional relationship between multiple entrance feature points Fent that have shading patterns that match the shading patterns of the multiple image portions, the parking assistance system 10 determines that it has detected a parking space 61 of a registered parking lot 62 on the left side of the vehicle 100.

On the other hand, if the positional relationship between the multiple image portions extracted based on the right side image data IMG4 matches the positional relationship between the multiple entrance feature points Fent that have shading patterns that match the shading patterns of the multiple image portions, the parking assistance system 10 determines that it has detected a parking space 61 of a registered parking lot 62 on the right side of the vehicle 100.

Furthermore, if no image portion having a shading pattern matching the shading patterns of the multiple entrance feature points Fent is extracted from the camera image CMR, the parking assistance system 10 determines that the parking space 61 in the registered parking lot 62 could not be detected. Alternatively, the parking assistance system 10 also determines that the parking space 61 in the registered parking lot 62 could not be detected if the positional relationship between the multiple extracted image portions does not match the positional relationship between the multiple entrance feature points Fent having shading patterns matching the shading patterns of the multiple image portions.

When the parking assistance system 10 determines that the parking assistance switch 15 (see FIG. 1) has been turned ON and that a registered parking lot 62 has been detected, it displays a planar image 51P and a parking start button image 56 on the display 50, as shown in FIG. 18(A). At this time, a parking space line image 52 corresponding to the registered parking space 61 is superimposed on the planar image 51P.

When the driver touches the parking start button image 56, the parking assistance system 10 erases the parking start button image 56 from the display 50, as shown in FIG. 18(B). Also, when the driver touches the parking start button image 56, the parking assistance system 10 sets the parking space 61 at a position corresponding to the parking space line 52 in the planar image 51P displayed on the display 50 as the target parking space 61tgt.

Furthermore, when the parking start button image 56 is touched by the driver, the parking assistance system 10 sets a target driving route Rtgt along which the vehicle 100 will travel in order to park the vehicle 100 in the target parking space 61tgt.

The parking assistance system 10 then performs automatic parking and driving processing to drive the vehicle 100 along the target driving route Rtgt to the target parking space 61tgt. The automatic parking and driving processing is processing that controls the operation of the drive device 11, the steering device 12, and the brake device 13 so that the vehicle 100 drives along the target driving route Rtgt based on the image data IMG, object information OBJ, steering angle θst, steering torque TQst, vehicle speed SPD, etc.

The parking assistance system 10 performs a safety judgment process to determine whether the vehicle 100 can be safely driven to the target parking space 61tgt without coming into contact with a three-dimensional object in the parking lot 62 when the vehicle 100 is driven along the target driving route Rtgt during the execution of the automatic parking and driving process. If the parking assistance system 10 determines that the vehicle 100 cannot be safely driven to the target parking space 61tgt, it corrects the target driving route Rtgt so that the vehicle 100 can be safely driven to the target parking space 61tgt. The vehicle parking assistance device 10 performs the above safety judgment process using the image data IMG and object information OBJ acquired during the execution of the automatic parking and driving process.

During the automatic parking and maneuvering process, the parking assistance system 10 performs a search process in which it searches the camera image CMR for image portions having shading patterns that match the shading patterns of the multiple registered feature points Freg. The parking assistance system 10 performs this search process using the rear image data IMG2, the left side image data IMG3, and the right side image data IMG4. Here, the registered feature points Freg are feature points F for which the intermediate feature point Fmid and the final feature point Ffin have been registered by the parking assistance control described above.

When image portions having shading patterns that match the shading patterns of the multiple registered feature points Freg are extracted from the camera image CMR, the parking assistance system 10 performs a matching process between the positional relationship between the multiple extracted image portions and the positional relationship between the multiple registered feature points Freg having shading patterns that match the shading patterns of the multiple image portions.

When the positional relationship between the extracted image portions coincides with the positional relationship between the registered feature points Freg having a shading pattern that coincides with the shading pattern of the extracted image portions, the parking assistance system 10 performs a parking position determination process to determine whether the position of the target parking space 61tgt in the parking lot 62 coincides with the position indicated by the registered space coordinates XYarea based on the positional relationship between the coordinates XY of the registered feature point Freg having a shading pattern that coincides with the shading pattern of the extracted image portions and the registered space coordinates XYarea. When the parking assistance system 10 determines that the position of the target parking space 61tgt in the parking lot 62 does not coincide with the position indicated by the registered space coordinates XYarea, it corrects the position of the target parking space 61tgt so that the position of the target parking space 61tgt in the parking lot 62 coincides with the position indicated by the registered space coordinates XYarea, and corrects the target driving route Rtgt so that the vehicle 100 can be parked in the target parking space 61tgt whose position has been corrected.

When the entire vehicle 100 is within the target parking space 61tgt, the parking assistance system 10 stops the vehicle 100 and ends the automatic parking and maneuvering process. This completes parking of the vehicle 100 in the parking lot 62.

[Display control of display 50]
The parking assistance control for the above-mentioned registered parking lot 62 can be started when the parking assistance system 10 matches the feature points of the camera image CMR (left side image data IMG3, right side image data IMG4) captured by the camera 45 with the entrance feature point Fent of the registered parking lot 62 and detects the parking space 61 of the parking lot 62.

Therefore, if the current position of the vehicle 100 is significantly away from the position where the vehicle 100 was parked when the parking lot 62 was registered, specifically, the position where the vehicle 100 was parked near the entrance 62ent of the parking lot 62 and the registration operation was performed (hereinafter referred to as the vehicle position at the time of registration), the parking assistance system 10 fails in the matching process and is unable to start parking assistance control.

In such a situation, in order for the parking assistance system 10 to start parking assistance control, the driver needs to align the current position of the vehicle 100 with the vehicle position at the time of registration. However, the parking assistance system 10 is in a state where it is unable to detect the parking space 61 in the parking lot 62, and there is an issue in that it is unable to properly present to the driver where to move the vehicle 100 to align it with the vehicle position at the time of registration.

On the other hand, the driver may forget the vehicle's location at the time of registration, or may not know the registered parking lot 62 because the registration operation was performed by someone other than the driver (for example, a family member who shares the vehicle 100). In such cases, the driver may not realize that the current location of the vehicle 100 is far from the vehicle's location at the time of registration, and may not understand why the parking assistance system 10 cannot start parking assistance control.

To solve this problem, the parking assistance system 10 of the present embodiment performs display control to display a registered parking lot image 60 and an image (planar image) of the current surroundings of the vehicle 100 side by side on the display 50 when the current position of the vehicle 100 is within a predetermined distance range from a registered parking lot 62 and a parking space 61 in the parking lot 62 cannot be detected. By displaying these images side by side, it is possible to properly inform the driver that the current position of the vehicle 100 is far from the vehicle position at the time of registration. Details of the display control are described below.

Figure 19 is a flowchart showing a display control routine. The ECU 90 (CPU) of the parking assistance system 10 starts the routine shown in Figure 19 at the same time that the vehicle 100 starts traveling.

In step S100, the ECU 90 determines whether the vehicle speed SPD of the vehicle 100 is equal to or less than a predetermined vehicle speed SPDth based on detection by the vehicle speed sensor 25. If the vehicle speed SPD is equal to or less than the predetermined vehicle speed SPDth (Yes), the ECU 90 advances the process to step S110. On the other hand, if the vehicle speed SPD is not equal to or less than the predetermined vehicle speed SPDth (No), the ECU 90 repeats the determination process of step S100.

In step S110, the ECU 90 acquires a camera image CMR of the surroundings of the vehicle 100 captured by the camera 45. Next, in step S120, the ECU 90 performs a matching process between the feature points of the camera image CMR (left side image data IMG3, right side image data IMG4) acquired in step S110 and the registered entrance feature point Fent.

In step S130, the ECU 90 determines whether the parking assist switch 15 has been turned ON. If the parking assist switch 15 has been turned ON (Yes), the ECU 90 accepts a parking assist request from the driver and proceeds to step S140. On the other hand, if the parking assist switch 15 has not been turned ON (No), in other words, if the parking assist switch 15 is OFF, the ECU 90 returns the process to step S100.

In step S140, the ECU 90 determines whether or not a parking space 61 in a registered parking lot 62 has been detected by the matching process in step S120. If a parking space 61 in a parking lot 62 has been detected (Yes), the ECU 90 advances the process to step S150. In step S150, the ECU 90 displays the planar image 51P, the registered parking space line image 52, and the parking start button image 56 on the display 50 (see FIG. 18(A)).

Next, in step S152, the ECU 90 determines whether a predetermined end condition is met. Here, the end conditions include (1) when the parking assist switch 15 is turned OFF, (2) when the vehicle speed SPD of the vehicle 100 becomes higher than a predetermined vehicle speed SPDth, and (3) when a predetermined upper limit time has elapsed since the image display in step S150. If any of these end conditions (1) to (3) is met, the ECU 90 determines that the end condition is met and proceeds to step S190. In step S190, the ECU 90 ends the image display and then ends (returns) this routine.

On the other hand, if it is determined in step S152 that the termination condition is not satisfied, the ECU 90 advances the process to step S154. In step S154, the ECU 90 determines whether or not the parking start button image 56 has been touched by the driver. If the parking start button image 56 has not been touched (No), the ECU 90 returns the process to step S150. In other words, the image display is continued. On the other hand, if the parking start button image 56 has been touched (Yes), the ECU 90 advances the process to step S156.

In step S156, the ECU 90 performs automatic parking and driving processing to automatically park the vehicle 100 in a registered parking lot 62, and then ends this routine.

On the other hand, if it is determined in step S140 that the parking space 61 of the registered parking lot 62 cannot be detected (No), the ECU 90 advances the process to step S145. In step S145, the ECU 90 determines whether the current position of the vehicle 100 is within a predetermined distance range from the registered parking lot 62. For the current position of the vehicle 100, the position information acquired by the position information acquisition device 30 is used. For the position information of the registered parking lot 62, the position information stored in association with the parking lot image 60 at the time of registration is used. The ECU 90 compares these position information to determine whether a registered parking lot 62 exists within a predetermined distance range from the current position of the vehicle 100. If there are multiple registered parking lots 62, the ECU 90 identifies the parking lot 62 closest to the current position of the vehicle 100 and compares the position information of the identified parking lot 62 with the current position of the vehicle 100.

If it is determined in step S145 that the current position of the vehicle 100 is within a predetermined distance range from the registered parking lot 62 (Yes), the ECU 90 advances the process to step S160. Here, the predetermined distance used in the determination in step S145 is not particularly limited, but it is preferable to set it based on a distance (e.g., several meters to several tens of meters) that allows the driver to easily move the vehicle 100 to the vehicle position at the time of registration while referring to the images 60, 51P displayed on the display 50 in step S160.

In step S160, the ECU 90 displays the parking lot image 60 of the registered parking lot 62 and the image of the current surroundings of the vehicle 100 captured by the camera 45 (planar image 51P) side by side on the display 50. Furthermore, if there are multiple registered parking lots 62, the ECU 90 displays the parking lot image 60 of the parking lot 62 identified in the processing of step S145 as being closest to the current position of the vehicle 100 side by side with the image of the current surroundings of the vehicle 100 (planar image 51P). In other words, even if multiple parking lots (62) are registered, the parking lot image 60 of the parking lot 62 that is most suitable for the current position of the vehicle (100) is displayed on the display 50.

Figure 20 is an example of an image displayed on the display 50 in the processing of step S160. The ECU 90 displays a parking lot image 60 of a registered parking lot 62 on the left side of the display 50, and displays an image of the current surroundings of the vehicle 100 captured by the camera 45 (planar image 51P) on the right side of the display 50. Note that the positional relationship of these images is not limited to the illustrated example, and the left-right positional relationship may be swapped, or these images may be arranged one above the other.

The parking lot image 60 is an image of the parking lot 62 registered by the above-mentioned registration process, viewed from above in the vertical direction. An image showing the vehicle 100 parked at the vehicle position P at the time of registration and a parking space line image 52 showing the parking space (61) are superimposed on the parking lot image 60. The image of the current surroundings of the vehicle 100 is a planar view image 51P generated based on image data IMG captured by the camera 45, and includes a vehicle planar view image and a vehicle surroundings image. The vehicle planar view image is an image showing the vehicle 100 when viewed from above in the vertical direction. The vehicle surroundings image is an image showing the surroundings of the vehicle 100 when viewed from above in the vertical direction. In the example shown in FIG. 20, a part of the parking lot 62 is captured in front of the vehicle 100 in the planar view image 51P.

By referring to the parking lot image 60 and the image of the current surroundings of the vehicle 100 (planar image 51P) displayed side by side on the display 50, the driver can easily grasp how far the current position of the vehicle 100 is from the vehicle position P at the time of registration, and further, in which direction the vehicle 100 needs to be moved to align with the vehicle position P at the time of registration. In the illustrated example, the vehicle position P at the time of registration is located in front of the vehicle 100, and the driver can grasp that moving the vehicle 100 forward will align the vehicle with the vehicle position P at the time of registration. In other words, by appropriately presenting to the driver through an image in which direction and how far the vehicle 100 needs to be moved to align with the vehicle position P at the time of registration, convenience is improved.

In step S162, the ECU 90 again determines whether or not a parking space 61 in a registered parking lot 62 has been detected through the matching process. If a parking space 61 in the parking lot 62 has been detected (Yes), the ECU 90 advances the process to step S150. That is, the ECU 90 can start parking assistance control. On the other hand, if a parking space 61 in the parking lot 62 cannot be detected (No), the ECU 90 advances the process to step S164.

In step S164, the ECU 90 determines whether a predetermined end condition is met. The end conditions include (1) when the parking assist switch 15 is turned OFF, (2) when the vehicle speed SPD of the vehicle 100 becomes higher than a predetermined vehicle speed SPDth, and (3) when a predetermined upper limit time has elapsed since the image display in step S160. If any of the end conditions (1) to (3) is met, the ECU 90 determines that the end condition is met and proceeds to step S190. In step S190, the ECU 90 ends the image display and then ends this routine.

On the other hand, if it is determined in step S145 that the current position of the vehicle 100 is not within the predetermined distance range from the registered parking lot 62 (No), i.e., if the vehicle 100 is far away from the registered parking lot 62, the ECU 90 proceeds to step S170 and displays a message image on the display 50 indicating that "parking assistance is not possible." In a situation in which the vehicle 100 is far away from the registered parking lot 62, even if the registered parking lot image 60 and the image 51P of the surroundings of the vehicle 100 are displayed on the display 50 as in step S160, it is difficult for the driver to align the vehicle 100 with the vehicle position P at the time of registration, and this is ineffective. In such a case, the ECU 90 notifies the driver that the system cannot perform parking assistance control.

Next, in step S175, the ECU 90 determines whether a predetermined end condition is met. Here, the end conditions include (1) when the parking assist switch 15 is turned OFF, (2) when the vehicle speed SPD of the vehicle 100 becomes higher than a predetermined vehicle speed SPDth, and (3) when a predetermined upper limit time has elapsed since the image display in step S170. If any of these end conditions (1) to (3) is met, the ECU 90 determines that the end condition is met and proceeds to step S190. In step S190, the ECU 90 ends the image display and then ends this routine.

The above is a specific operation of the parking assistance system 10. According to the parking assistance system 10, when a parking assistance request is received from the driver, if the current position of the vehicle 100 is far from the vehicle position P at the time of registration and the parking space 61 of the registered parking lot 62 cannot be detected by the matching process, the registered parking lot image 60 and the image 51P of the current surroundings of the vehicle 100 are displayed side by side on the display 50. In other words, it is possible to properly inform the driver that the current position of the vehicle 100 is far from the vehicle position P at the time of registration. This allows the driver to easily know in which direction to move the vehicle 100 to align it with the vehicle position P at the time of registration, which reliably improves convenience.

The present invention is not limited to the above embodiment, and various modifications can be made within the scope of the present invention.

10...Parking assistance system, 11...Drive device, 12...Steering device, 13...Braking device, 15...Parking assistance switch, 30...Position information acquisition device, 40...Surrounding recognition device, 41...Sonar sensor, 44...Camera sensor, 51P...Planar image, 52...Parking space line image, 60...Parking space image, 62...Parking space, 90...ECU, 100...Vehicle

Claims (3)

an imaging device mounted on the vehicle for capturing images of the surroundings of the vehicle and images of a parking lot of the parking lot for use in automatically parking the vehicle in the parking lot;
a control device that registers the parking lot image captured by the imaging device as parking lot information for the parking lot, and performs parking assistance control to automatically park the vehicle in the parking lot based on the parking lot information when a parking assistance request is received from a driver;
A parking assistance system including a display device capable of displaying an image captured by the imaging device,
The control device includes:
The surrounding image captured by the imaging device is compared with the parking lot image registered as the parking lot information, and the parking assistance control is started when the parking lot is detected by the comparison process.
When the parking lot cannot be detected by the matching process, the parking lot image registered as the parking lot information and the current surrounding image captured by the imaging device are displayed on the display device. 2. The parking assistance system according to claim 1,
A position information acquisition device for acquiring position information of the vehicle is further provided.
The control device includes:
The parking lot image captured by the imaging device is associated with location information acquired by the location information acquisition device when the image is captured, and is registered as the parking lot information of the parking lot;
If the parking lot cannot be detected by the matching process, it is determined whether the current position of the vehicle is within a predetermined distance range from the parking lot based on the current position information of the vehicle acquired by the position information acquisition device and the position information of the parking lot registered as the parking lot information,
When it is determined that the current position of the vehicle is within a predetermined distance range from the parking lot, the parking lot image registered as the parking lot information and the current surrounding image captured by the imaging device are displayed on the display device, while when it is determined that the current position of the vehicle is not within a predetermined distance range from the parking lot, a message to the effect that the parking assistance control cannot be executed is displayed on the display device. 3. The parking assistance system according to claim 2,
The control device includes:
The parking lot images of the plurality of parking lots captured by the imaging device can be registered as parking lot information of each parking lot in association with location information acquired by the location information acquisition device when the images are captured,
If none of the multiple parking lots can be detected by the matching process, the parking lot closest to the current position of the vehicle among the multiple parking lots is identified based on the current position information of the vehicle acquired by the position information acquisition device and the position information of the multiple parking lots registered as the parking lot information, and if it is determined that the current position of the vehicle is within a predetermined distance range from the identified parking lot, a parking lot image of the identified parking lot and the current surrounding image captured by the imaging device are displayed on the display device.

Images