JP7669092B2 - Parking assistance method and parking assistance device - Google Patents

Description

This disclosure relates to a parking assistance method and a parking assistance device.

Conventionally, parking assistance technology is known that moves a vehicle by automatic driving when parking the vehicle. One such parking assistance technology is a technology that learns a driving route based on teacher driving by the driver and uses the learning results to provide parking assistance. This technology is used, for example, when parking in a fixed parking position repeatedly, such as a parking lot at the user's home or workplace.

Patent No. 3286306

However, it can be difficult for users to objectively grasp the vehicle's driving path based on the supervised driving and the positional relationship between the objects and roads around the vehicle.

This disclosure provides a parking assistance method and a parking assistance device that allow a user to easily understand the vehicle's driving path based on teacher driving and the positional relationship between the objects and roads around the vehicle.

The parking assistance method according to the present disclosure is a parking assistance method for automatically driving a vehicle based on teacher driving by a driver, and includes a first acquisition step of acquiring captured images of the surroundings of the vehicle during teacher driving in time series together with the movement of the vehicle from an imaging device provided in the vehicle, a route recording step of recording the driving route of the vehicle during teacher driving, a first extended overhead image generation step of generating a first extended overhead image of a display target range including the driving route and the surroundings of the driving route from above based on the captured images, a first display step of displaying the first extended overhead image on a display unit, and a start/end operation reception step of receiving an operation of starting and ending the recording of the teacher driving by a user. In the first display step, an overhead route image in which a driving route image showing the recorded driving route is superimposed on the first extended overhead image is displayed on the display unit. In addition, when an operation of ending the recording of the teacher driving is received in the first display step, the display of the overhead route image is executed by the first display step. In the first display step, the bird's-eye view route image is displayed before an operation for registering the teacher driving is accepted. In the first acquisition step, information regarding the distance between the vehicle and objects around the vehicle during teacher driving is further acquired. In the first display step, an outline image showing a movement trajectory of the outline of the vehicle body during teacher driving is superimposed on the first extended bird's-eye view image and a portion of the outline image whose distance is equal to or less than a threshold is displayed on the bird's-eye view route image in a manner different from a portion whose distance is greater than the threshold.

The parking assistance method and parking assistance device disclosed herein allow the user to easily understand the vehicle's driving path based on the supervised driving and the positional relationship between the objects and roads around the vehicle.

FIG. 1 is a diagram illustrating an example of a vehicle equipped with a parking assistance device according to a first embodiment. FIG. 2 is a diagram showing an example of a configuration in the vicinity of a driver's seat of a vehicle according to the first embodiment. FIG. 3 is a diagram illustrating an example of a hardware configuration of the parking assistance device according to the first embodiment. FIG. 4 is a diagram illustrating an example of teacher running according to the first embodiment. FIG. 5 is a block diagram showing an example of functions provided in the parking assistance device according to the first embodiment. FIG. 6 is a diagram illustrating an example of a driving route image according to the first embodiment. FIG. 7 is a diagram illustrating an example of an outline image according to the first embodiment. FIG. 8 is a diagram illustrating an example of an overhead image according to the first embodiment. FIG. 9 is a diagram illustrating an example of an extended overhead image according to the first embodiment. FIG. 10 is a diagram illustrating an example of an overhead route image according to the first embodiment. FIG. 11 is a flowchart showing an example of the flow of a driving route recording process executed by the parking assistance device according to the first embodiment. FIG. 12 is a diagram illustrating an example of an overhead route image according to the second embodiment. FIG. 13 is a diagram illustrating an example of an overhead route image according to the third embodiment. FIG. 14 is a diagram showing an example of a driving route selection screen according to the fourth embodiment. FIG. 15 is a flowchart showing an example of the flow of a driving route recording process executed by the parking assistance device according to the fourth embodiment. FIG. 16 is a diagram illustrating an example of an overhead route image according to the fifth embodiment.

Below, we will explain embodiments of the parking assistance method and parking assistance device disclosed herein with reference to the drawings.

(First embodiment)
Fig. 1 is a diagram showing an example of a vehicle 1 equipped with a parking assistance device 100 according to the first embodiment. As shown in Fig. 1, the vehicle 1 includes a vehicle body 12 and two pairs of wheels 13 arranged along a predetermined direction on the vehicle body 12. The two pairs of wheels 13 include a pair of front tires 13f and a pair of rear tires 13r.

The front tire 13f shown in FIG. 1 is an example of a first wheel in this embodiment. The rear tire 13r is an example of a second wheel in this embodiment. Note that the vehicle 1 shown in FIG. 1 has four wheels 13, but the number of wheels 13 is not limited to this. For example, the vehicle 1 may be a two-wheeled vehicle.

The vehicle body 12 is coupled to wheels 13 and can move by the wheels 13. In this case, the predetermined direction in which the two pairs of wheels 13 are arranged is the traveling direction of the vehicle 1. The vehicle 1 can move forward or backward by switching gears (not shown) or the like. The vehicle 1 can also turn right or left by steering.

The vehicle body 12 has a front end F, which is the end on the front tire 13f side, and a rear end R, which is the end on the rear tire 13r side. The vehicle body 12 has a generally rectangular shape when viewed from above, and the four corners of the generally rectangular shape are sometimes called ends. The vehicle 1 also has a display device, a speaker, and an operation unit, although these are not shown in FIG. 1.

A pair of bumpers 14 are provided at the front and rear ends F and R of the vehicle body 12 near the bottom end of the vehicle body 12. Of the pair of bumpers 14, the front bumper 14f covers the entire front surface and part of the side surface near the bottom end of the vehicle body 12. Of the pair of bumpers 14, the rear bumper 14r covers the entire rear surface and part of the side surface near the bottom end of the vehicle body 12.

At predetermined ends of the vehicle body 12, wave transmitting/receiving units 15f, 15r that transmit and receive sound waves such as ultrasonic waves are disposed. For example, one or more wave transmitting/receiving units 15f are disposed on the front bumper 14f, and one or more wave transmitting/receiving units 15r are disposed on the rear bumper 14r. Hereinafter, when the wave transmitting/receiving units 15f, 15r are not particularly limited, they will be simply referred to as the wave transmitting/receiving unit 15. Furthermore, the number and positions of the wave transmitting/receiving units 15 are not limited to the example shown in FIG. 1. For example, the vehicle 1 may be provided with wave transmitting/receiving units 15 on the left and right sides.

In this embodiment, sonar using ultrasonic waves is described as an example of the wave transmitting and receiving unit 15, but the wave transmitting and receiving unit 15 may be a radar that transmits and receives electromagnetic waves. Alternatively, the vehicle 1 may be equipped with both sonar and radar. Also, the wave transmitting and receiving unit 15 may simply be referred to as a sensor.

The wave transmitting/receiving unit 15 detects obstacles around the vehicle 1 based on the results of transmitting and receiving sound waves or electromagnetic waves. The wave transmitting/receiving unit 15 also measures the distance between the vehicle 1 and obstacles around the vehicle 1 based on the results of transmitting and receiving sound waves or electromagnetic waves.

The vehicle 1 also includes a first imaging device 16a that captures an image in front of the vehicle 1, a second imaging device 16b that captures an image behind the vehicle 1, a third imaging device 16c that captures an image on the left side of the vehicle 1, and a fourth imaging device that captures an image on the right side of the vehicle 1. The fourth imaging device is not shown.

Hereinafter, when there is no particular distinction between the first imaging device 16a, the second imaging device 16b, the third imaging device 16c, and the fourth imaging device, they will simply be referred to as imaging device 16. The positions and number of imaging devices are not limited to the example shown in FIG. 1. For example, the vehicle 1 may also be equipped with only two imaging devices, the first imaging device 16a and the second imaging device 16b. Alternatively, the vehicle 1 may have further imaging devices in addition to the above examples.

The imaging device 16 is capable of capturing images of the surroundings of the vehicle 1, and is, for example, a camera that captures color images. The images captured by the imaging device 16 may be video or still images. The imaging device 16 may also be a camera built into the vehicle 1, or may be a camera of a drive recorder that is retrofitted to the vehicle 1.

The vehicle 1 is also equipped with a parking assistance device 100. The parking assistance device 100 is an information processing device that can be installed in the vehicle 1, and is, for example, an ECU (Electronic Control Unit) or an OBU (On Board Unit) provided inside the vehicle 1. Alternatively, the parking assistance device 100 may be an external device installed near the dashboard of the vehicle 1. The parking assistance device 100 may also function as a car navigation device, etc.

Next, the configuration of the vicinity of the driver's seat of the vehicle 1 of this embodiment will be described. Figure 2 is a diagram showing an example of the configuration of the vicinity of the driver's seat 130a of the vehicle 1 according to the first embodiment.

As shown in FIG. 2, the vehicle 1 has a driver's seat 130a and a passenger seat 130b. In front of the driver's seat 130a, a windshield 180, a dashboard 190, a steering wheel 140, a display device 120, and operation buttons 141 are provided.

The display device 120 is a display provided on the dashboard 190 of the vehicle 1. As an example, the display device 120 is located in the center of the dashboard 190 as shown in FIG. 2. The display device 120 is, for example, a liquid crystal display or an organic EL (Electro Luminescence) display. The display device 120 may also function as a touch panel. The display device 120 is an example of a display unit in this embodiment.

The steering wheel 140 is provided in front of the driver's seat 130a and can be operated by the driver. The rotation angle of the steering wheel 140, i.e., the steering angle, is electrically or mechanically linked to the change in the direction of the front tires 13f, which are the steered wheels. The steered wheels may be the rear tires 13r, or both the front tires 13f and the rear tires 13r may be steered wheels.

The operation button 141 is a button that can accept an operation by a user. In this embodiment, the user is, for example, the driver of the vehicle 1. The operation button 141 accepts an operation from the driver to start parking assistance, for example, by accepting a press from the driver. The position of the operation button 141 is not limited to the example shown in FIG. 2, and may be provided on the steering wheel 140, for example. The operation button 141 is an example of an operation unit in this embodiment. In addition, if the display device 120 also serves as a touch panel, the display device 120 may be an example of an operation unit. In addition, an operation terminal that can transmit a signal to the vehicle 1 from outside the vehicle 1, such as a tablet terminal, smartphone, remote controller, or electronic key (not shown), may be an example of an operation unit.

Next, the hardware configuration of the parking assistance device 100 will be described. FIG. 3 is a diagram showing an example of the hardware configuration of the parking assistance device 100 according to the first embodiment. As shown in FIG. 3, the parking assistance device 100 has a CPU (Central Processing Unit) 11A, a ROM (Read Only Memory) 11B, a RAM (Random Access Memory) 11C, an I/F (Interface) 11D, a HDD (Hard Disk Drive) 11E, etc., which are interconnected by a bus 11F, and has a hardware configuration that utilizes a normal computer.

The CPU 11A is a calculation device that controls the entire ECU. The CPU 11A is an example of a processor in the parking assistance device 100 of this embodiment, and another processor or processing circuit may be provided instead of the CPU 11A. The ROM 11B stores programs and the like that realize various processes by the CPU 11A. The RAM 11C is, for example, a main storage device of the parking assistance device 100, and stores data necessary for various processes by the CPU 11A. The I/F 11D is an interface for transmitting and receiving data. The I/F 11D transmits and receives data to and from other devices provided in the vehicle 1, such as the display device 120. The I/F 11D may also transmit and receive information to and from other ECUs mounted on the vehicle 1 via a CAN (Controller Area Network) or the like within the vehicle 1, or may communicate with an information processing device outside the vehicle 1 via a network such as the Internet.

Note that in Figures 1 to 3, the display device 120 is illustrated as a device separate from the parking assistance device 100, but the display device 120 may be included in the parking assistance device 100.

The parking assistance device 100 of this embodiment learns a driving route based on the driver's supervised driving, and performs parking assistance using the learning results. In other words, the parking assistance method executed by the parking assistance device 100 is a method of automatically driving the vehicle 1 based on the driver's supervised driving. This parking assistance method is effective for reducing the driver's parking effort when repeatedly parking in a fixed parking position, such as the driver's garage at home, a contracted parking position at an apartment complex, or a specified parking position in a parking lot at the driver's workplace, for example. This type of parking assistance is called home zone parking.

Figure 4 is a diagram illustrating an example of teacher driving according to the first embodiment. In the example shown in Figure 4, a parking space 910 located near the home 70 of the driver of vehicle 1 is the target parking position of vehicle 1. In addition, a garden 721 and a corner space 722 exist near the parking space 910. In the garden 721, there are trees 73a and 73b on the side facing the parking space 910.

In addition, around the parking space 910, there are other houses 71a, 71b, the parking spaces 711a-711c of the other houses 71a, 71b, and other vehicles 30a-30c. The trees 73a, 73b in the yard 721, the other houses 71a, 71b, the parking spaces 711a-711c of the other houses 71a, 71b, and other vehicles 30a-30c, etc., can be obstacles when the vehicle 1 is parking in the parking space 910. For this reason, the driver drives the vehicle 1 to avoid these objects when driving as a teacher. Note that objects that can be obstacles for the vehicle 1 are not limited to these.

In teacher driving, the driver drives the vehicle 1 manually from a starting position 900 on the road 40 to a parking space 910. The parking assistance device 100 records a driving route 80 from the starting position 900 to the parking space 910 in teacher driving. When the user determines that the recorded driving route 80 is the driving route 80 for automatic driving, the parking assistance device 100 registers the driving route 80 as the driving route 80 for automatic driving. Note that the user may repeat teacher driving multiple times until the desired driving route 80 is obtained.

After the registration of the driving route 80 by such teacher driving is completed, the parking assistance device 100 can park the vehicle 1 in the parking space 910 by automatically driving the vehicle 1 along the registered driving route 80.

Note that the parking space 910 and its surrounding environment shown in FIG. 4 are merely examples and are not limiting.

Next, the functions of the parking assistance device 100 of this embodiment will be described in detail. FIG. 5 is a block diagram showing an example of the functions of the parking assistance device 100 according to the first embodiment.

As shown in FIG. 5, the parking assistance device 100 of this embodiment includes a reception unit 101, an acquisition unit 102, an extraction unit 103, a route recording unit 104, a route image generation unit 105, an extended overhead image generation unit 106, a display control unit 107, a registration processing unit 108, an estimation unit 109, a vehicle control unit 110, and a memory unit 111.

The memory unit 111 is configured, for example, with a ROM 11B, a RAM 11C, or a HDD 11E. Note that, although FIG. 5 illustrates one memory unit 111 being included in the parking assistance device 100, multiple storage media may function as the memory unit 111.

The storage unit 111 stores programs and data used in various processes executed by the parking assistance device 100. For example, the program executed by the parking assistance device 100 of this embodiment has a modular configuration including the above-mentioned functional units (reception unit 101, acquisition unit 102, extraction unit 103, route recording unit 104, route image generation unit 105, extended overhead image generation unit 106, display control unit 107, registration processing unit 108, estimation unit 109, vehicle control unit 110), and as actual hardware, the CPU 11A reads out and executes the program from the storage unit 111, so that the above units are loaded onto the RAM 11C, and the reception unit 101, acquisition unit 102, extraction unit 103, route recording unit 104, route image generation unit 105, extended overhead image generation unit 106, display control unit 107, registration processing unit 108, estimation unit 109, and vehicle control unit 110 are generated on the RAM 11C. The processes performed by each functional unit of the parking assistance device 100 are also referred to as steps.

The program executed by the parking assistance device 100 of this embodiment is provided in the form of an installable or executable file recorded on a computer-readable recording medium such as a flash memory, CD-ROM, flexible disk (FD), CD-R, or DVD (Digital Versatile Disk).

The program executed by the parking assistance device 100 of this embodiment may be stored on a computer connected to a network such as the Internet and provided by downloading it via the network. The program executed by the parking assistance device 100 of this embodiment may be provided or distributed via a network such as the Internet. The program executed by the parking assistance device 100 of this embodiment may be provided by being pre-installed in ROM 11B or the like.

The reception unit 101 receives various operations from the user. For example, when the operation button 141 is pressed, the reception unit 101 receives operations by the user to start and end recording of the teacher run. In this embodiment, the start operation reception step, which is a process of receiving an operation to start recording of the teacher run, and the end operation reception step, which is a process of receiving an operation to end recording of the teacher run, are collectively referred to as the start and end operation reception step.

In addition, after the teacher driving is completed, the reception unit 101 receives a user operation to instruct whether or not to register the driving route 80 of the teacher driving as a driving route 80 for automatic driving.

In addition, the reception unit 101 receives an operation by the user to start parking assistance when, for example, the driving route 80 has already been registered and the operation button 141 is pressed.

The means by which the reception unit 101 receives user operations is not particularly limited. For example, if the display device 120 is a touch panel, the reception unit 101 may receive operations to start and end recording of a teacher drive or to start parking assistance when an image button on the touch panel is pressed.

The acquisition unit 102 acquires, from the multiple image capture devices 16a to 16c, multiple captured images of the surroundings of the vehicle 1 during teacher driving in time series as the vehicle 1 moves. Note that since the captured images are images of the surroundings of the vehicle 1, they are referred to as surrounding images in this embodiment. This acquisition process is an example of the first acquisition step in this embodiment. In addition, in the first acquisition step, the acquisition unit 102 further acquires information regarding the distance between the vehicle and objects surrounding the vehicle 1 during teacher driving.

The information regarding the distance between the vehicle 1 and objects around the vehicle 1 is, for example, the presence or absence of an obstacle detected by the wave transmitting/receiving unit 15, and the distance between the vehicle 1 and the obstacle detected around the vehicle 1. The presence or absence of an obstacle detected by the wave transmitting/receiving unit 15, and the distance between the vehicle 1 and the obstacle around the vehicle 1 measured by the wave transmitting/receiving unit 15 may be referred to as environmental information regarding the environment around the vehicle 1. Note that the environmental information is not limited to this information.

The acquisition unit 102 also acquires vehicle information of the vehicle 1 from various sensors of the vehicle 1 or from other ECUs. The vehicle information includes, for example, information related to the speed, steering angle, and braking operation of the vehicle 1. Each piece of information included in the vehicle information of the vehicle 1 is stored in the storage unit 111 in association with the time at which each piece of information was detected. Note that the vehicle information of the vehicle 1 may further include the wheel speed or the number of rotations of the wheels 13, the acceleration of the vehicle 1 measured by a gyro sensor, etc.

The extraction unit 103 extracts feature points from the surrounding image. The method of extracting feature points by the extraction unit 103 is not particularly limited, and a known method may be applied. For example, the extraction unit 103 extracts feature points by a method such as FAST (Features from Accelerated Segment Test) or ORB (Oriented FAST and Rotated BRIEF). In addition, when learning the driving route 80, the extraction unit 103 may preferentially record feature points that satisfy a specified condition among the extracted feature points. For example, among multiple consecutive surrounding images in time series, feature points extracted from an image in which the vehicle 1 traveled a longer distance during image capture may be preferentially selected as feature points.

The route recording unit 104 records the driving route 80 of the vehicle 1 during the teacher driving. The route recording unit 104 estimates the position of the vehicle 1 during the teacher driving based on, for example, changes in feature points extracted from multiple surrounding images and vehicle information of the vehicle 1, and identifies the driving route 80 from the time-series changes in the position. The recording process of the driving route 80 is an example of a route recording step.

More specifically, the route recording unit 104 identifies changes in the position of the vehicle 1 based on time-series changes in feature points extracted from multiple surrounding images captured during teacher driving. The route recording unit 104 may also correct the position of the vehicle 1 identified from the feature points based on the acquired vehicle information.

In this embodiment, the information that associates the driving route 80 for automatic driving, the speed of the vehicle 1 traveling on the driving route 80, the steering angle, the braking operation, and feature points extracted from multiple surrounding images captured as the vehicle 1 moves during teacher driving in a chronological order is called driving route information. The route recording unit 104 stores the driving route information in the memory unit 111. The driving route information is used by the vehicle control unit 110 during automatic driving, which will be described later. Note that the recording method of the driving route 80 and the definition of the driving route information are not limited to this example.

The route recording unit 104 also defines the environment around the vehicle 1 as a map based on feature points extracted from the surrounding images captured during teacher driving, and stores the map in the memory unit 111. The process of recording the driving route 80 based on teacher driving may be called a learning process.

The route image generating unit 105 generates a driving route image representing the driving route 80 recorded by the route recording unit 104.

Fig. 6 is a diagram showing an example of a driving route image 801 according to the first embodiment. The driving route image 801 is an image that shows a line from the start position to the end position of the driving route 80, as shown in Fig. 6, for example. In the driving route image 801, the driving route 80 is shown based on the movement trajectory of the center of the body 12 of the vehicle 1, for example. In the example shown in Fig. 6, the tip of the line that is the driving route image 801 is an arrow indicating the traveling direction of the vehicle 1. Note that the form of the driving route image 801 is not limited to this.

The route image generating unit 105 also generates an outline image that represents the movement trajectory of the outline of the vehicle body 12 of the vehicle 1 during teacher driving.

Fig. 7 is a diagram showing an example of an outer contour image 820 according to the first embodiment. Note that Fig. 7 also shows a driving route image 801 together with the outer contour image 820. The outer contour image 820 represents, for example, the movement trajectory of the outer contour of the vehicle body 12 of the vehicle 1 when the vehicle 1 travels along a driving route 80, as shown in Fig. 7. The route image generating unit 105 estimates the movement trajectory of the outer contour of the vehicle body 12, for example, based on the recorded driving route 80 and the size of the vehicle body 12. Note that the size of the vehicle body 12 is assumed to be stored in the memory unit 111, for example.

In this embodiment, the contour image 820 includes at least a vehicle width image 821 that represents the movement trajectory of both the left and right ends of the body 12 of the vehicle 1. In addition, if the travel route 80 includes a turnaround, the contour image includes an end image 822 that represents the position of the front end or rear end of the body 12 of the vehicle 1 at the turnaround point.

Returning to FIG. 5, the extended overhead image generating unit 106 generates an extended overhead image based on the surrounding image, which is an overhead view of the display target range including the driving route 80 and the surroundings of the driving route 80. The display target range is, for example, a range including the start position 900 to the end position of the driving route 80 in the teacher's run. The extended overhead image is an example of a first extended overhead image in this embodiment. This process is an example of a first extended overhead image generating step.

Figure 8 is a diagram showing an example of an overhead image 61 according to the first embodiment. The extended overhead image generating unit 106 converts multiple surrounding images captured in time series by the multiple imaging devices 16a to 16c at multiple positions on the travel route 80 into images viewed from a virtual viewpoint and synthesizes them to generate multiple overhead images 61 of the surroundings of the vehicle 1 for each of multiple positions on the travel route 80. Such image processing of viewpoint conversion and synthesis is an example of an overhead image generating step.

The virtual viewpoint is, for example, located directly above the vehicle 1. For this reason, the overhead image is also called a top view. Publicly known technology can be used to generate the overhead image 61.

The overhead image 61 shown in FIG. 8 is a composite image generated based on multiple surrounding images captured by the multiple imaging devices 16a to 16c when the vehicle 1 is located in the parking space 910, which is the end point of the travel route 80. Since the imaging ranges of the multiple imaging devices 16a to 16c change as the vehicle 1 moves, the overhead image 61 also differs when the vehicle 1 is in a different position on the travel route 80.

The extended overhead image generating unit 106 then synthesizes the multiple overhead images 61 in accordance with the time series changes in the capture times of the surrounding images corresponding to each overhead image 61, thereby generating an image depicting the surroundings of the vehicle 1 along the entire travel route 80. Since such an image depicts a wider range than each individual overhead image 61, it is referred to as an extended overhead image in this embodiment. Note that the extended overhead image may also be simply referred to as an overhead image.

FIG. 9 is a diagram showing an example of an extended overhead image 60 according to the first embodiment. The extended overhead image generating unit 106 generates one extended overhead image 60 by stitching together a plurality of overhead images 61, for example, based on feature points contained in the plurality of overhead images 61. The extended overhead image 60 includes the road surface or ground on which the vehicle 1 travels from the start position 900 of the travel route 80 to the parking space 910, which is the end position, and a background depicting the periphery of the travel route 80.

8 and 9, the extended overhead image generating unit 106 has been described as generating multiple overhead images 61 from multiple peripheral images, and then generating the extended overhead image 60 from the multiple overhead images 61, but the flow of the generation process is not limited to this. For example, the extended overhead image generating unit 106 may generate the extended overhead image 60 from multiple peripheral images.

Returning to FIG. 5, the display control unit 107 superimposes the driving route image 801 on the extended overhead image 60 and displays it on the display device 120. Hereinafter, the extended overhead image 60 on which the driving route image 801 is superimposed is referred to as an overhead route image. This display process is an example of the first display step in this embodiment.

Figure 10 is a diagram showing an example of an overhead route image 62 according to the first embodiment. In the overhead route image 62 shown in Figure 10, a driving route image 801 and an outer boundary image 820 are superimposed on the extended overhead image 60. Note that the outer boundary image 820 is not essential, and it is sufficient that at least the driving route image 801 is superimposed on the extended overhead image 60.

In the example shown in FIG. 10, the display control unit 107 displays a rectangular background region 602 as the background of the extended overhead image 60. Note that the extended overhead image 60 may also include the background region 602. Note that, although the background region 602 is displayed in black in FIG. 10, this is merely an example, and the display form is not particularly limited.

The bird's-eye view route image 62 is displayed, for example, when the user performs an operation to end the teacher driving. When the reception unit 101 receives an operation to end the recording of the teacher driving, the display control unit 107 causes the display device 120 to display the bird's-eye view route image 62.

For example, the bird's-eye view route image 62 depicts the road 40 and obstacles around the driving route 80, such as trees 73a, 73b, other houses 71a, other parking spaces 711a, 711c, and other vehicles 30a, together with the driving route image 801 and the outline image 820. This allows the user to confirm the positional relationship between the vehicle 1 and the surrounding obstacles and roads when driving as a teacher.

The display control unit 107 also causes the display device 120 to display the overhead route image 62 at an angle such that the forward direction of the vehicle 1 at the end of the teacher run faces the vertical direction of the display device 120. In the example shown in FIG. 10, the vertical direction of the display device 120 is the Y direction, and the horizontal direction is the X direction. The vertical direction is also referred to as the upper side of the screen. Note that the forward direction of the vehicle 1 means the direction in which the front end of the vehicle 1 faces.

As shown in FIG. 4, in this embodiment, when the vehicle 1 is traveling as a teacher, it moves forward and then reverses to enter the parking space 910. Therefore, at the end of the teacher traveling, the front of the vehicle 1 faces the entrance side of the parking space 910, and the driver also faces the entrance side of the parking space 910. In other words, the display control unit 107 displays the overhead route image 62 on the display device 120 so that the front direction of the vehicle 1 faces the Y direction at the end of the teacher traveling, so that the direction in which the driver actually views the scenery outside the vehicle 1 at the end of the teacher traveling matches the direction of the overhead route image 62 displayed on the display device 120. Therefore, the driver can easily grasp the relationship between the traveling route 80 and the actual surrounding environment.

The display control unit 107 also causes the display device 120 to display a registration button 141a and a redo button 141b along with the overhead route image 62. The registration button 141a is an image button that accepts a user operation to instruct the user to register the driving route 80 corresponding to the overhead route image 62. The redo button 141b is an image button that accepts a user operation to instruct the user not to register the driving route 80 corresponding to the overhead route image 62. When the registration button 141a and the redo button 141b are pressed, the reception unit 101 receives the operation corresponding to each button. Note that the method of receiving the operation to register or delete the driving route 80 is not limited to this.

Returning to FIG. 5, the registration processing unit 108 registers the driving route 80 generated by the route recording unit 104 in the storage unit 111. In this embodiment, the registration processing unit 108 registers only the driving route 80 selected by the user as the driving route 80 for automatic driving. For example, when the user presses the registration button 141a displayed together with the overhead route image 62, the driving route 80 corresponding to the overhead route image 62 is registered as the driving route 80 for automatic driving. Also, when the user presses the redo button 141b displayed together with the overhead route image 62, the driving route 80 corresponding to the overhead route image 62 is deleted.

The estimation unit 109 estimates the position and orientation of the vehicle 1 based on the surrounding image when the vehicle 1 is traveling automatically based on the travel route 80 by the vehicle control unit 110 described below.

For example, the estimation unit 109 estimates the position of the vehicle 1 by comparing feature points of a surrounding image captured during teacher driving with feature points of a current surrounding image. Note that the method by which the estimation unit 109 estimates the position and orientation of the vehicle 1 is not limited to this example.

The vehicle control unit 110 moves the vehicle 1 to the parking space 910 by automatic driving based on the driving route 80. The vehicle control unit 110 controls the steering, braking, and acceleration/deceleration of the vehicle 1 to cause the vehicle 1 to automatically drive along the registered driving route 80. This type of driving control method is also called reproducing the driving route 80. The automatic driving process by the vehicle control unit 110 is an example of a driving control step.

In addition, during autonomous driving, the vehicle control unit 110 may cause the vehicle 1 to perform operations different from teacher driving, based on the vehicle information of the vehicle 1 and the environmental information acquired by the acquisition unit 102. As described above, the environmental information includes the presence or absence of objects around the vehicle 1, and the distance between the vehicle 1 and obstacles around the vehicle 1. For example, if an obstacle such as another vehicle approaches during autonomous driving, the vehicle control unit 110 executes control such as stopping the vehicle 1.

Note that while the vehicle control unit 110 is driving autonomously, the driver may be seated in the driver's seat 130a of the vehicle 1 or may be out of the vehicle 1.

In addition, when the vehicle 1 deviates from the driving route 80 during autonomous driving, the vehicle control unit 110 may use feedback control to move the vehicle 1 back to the driving route 80. For example, the vehicle control unit 110 identifies a difference between the position of the vehicle 1 estimated by the estimation unit 109 and the driving route 80, and drives the vehicle 1 so as to reduce the difference.

In addition, while FIG. 5 illustrates the parking assistance device 100 as including a vehicle control unit 110, the vehicle control unit 110 may be realized by an ECU other than the parking assistance device 100.

Next, we will explain the flow of the parking assistance process executed by the parking assistance device 100 of this embodiment configured as described above.

Figure 11 is a flowchart showing an example of the flow of the driving route recording process executed by the parking assistance device 100 according to the first embodiment.

First, the reception unit 101 determines whether or not a teacher running start operation by the user has been received (S101). If a teacher running start operation by the user has not been received (S101 "No"), the reception unit 101 waits while repeating the process of S101.

When the reception unit 101 receives an instruction to start teacher driving by the user (S101 "Yes"), the process of recording the teacher driving starts. After starting the teacher driving, the driver manually drives the vehicle 1 and moves it to the parking space 910.

Then, while the driver is driving the teacher, the acquisition unit 102 acquires surrounding images from the multiple image capture devices 16a to 16c. The acquisition unit 102 also acquires the speed, steering angle, and braking operation of the vehicle 1 from various sensors in the vehicle 1 or from other ECUs. The acquisition unit 102 also acquires the distance measurement results of the distance between the vehicle 1 and surrounding obstacles from the wave transmitting/receiving unit 15 (S102).

The extraction unit 103 extracts feature points from the multiple acquired surrounding images (S103).

Then, the route recording unit 104 estimates the position of the vehicle 1 during the teacher drive based on the changes in the feature points extracted from the multiple surrounding images, as well as the speed, steering angle, braking operation, and distance measurement results, and records the time-series changes in the position as the driving route 80 (S104).

Then, the reception unit 101 judges whether or not an operation to end the teacher driving has been received (S105). If an operation to end the teacher driving by the user has not been received (S105 "No"), the reception unit 101 repeats the processes of S102 to S105.

When the reception unit 101 receives an operation to end the teacher driving by the user (S105 "Yes"), the route image generation unit 105 generates a driving route image 801 based on the driving route 80 recorded by the route recording unit 104 (S106).

The route image generating unit 105 also generates an outline image 820 that represents the movement trajectory of the outline of the vehicle body 12 of the vehicle 1 during the teacher run (S107).

The extended overhead image generating unit 106 also generates an extended overhead image 60 based on multiple surrounding images captured in time series by multiple imaging devices 16a to 16c (S108).

Next, the display control unit 107 causes the display device 120 to display an overhead route image 62 in which the driving route image 801 and the contour image 820 are superimposed on the extended overhead image 60 (S109). In addition, the display control unit 107 causes the display device 120 to display a registration button 141a and a redo button 141b, as shown in FIG. 10.

Then, when the reception unit 101 receives an operation to redo the teacher driving by the user (S110 "Redo"), the process returns to S101. Also, when the reception unit 101 receives an operation to register the teacher driving by the user (S110 "Register"), the registration processing unit 108 registers the driving route 80 in the memory unit 111 as a driving route 80 for automatic driving (S111). Here, the process of this flowchart ends.

In this manner, in the parking assistance method executed by the parking assistance device 100 of this embodiment, captured images of the surroundings of the vehicle 1 during teacher driving are acquired from the imaging device 16 provided on the vehicle 1 in time series with the movement of the vehicle 1, and the driving route 80 of the vehicle 1 during teacher driving is recorded. In this parking assistance method, an extended overhead image 60 is generated based on the captured images, overlooking the display target range including the driving route 80 and the surroundings of the driving route 80 from above, and the generated extended overhead image 60 is displayed on the display device 120. Therefore, according to the parking assistance method and parking assistance device 100 of this embodiment, the user can easily grasp the positional relationship between the driving route 80 of the vehicle 1 based on teacher driving and the objects and road 40 around the vehicle 1. In addition, in the parking assistance device 100 of this embodiment, the display device 120 displays an overhead route image 62 in which a driving route image 801 is superimposed on an extended overhead image 60, making it easier for the user to understand the positional relationship between the driving route 80 of the vehicle 1 based on the teacher driving and the objects and road 40 around the vehicle 1.

As a comparative example, in a normal overhead image, the current state of the surroundings of the vehicle is depicted from an upward perspective, but the range covering the entire driving route and the area extending to the surroundings are not depicted. For this reason, while it is suitable for the driver to check in real time, it may not be sufficient to check the entire driving route. In contrast, in the parking assistance method executed by the parking assistance device 100 of this embodiment, the user can objectively grasp the entire driving route 80 during teacher driving and the state of the surroundings.

In addition, in the parking assistance method executed by the parking assistance device 100 of this embodiment, an outer contour image 820 showing the movement trajectory of the outer contour of the body 12 of the vehicle 1 during teacher driving is displayed superimposed on the extended overhead image 60. Therefore, according to the parking assistance method executed by the parking assistance device 100 of this embodiment, the driver can easily grasp the positional relationship and distance between the outer contour of the body 12 of the vehicle 1 and surrounding obstacles, etc. For example, even if the vehicle 1 reaches the parking space during teacher driving without contacting surrounding obstacles, the outer contour of the body 12 may be close to the surrounding obstacles, etc. The user can check the distance between the outer contour of the body 12 and the surrounding obstacles, etc. from an upper viewpoint by visually checking the outer contour image 820 on the extended overhead image 60.

For example, in the example shown in FIG. 10, the contour image 820 of the vehicle 1 during teacher driving is also depicted in the area corresponding to the corner space 722 shown in FIG. 4. Therefore, it is clear that the body 12 of the vehicle 1 passes through the corner space 722, even though the center of the vehicle 1 does not pass through the corner space 722 during teacher driving. For example, it is assumed that there is nothing placed in the corner space 722 during teacher driving, and the corner space 722 may be used as a bicycle parking area. In this case, the user can avoid using the driving route 80 that passes through the corner space 722 for automatic driving by checking the movement trajectory of the body 12 of the vehicle 1 in the contour image 820 on the overhead route image 62.

In addition, in the parking assistance method executed by the parking assistance device 100 of this embodiment, when an operation to start and end the recording of a teacher's run is accepted by the user, an overhead route image 62 is displayed. Therefore, according to the parking assistance method executed by the parking assistance device 100 of this embodiment, after the teacher's run is completed, the user can immediately check the positional relationship between the vehicle 1 and surrounding obstacles and roads during the teacher's run from the overhead route image 62. Furthermore, by making such confirmation easier, it is possible to support the user's decision-making regarding whether to register a run route based on the teacher's run or to redo the teacher's run after the teacher's run is completed.

In addition, in the parking assistance method executed by the parking assistance device 100 of this embodiment, the bird's-eye view route image 62 is displayed on the display device 120 at an angle such that the forward direction of the vehicle 1 at the end of the teacher driving is facing upward on the screen of the display device 120. With this configuration, the direction in which the driver actually views the scenery outside the vehicle 1 at the end of the teacher driving matches the orientation of the bird's-eye view route image 62 displayed on the display device 120. This allows the driver to easily grasp the relationship between the driving route 80 and the actual surrounding environment.

Second Embodiment
In the first embodiment described above, the parking assistance device 100 superimposes the driving path image 801 and the contour image 820 on the extended overhead image 60. In the second embodiment, the parking assistance device 100 further changes the display mode of the contour image 820 according to the distance to the obstacles around the vehicle 1.

The vehicle 1 of this embodiment has a configuration similar to that of the first embodiment described in Figures 1 and 2. In addition, the hardware configuration of the parking assistance device 100 of this embodiment is similar to that of the first embodiment described in Figure 3.

The parking assistance device 100 of this embodiment includes a reception unit 101, an acquisition unit 102, an extraction unit 103, a route recording unit 104, a route image generating unit 105, an extended overhead image generating unit 106, a display control unit 107, a registration processing unit 108, an estimation unit 109, a vehicle control unit 110, and a storage unit 111, similar to the first embodiment. The reception unit 101, the acquisition unit 102, the extraction unit 103, the route recording unit 104, the extended overhead image generating unit 106, the display control unit 107, the registration processing unit 108, the estimation unit 109, the vehicle control unit 110, and the storage unit 111 have the same functions as the first embodiment.

The route image generating unit 105 of this embodiment has the same functions as the first embodiment, and generates the outline image of the areas where the distance between the vehicle 1 and objects around the vehicle 1 during teacher driving is less than a threshold in a manner different from the areas where the distance is greater than the threshold. The threshold value for the distance between the vehicle 1 and objects around the vehicle 1 is not particularly limited, but may be, for example, 50 cm. The threshold value may be a fixed value set in advance, or may be changeable by the user.

Figure 12 is a diagram showing an example of an overhead route image 62a according to the second embodiment. In the example shown in Figure 12, at the turning point of the vehicle 1 on the travel route 80, the distance between the vehicle 1 and surrounding obstacles is equal to or less than a threshold value.

The route image generating unit 105 generates an outline image 820a by, for example, using a solid line where the distance between the vehicle 1 and objects around the vehicle 1 is greater than a threshold, and using a double line where the distance between the vehicle 1 and objects around the vehicle 1 is equal to or less than the threshold. In the example shown in FIG. 12, the route image generating unit 105 uses a double line for the end image 822a corresponding to the area where the distance between the vehicle 1 and surrounding obstacles is equal to or less than the threshold, and a solid line for the vehicle width image 821 corresponding to the area where the distance is greater than the threshold.

The display control unit 107 of this embodiment has the same functions as the first embodiment, and displays the end image 822a and the vehicle width image 821 generated by the route image generation unit 105 in a superimposed manner on the extended overhead image 60, thereby displaying the parts of the contour image 820 where the distance to surrounding obstacles is less than a threshold value in a manner different from the parts where the distance is greater than the threshold value.

The display mode is not limited to the example shown in FIG. 12, and the route image generating unit 105 may change the color of the areas where the distance between the vehicle 1 and objects around the vehicle 1 is less than a threshold value to a different color from other areas, or may add a warning mark or message.

In this way, in the parking assistance method executed by the parking assistance device 100 of this embodiment, the portions of the contour image 820 where the distance to surrounding obstacles is less than a threshold value are displayed in a different manner from the portions where the distance is greater than the threshold value. Therefore, according to the parking assistance method executed by the parking assistance device 100 of this embodiment, by allowing the user to understand the portions where the distance between the vehicle 1 and surrounding obstacles is close during teacher driving, it is possible to support the user's decision-making regarding whether to register a driving route based on the teacher driving after the teacher driving ends, or to redo the teacher driving.

Third Embodiment
In the third embodiment, the parking assistance device 100 further displays a vehicle image representing the vehicle 1 on the overhead route image.

The vehicle 1 of this embodiment has a configuration similar to that of the first embodiment described in Figures 1 and 2. In addition, the hardware configuration of the parking assistance device 100 of this embodiment is similar to that of the first embodiment described in Figure 3.

The parking assistance device 100 of this embodiment includes a reception unit 101, an acquisition unit 102, an extraction unit 103, a route recording unit 104, a route image generating unit 105, an extended overhead image generating unit 106, a display control unit 107, a registration processing unit 108, an estimation unit 109, a vehicle control unit 110, and a storage unit 111, similar to the first embodiment. The reception unit 101, the acquisition unit 102, the extraction unit 103, the route recording unit 104, the route image generating unit 105, the extended overhead image generating unit 106, the registration processing unit 108, the estimation unit 109, the vehicle control unit 110, and the storage unit 111 have the same functions as the first embodiment.

The display control unit 107 of this embodiment has the same functions as the first embodiment, and displays a vehicle image representing the vehicle 1 at the position of the vehicle 1 at the end of the teacher run on the overhead route image. The vehicle image is, for example, a CG (Computer Graphics) image that imitates the shape of the vehicle 1. The vehicle image is, for example, stored in advance in the storage unit 111.

Figure 13 shows an example of an overhead route image 62b according to the third embodiment.

As shown in FIG. 13, the display control unit 107 of this embodiment superimposes a vehicle image 83 on the extended overhead image 60 at a position corresponding to the parking space 910. The orientation of the vehicle image 83 is the orientation of the vehicle 1 at the end of the teacher run.

In this embodiment, the position of vehicle 1 at the end of the teacher driving is parking space 910. For example, when the driver checks the overhead route image 62b while remaining in vehicle 1 after the teacher driving ends, the position of vehicle 1 in which the driver is riding is displayed on the overhead route image 62b, so that the driver can easily grasp the positional relationship between the position of vehicle 1 and the driving route 80.

(Fourth embodiment)
In the first embodiment described above, the parking assistance device 100 accepted a user's operation to register or delete the driving route 80 obtained in each teacher driving session. In this fourth embodiment, after the completion of multiple teacher driving sessions, the parking assistance device 100 registers the driving route 80 selected by the user from among the multiple driving routes 80 obtained in the multiple teacher driving sessions.

The vehicle 1 of this embodiment has a configuration similar to that of the first embodiment described in Figures 1 and 2. In addition, the hardware configuration of the parking assistance device 100 of this embodiment is similar to that of the first embodiment described in Figure 3.

The parking assistance device 100 of this embodiment includes a reception unit 101, an acquisition unit 102, an extraction unit 103, a route recording unit 104, a route image generating unit 105, an extended overhead image generating unit 106, a display control unit 107, a registration processing unit 108, an estimation unit 109, a vehicle control unit 110, and a storage unit 111, similar to the first embodiment. The acquisition unit 102, the extraction unit 103, the route recording unit 104, the route image generating unit 105, the extended overhead image generating unit 106, the estimation unit 109, the vehicle control unit 110, and the storage unit 111 have the same functions as the first embodiment.

The display control unit 107 of this embodiment has the same functions as the first embodiment, and after performing multiple teacher runs, causes the display device 120 to display multiple overhead route images based on each of the multiple teacher runs. This display process is an example of a second display step in this embodiment.

Figure 14 is a diagram showing an example of a driving route selection screen 121 according to the fourth embodiment. The display control unit 107 causes the display device 120 to display the driving route selection screen 121 including a plurality of overhead route images 62c to 62f. Each of the overhead route images 62c to 62f corresponds to the first to fourth teacher runs, respectively. Note that the number of teacher runs and overhead route images 62 shown in Figure 14 is an example and is not limited to four runs.

Each of the overhead route images 62c to 62f is an image in which a driving route image 801c to 801f and an outer boundary image 820c to 820f are superimposed on the extended overhead image 60c to 60f. The outer boundary images 820c to 820f include vehicle width images 821c to 820f and end images 822c to 822f.

The display control unit 107 also displays selection buttons 143a to 143d, which allow the user to select each of the overhead route images 62c to 62f, and a registration button 142a on the driving route selection screen 121. Each of the selection buttons 143a to 143d corresponds one-to-one with each of the overhead route images 62c to 62f.

The reception unit 101 of this embodiment has the same functions as the first embodiment, and receives a user operation to select one overhead route image 62 corresponding to the driving route 80 to be registered from among the multiple overhead route images 62c to 62f. This reception process is an example of a selection reception step in this embodiment.

For example, the user presses one of the selection buttons 143a to 143d, and then presses the registration button 142a to select one overhead route image 62 that corresponds to the driving route 80 to be registered. In the example shown in FIG. 14, the user selects the selection button 143c that corresponds to the third teacher-guided driving. When the user presses the registration button 142a in this state, the reception unit 101 receives the user's operation to register the driving route 80 obtained in the third teacher-guided driving as a driving route 80 for automatic driving.

The registration processing unit 108 of this embodiment has the same functions as the first embodiment, and registers the driving route 80 selected by the user in the memory unit 111 as a driving route 80 for automatic driving.

Figure 15 is a flowchart showing an example of the flow of a driving route recording process executed by the parking assistance device 100 according to the fourth embodiment.

The process from determining whether or not a teacher run start operation has been accepted in S101 to displaying an overhead route image 62 for each teacher run in S109 is the same as in the first embodiment.

Then, if the reception unit 101 receives an operation by the user to start the teacher driving again after S109 (S201 "Start teacher driving again"), the process returns to S101. In this case, the driver also causes the vehicle 1 to leave the parking space 910 and move the vehicle 1 to the starting position 900.

In addition, when the reception unit 101 receives an operation by the user to start selecting a driving route 80 (S201 "Start selecting a driving route"), the display control unit 107 causes the display device 120 to display a driving route selection screen 121 (S202).

Then, the reception unit 101 determines whether or not an operation by the user to select the driving route 80 to be registered has been received (S203). If the reception unit 101 has not received an operation by the user to select the driving route 80 to be registered (S203 "No"), the display control unit 107 continues to display the driving route selection screen 121.

When the reception unit 101 receives an operation by the user to select a driving route 80 to be registered, the registration processing unit 108 registers the driving route 80 selected by the user in the storage unit 111 as a driving route 80 for automatic driving (S204). At this point, the processing of this flowchart ends.

In this way, in the parking assistance method executed by the parking assistance device 100 of this embodiment, after multiple teacher runs are performed, multiple overhead route images 62c to 62f based on each of the multiple teacher runs are displayed on the display device 120, and a user operation is accepted to select the overhead route image 62 corresponding to the driving route 80 to be registered from the multiple overhead route images 62c to 62f. Therefore, according to the parking assistance method executed by the parking assistance device 100 of this embodiment, the user can compare the multiple driving routes 80 obtained from multiple teacher runs and select the desired driving route 80 to be registered.

Fifth Embodiment
In the above-mentioned first to fourth embodiments, the display of the overhead route image 62, 62a to 62f after teacher driving has been described, but in this fifth embodiment, the display of the overhead route image during automatic driving will be described.

The vehicle 1 of this embodiment has a configuration similar to that of the first embodiment described in Figures 1 and 2. In addition, the hardware configuration of the parking assistance device 100 of this embodiment is similar to that of the first embodiment described in Figure 3.

The parking assistance device 100 of this embodiment includes, as in the first embodiment, a reception unit 101, an acquisition unit 102, an extraction unit 103, a route recording unit 104, a route image generation unit 105, an extended overhead image generation unit 106, a display control unit 107, a registration processing unit 108, an estimation unit 109, a vehicle control unit 110, and a storage unit 111. The reception unit 101, the route recording unit 104, the route image generation unit 105, the registration processing unit 108, the estimation unit 109, the vehicle control unit 110, and the storage unit 111 have the same functions as in the first embodiment.

The acquisition unit 102 of this embodiment has the same functions as the first embodiment, and acquires captured images of the surroundings of the vehicle 1 during autonomous driving in time series as the vehicle 1 moves, that is, surrounding images, from multiple image capture devices 16a to 16c. This acquisition process during autonomous driving is an example of a second acquisition step in this embodiment.

The extraction unit 103 of this embodiment has the same functions as the first embodiment, and extracts feature points from surrounding images acquired during autonomous driving.

The extended overhead image generating unit 106 of this embodiment has the same functions as the first embodiment, and generates an overhead image 61 from surrounding images acquired during autonomous driving.

The extended overhead image generating unit 106 also updates the extended overhead image 60 as the vehicle 1 travels during autonomous driving by synthesizing an overhead image 61 based on a surrounding image captured during autonomous driving with the extended overhead image 60 during teacher driving stored in the memory unit 111. This synthesis process is an example of a synthesis step.

For example, the extended overhead image generating unit 106 identifies an image range of the extended overhead image 60 during teacher driving that corresponds to the overhead image 61 based on the surrounding image captured during automatic driving based on the feature points extracted by the extraction unit 103, and replaces the identified image range with the overhead image 61. By repeating such processing, the range of the extended overhead image 60 during teacher driving that was captured by the imaging device 16 along the progress of the vehicle 1 is sequentially replaced. Note that the method of synthesizing the extended overhead image 60 during teacher driving and the overhead image 61 based on the surrounding image captured during automatic driving is not limited to this.

In this embodiment, the extended overhead image 60 during teacher driving is called the first extended overhead image 60, and the image obtained by combining the first extended overhead image 60 with an overhead image 61 based on a surrounding image captured during automatic driving is called the second extended overhead image to distinguish between the two.

The display control unit 107 of this embodiment has the same functions as the first embodiment, and displays the second extended overhead image generated by the extended overhead image generating unit 106 during autonomous driving on the display device 120. This display process is an example of a third display step.

Figure 16 is a diagram showing an example of an overhead route image 62g according to the fifth embodiment. In the second extended overhead image 63 shown in Figure 16, the range in which a peripheral image has been captured by the imaging device 16 of the vehicle 1 located near the starting position 900 of the travel route 80 has been replaced with the overhead image 61 generated during automatic driving. In addition, in the second extended overhead image 63, the range in which a peripheral image has not yet been captured by the imaging device 16 is still displayed as the first extended overhead image 60.

When the vehicle 1 is being driven automatically by the vehicle control unit 110, the display control unit 107 displays an overhead route image 62g in which the driving route image 802 and the vehicle image 83 are superimposed on the second extended overhead image 63, as shown in FIG. 16.

More specifically, the display control unit 107 displays the vehicle image 83 at a location on the second extended overhead image 63 that corresponds to the current position of the vehicle 1. The display control unit 107 moves the vehicle image 83 on the second extended overhead image 63 in accordance with the movement of the vehicle 1 due to autonomous driving.

In the example shown in FIG. 16, the display control unit 107 displays in the driving route image 802 the area where the vehicle 1 has already traveled with a solid line, and the area where the vehicle 1 has not yet traveled with a dashed line. This display allows the user to know where the vehicle 1 is located on the driving route 80.

The display mode of the driving route image 802 shown in FIG. 16 is an example, and the display mode does not need to be changed depending on whether the vehicle 1 is traveling or not. In addition, the display control unit 107 does not need to display the driving route image 802 during automatic traveling.

In the example shown in FIG. 16, the display control unit 107 displays the overhead route image 62g so that the current forward direction of the vehicle 1 faces the Y direction of the display device 120. In this case, when the direction of the vehicle 1 changes, the orientation of the overhead route image 62g also changes.

Note that this display direction is an example and may be changeable. For example, the display control unit 107 may have a first display mode in which the overhead route image 62g is displayed so that the current forward direction of the vehicle 1 faces the Y direction of the display device 120, and a second display mode in which the overhead route image 62g is displayed so that the forward direction of the vehicle 1 at the end point of the driving route 80 faces the Y direction of the display device 120. In the second display mode, the orientation of the overhead route image 62g is fixed regardless of changes in the orientation of the vehicle 1 during automatic driving.

For example, the reception unit 101 may receive an operation by the user to select the first display mode or the second display mode. The display control unit 107 displays the overhead route image 62g in the display mode selected by the user.

In this manner, in the parking assistance method executed by the parking assistance device 100 of this embodiment, the second extended overhead image 63 is generated by combining the overhead image 61 generated from the surrounding image captured during the automatic driving of the vehicle 1 based on the driving route 80 with the extended overhead image 60 during the teacher driving. Then, in this parking assistance method, the generated second extended overhead image 63 is displayed on the display device 120. Therefore, according to the parking assistance method executed by the parking assistance device 100 of this embodiment, the second extended overhead image 63 reflecting the current state of the surroundings of the vehicle 1 can be presented to the driver during the automatic driving of the vehicle 1 based on the driving route 80.

For example, if an obstacle that was not present during teacher driving is present around the vehicle 1, the parking assistance method executed by the parking assistance device 100 of this embodiment can depict the obstacle on the second extended overhead image 63.

(Variation 1)
In each of the above-described embodiments, the display device 120 and the operation unit are mounted on the vehicle 1, but may be provided outside the vehicle 1. For example, the display device 120 and the operation unit may be a mobile terminal such as a tablet terminal or a smartphone capable of wireless or wired communication with the parking assistance device 100. Note that it is sufficient that the tablet terminal or the like is not directly connected to the parking assistance device 100 of the vehicle 1 and data and control signals can be transmitted and received via a server device or a cloud environment.

If the display device 120 and the operation unit are a tablet terminal or the like, the driver may use the tablet terminal or the like within the vehicle 1, or may use the tablet terminal or the like after getting out of the vehicle 1.

Alternatively, if the vehicle 1 is equipped with a head-up display, the head-up display may be an example of a display unit. For example, the windshield 180 of the vehicle 1 may become a head-up display capable of displaying an image by projecting an image onto it using a projection device (not shown).

(Variation 2)
Further, the display timing at which the display control unit 107 of the parking assistance device 100 displays the overhead route images 62, 62a to 62g is not limited to the examples given in the above-mentioned embodiments.

For example, the parking assistance device 100 of this modified example has a function of accepting a user's operation to modify the driving route 80. In this case, after the teacher driving is completed, when the reception unit 101 accepts an operation by the user to instruct the display of a modification screen of the driving route 80, the display control unit 107 causes the display device 120 to display the overhead route images 62, 62a to 62g.

The user's operation to instruct the display of the revision screen for the driving route 80 is, for example, an operation to modify the driving route on the overhead route image 62, 62a to 62g. The process of accepting this operation is an example of a revision operation acceptance step.

More specifically, the user operates the driving route image 801 included in the overhead route images 62, 62a to 62g with a finger or a mouse, etc., to input a correction operation for the driving route 80. When the parking assistance device 100 receives a correction operation for the driving route 80 by the user, it executes a recalculation based on the correction operation and corrects the driving route 80.

The user may instruct the display of the revision screen for the driving route 80 while the user is in the vehicle 1, or may perform the operation after the user has exited the vehicle 1, using a tablet terminal or the like as a display device and operation unit. In addition, the user may be able to perform revision operations not only for the driving route 80 immediately after it has been recorded during teacher driving, but also for driving routes 80 that have already been registered as driving routes for automatic driving.

(Variation 3)
In each of the above-described embodiments, the parking assistance device 100 uses the driving trajectory of the vehicle 1 during teacher driving as the driving route 80. However, the parking assistance device 100 may generate the driving route 80 by applying corrections to the speed and driving position of the driving trajectory of the vehicle 1 during teacher driving.

In addition, in each of the above-described embodiments, the teacher driving is performed by driving the vehicle 1 from the starting position 900 to the parking space 910 in the same manner as the vehicle 1 drives in the automatic driving mode, but the teacher driving may be performed in the opposite direction to the automatic driving mode.

More specifically, the parking space 910 may be the starting point of the teacher driving, and the starting position 900 may be the end point of the teacher driving. In this case, the route recording unit 104 generates the driving route 80 for automatic driving by converting the route of the teacher driving in the reverse direction. In other words, the vehicle control unit 110 drives the vehicle 1 from the starting position 900 to the parking space 910 in the opposite direction to the teacher driving. This method is also called replaying the driving route in reverse.

(Variation 4)
Further, some of the functions of the parking assistance device 100 in each of the above-described embodiments may be executed by an information processing device provided outside the vehicle 1. The information processing device provided outside the vehicle 1 is, for example, a mobile terminal such as a smartphone, a PC, a server device, etc. The information processing device may be provided in a cloud environment.

Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. These embodiments and their modifications are within the scope of the invention and its equivalents as set forth in the claims, as well as the scope and gist of the invention.

REFERENCE SIGNS LIST 1 vehicle 12 vehicle body 15, 15f, 15r wave transmitting/receiving unit 16, 16a to 16d imaging device 40 road 60, 60c to 60f extended overhead image (first extended overhead image)
61 Bird's-eye image 62, 62a to 62g Bird's-eye view route image 63 Second extended bird's-eye image 70 Home 80 Driving route 83 Vehicle image 100 Parking assistance device 101 Reception unit 102 Acquisition unit 103 Extraction unit 104 Route recording unit 105 Route image generation unit 106 Extended bird's-eye view image generation unit 107 Display control unit 108 Registration processing unit 109 Estimation unit 110 Vehicle control unit 111 Storage unit 120 Display device 121 Driving route selection screen 141 Operation button 141a Registration button 141b Redo button 143a to 143d Selection buttons 180 Windshield 801, 801c to 801f, 802 Driving route image 820, 820a, 820c to 820f Outer contour image 821, 821c to 820f Vehicle width image 822, 822a, 822c to 822f End image 900 Start position 910 Parking space

Claims (9)

A parking assistance method for automatically driving a vehicle based on a teacher driving by a driver, comprising:
a first acquisition step of acquiring, from an imaging device provided on the vehicle, images of the surroundings of the vehicle during the teacher running in time series with the movement of the vehicle;
a route recording step of recording a travel route of the vehicle during the teacher travel;
a first extended overhead image generating step of generating a first extended overhead image, which is an overhead view of a display target range including the travel route and a periphery of the travel route, based on the captured image;
a first display step of displaying the first extended overhead image on a display unit;
a start/end operation receiving step of receiving an operation by a user to start and end recording of the teacher running;
Including,
In the first display step, an overhead route image in which a travel route image showing the recorded travel route is superimposed on the first extended overhead image is displayed on the display unit;
When an operation to end the recording of the teacher driving is accepted, the display of the overhead view route image is executed by the first display step,
The bird's-eye view route image is displayed before an operation for registering the teacher's run is accepted ;
In the first acquisition step, information regarding a distance between the vehicle and objects around the vehicle during the teacher running is further acquired,
In the first display step, a contour image showing a movement trajectory of the contour of the vehicle body during the teacher driving is superimposed on the first extended overhead image and a portion of the contour image where the distance is equal to or less than a threshold is displayed on the overhead route image in a manner different from a portion where the distance is greater than the threshold.
Parking assistance methods. the display target range is a range including a start position to an end position of a travel route in the teacher travel,
The parking assistance method according to claim 1 . In the first display step, a vehicle image representing the vehicle is displayed on the overhead route image at a position of the vehicle at the end of the teacher running.
3. A parking assistance method according to claim 1 or 2 . In the first display step, the bird's-eye view route image is displayed on the display unit at an angle such that a forward direction of the vehicle at the time when the teacher running ends faces upward on a screen of the display unit.
The parking assistance method according to claim 1 . A correction operation receiving step of receiving a user's operation to correct the driving route on the overhead route image,
A parking assistance method according to any one of claims 1 to 4 . a second display step of displaying, on the display unit, a plurality of overhead route images based on each of the plurality of teacher-runs after the plurality of teacher-runs have been executed;
and a selection receiving step of receiving a user's operation of selecting an overhead route image corresponding to a travel route to be registered from among the plurality of overhead route images.
A parking assistance method according to any one of claims 1 to 5 . a driving control step of moving the vehicle to a parking position by automatic driving based on the driving route;
a second acquisition step of acquiring from the imaging device captured images of the surroundings of the vehicle during the autonomous driving in time series together with the movement of the vehicle;
an overhead image generating step of generating an overhead image from the captured image acquired in the second acquisition step;
a synthesis step of generating a second extended overhead image by synthesizing the overhead image with the first extended overhead image generated in the first extended overhead image generation step;
and a third display step of displaying the second extended overhead image on the display unit.
A parking assistance method according to any one of claims 1 to 6 . A parking assistance method for automatically driving a vehicle based on a teacher driving by a driver, comprising:
acquiring, from an imaging device provided on the vehicle, captured images of the surroundings of the vehicle during the teacher running in time series with the movement of the vehicle;
Recording a travel route of the vehicle during the teacher travel;
generating a first extended overhead image in which a display target range including the travel route and a periphery of the travel route is viewed from above;
Displaying the first extended overhead image on a display unit;
accepting an operation by a user to start and end recording of the teacher driving;
Including,
displaying, on the display unit, an overhead route image in which a travel route image showing the recorded travel route is superimposed on the first extended overhead image;
When an operation to end the recording of the teacher's driving is accepted, the overhead view route image is displayed;
The bird's-eye view route image is displayed before an operation for registering the teacher's run is accepted ;
acquiring information regarding a distance between the vehicle and objects around the vehicle during the teacher driving;
an outer contour image showing a movement trajectory of an outer contour of the vehicle body during the teacher driving is displayed superimposed on the first extended overhead image, and a portion of the outer contour image where the distance is equal to or less than a threshold is displayed on the overhead route image in a manner different from a portion where the distance is greater than the threshold;
Parking assistance methods. A parking assistance device that performs parking assistance for automatically driving a vehicle based on a teacher driving by a driver,
an acquisition unit that acquires images of the surroundings of the vehicle during the teacher running from an imaging device provided in the vehicle in time series with the movement of the vehicle;
a route recording unit that records a travel route of the vehicle during the teacher travel;
an extended overhead image generating unit that generates a first extended overhead image based on the captured image, the first extended overhead image being an overhead view of a display target range including the travel route and a periphery of the travel route;
a display control unit that causes the first extended overhead image to be displayed on a display unit;
A reception unit that receives an operation by a user to start and end recording of the teacher running;
Equipped with
the display control unit causes the display unit to display an overhead view route image in which a travel route image showing the recorded travel route is superimposed on the first extended overhead view image, when an operation to end recording of the teacher-running is accepted before an operation to register the teacher-running is accepted ,
The acquisition unit further acquires information regarding a distance between the vehicle and objects around the vehicle during the teacher traveling,
The display control unit further causes an outer contour image representing a movement trajectory of an outer contour of the vehicle body during the teacher driving to be superimposed on the first extended overhead image, and causes a portion of the outer contour image where the distance is equal to or less than the threshold to be displayed on the overhead route image in a manner different from a portion where the distance is greater than the threshold.
Parking assistance device.

Images