JP7460145B2 - Autonomous vehicle systems - Google Patents

Description

The present invention relates to a system for use in autonomous vehicles.

With the spread of car navigation systems as typified by Patent Document 1, a major change has come in the way of driving a vehicle. Since then, autonomous vehicle technology has made remarkable progress in recent years, and the practical application of level 3 autonomous driving is just around the corner. Level 3 (conditional automated driving) refers to a system in which the system performs all driving tasks, but requires the driver to respond appropriately to system intervention requests (non-patent (See Reference 1). Level 3 self-driving vehicles are now permitted to operate in Japan under the revised Road Traffic Act, which came into effect on April 1, 2020.

However, at Level 3, the problem is that human drivers do not know when they will be asked to intervene in the driving task. During Level 3 autonomous driving, the driver has nothing to do, so to speak, and is left with nothing to do. Even if a situation that could lead to an accident occurs and the driver is required to drive immediately, it takes a certain amount of time for the human driver's brain to follow suit.

In other words, although the situation where automatic driving returns to manual driving is generally considered to be an urgent matter (exceeding the ability of automatic driving machines), There is an inherent problem in that a task other than driving cannot be transferred to a driving task (taking over driving) in an emergency.

These problems with Level 3 are referred to in the technical field as problems related to the "human factor," and various solutions are being explored. For example, methods have been proposed such as having the autonomous driving system give a warning to the human driver when taking over driving, having the driver perform a visually stimulating subtask such as a number guessing game while autonomous driving, and blowing cool air on the driver if the system determines that the driver is drowsy while autonomous driving (see Non-Patent Document 2).

Japanese Patent Application Laid-Open No. 01-263688

Regarding the classification of levels of autonomous driving, Ministry of Land, Infrastructure, Transport and Tourism, March 19, 2018 http://www.mlit.go.jp/common/001226541.pdf Presentation materials at the SIP Automated Driving System Human Factor Research and Development Results Symposium, Bellesalle Onarimon Tower 3F Hall, held/released on July 29, 2019, sponsored by the National Institute of Advanced Industrial Science and Technology https://seminar.tokyotobs- entry.jp/Symposium/ https://web.archive.org/web/20200818144741/https://seminar.tokyotobs-entry.jp/Symposium/

However, for the average human driver, being forced to intervene in automated driving using the conventional methods described above is unpleasant. Being placed in the unnatural state of being in the driver's seat for a long time even though you don't drive, and then suddenly being asked to return to the driving task due to circumstances unrelated to your own actions... This can be very stressful for human drivers.

Furthermore, events such as being forced to perform subtasks like a number guessing game or being hit with cold air are purely for the convenience of the autonomous driving system and do not take into account the psychological discomfort felt by human drivers.

Although resolving the above human factor issues is essential to popularizing Level 3 autonomous driving, the reality is that no effective solutions have yet been found.

In view of the actual situation in the technical field, the present inventor proposes a means for solving the human factor problem. That is, the embodiment of the present invention can provide the following aspects.

Aspect 1
1. A system for use in automated driving of a vehicle, comprising:
a storage means configured to store map image data associated with map coordinates and associated data associated with said map coordinates and not related to a driving task;
positioning means for determining a current vehicle position and status in said map coordinates;
an information presentation means for presenting information to a driver of the vehicle;
and a means for measuring reliability of the autonomous driving of the vehicle;
The system is characterized in that the information presentation means is configured to present to the driver information in which the map image data related to the current position of the vehicle in the map coordinates is superimposed with the accompanying data related to the current position of the vehicle in the map coordinates when the operation of the vehicle is determined to be in a state where no operation of the driver is required based on the measured reliability.

Aspect 2
Further, an imaging means for capturing an image of the scenery outside the vehicle;
The system according to aspect 1, further comprising: an editing unit that creates the associated data based on an image and/or a video of a landscape captured by the imaging unit.

Aspect 3
The accompanying data is associated with the map coordinates, including natural topography and geological data, past historical map image data, engineering technology data, data on historical figures or culture, data on literature and folklore, tourist information data, and data on habitats. 3. The system according to aspect 1 or 2, comprising one or more types selected from the group consisting of animal and plant data and disaster countermeasure data.

Aspect 4
4. The system according to any one of aspects 1 to 3, wherein the accompanying data includes audio, and the information presentation means includes an audio transmitter.

Aspect 5
5. The system according to aspect 4, wherein the audio transmitting device is a speaker mounted on the vehicle or a speaker equipped by the driver.

Aspect 6
The system of any one of aspects 1 to 5, further comprising a means for measuring the driver's readiness state, and wherein the information presentation means emphasizes when providing information if the driver's readiness state is determined to be below a given threshold.

Aspect 7
7. The system according to any one of aspects 1 to 6, wherein the map image data is displayed as a bird's-eye map, a two-dimensional map, or a three-dimensional map on the information presentation means.

Embodiments of the present invention have the remarkable effect of solving the human factor issues of Level 3 autonomous driving, and even providing a new driving experience (user experience) that could not be realized with existing vehicles.

FIG. 3 is a schematic diagram illustrating the principle by which environmental information is provided from unique surrounding light. FIG. 3 is a schematic diagram illustrating the principle that the recognition of the structure of the environment is updated as the observer moves. FIG. 1 is a schematic diagram showing the interior of a vehicle to which the system of the present invention is applied. FIG. 3 is a schematic diagram showing the interior of another vehicle to which the system of the present invention is applied. 1 shows an overview of an example in which the windshield of a vehicle is used as part of an information presentation means (an example of a head-up display) for use in a system according to an embodiment of the present invention. 1 is a diagram showing the configuration around the driver's seat of a standard passenger vehicle incorporating a system according to an embodiment of the present invention. 1 is a diagram showing the interior configuration of a standard passenger vehicle incorporating a system according to an embodiment of the present invention. FIG. 1 is a schematic diagram showing an overview of a system according to an embodiment of the present invention. FIG. 2 is a schematic diagram illustrating the principle of a high-order guide according to an embodiment of the present invention. An example of car navigation that can be realized by a system according to an embodiment of the present invention will be shown. 1 shows an overview of a planning module included in a system according to an embodiment of the present invention.

The following describes in detail an embodiment of the present invention with reference to the drawings, but does not limit the present invention in any way.

(Basic principles and affordances of the present invention)
The inventors have focused on the spatial relationship and cognitive relationship between a user of an autonomous vehicle and an object outside the vehicle, and have completed the present invention.

Affordance is a concept proposed by ecological psychologist James J. Gibson, and is defined as the meaning/value that an animal (organism) that exists in the environment can acquire by exploring the environment in which it lives. When an animal, such as a human, moves through space, the scenery of the surrounding environment changes as they move. This may seem obvious, but this changing scenery is nothing other than the light stimuli being captured by the animal's eyes. Let us now consider how the light that provides information about this scenery reaches the animal's eyes.

When light emitted from a light source such as the sun passes through the atmosphere or water, it collides with fine dust and is scattered, and is also scattered and reflected on the surface of an object. This scattered reflected light fills the atmosphere in all directions. This state is called ``illumination,'' and ``everywhere'' under this illumination is surrounded by light coming from ``everywhere.'' Gibson called this fact "ambient light."

Such ambient light carries information from the surrounding surface conditions (also called texture). In other words, ambient light projects the condition and layout of the surfaces that surround it in all directions. Therefore, no matter where you are on Earth, there is a unique light structure that exists only at that location (see Figure 1). This is why, when an animal moves, its eyes see different landscapes that correspond to each location at each time. In other words, as the animal moves, the structure of the environment that the animal perceives is also updated (see Figure 2).

Additionally, as animals move, they continue to notice changes in texture. During the movement, objects in the distance come closer and appear magnified within the perspective structure, and the approaching texture passes through the sides and under the animal and flows away behind it. During this time, the texture contains numerous affordances. There is a chain of affordances here. In this chain, environmental affordances in the depth direction (beyond the animal's direction of movement) are generally blocked by buildings in front and cannot be detected. However, by continuing to move, new textures and affordances emerge from behind the shield one after another.

Applying this to the driver and other occupants of a self-driving vehicle, the occupants can detect the scenery visible from the window of the driver's seat (provided by surrounding light), feel their own movement, and create a space in which the vehicle can run safely. This means that the user is dynamically receiving an affordance signal indicating the

In general, autonomous vehicles (and their occupants) move or stand still in space, and recognize objects in the surrounding coordinates from their coordinates. The following fundamental relationship exists between space and recognition.

First, the coordinates of the occupant are defined as the geometric coordinate range of the spatial situation in which it is potentially possible to view and perceive the object (in a broad sense, this is where the optical image of a particular composition of the object exists). affordance coordinates).

Secondly, when the user exerts their will and looks at the object, the image of the object is transmitted to the retina, and the composition is recognized.

For example, from a location that has the affordance of being able to see a particular mountain, that particular mountain (in real space) can be seen in a particular composition. This can be understood by considering the location from which Cezanne sketched Mont Sainte-Victoire.

Based on these fundamental relationships, the present invention can be realized by utilizing spatial coordinate information. For example, a design that adds value to the view from the garden paths of a historic strolling garden is a prime example of linking these coordinates with cognition.

The same can be said about virtual space (space represented on a car navigation system). For example, when a passenger (driver, etc.) of an autonomous vehicle in a certain space (moving, driving) is in a position coordinate (spatial situation) where a specific object (natural terrain outside the vehicle, such as mountains, hills, capes, inlets, passes, etc., or countryside, cities, facilities, buildings, and other landscapes) can be recognized on the map image on the navigation screen, the system according to the present invention can sequentially provide information (data) about the specific object to the driver, etc. Furthermore, such objects are not limited to objects that actually exist outside the vehicle, but can also be applied to information about historical spaces and things in literary works. Examples of such information include, but are not limited to, data about old townscapes that are now lost, data about old maps, data about people and literary works associated with the area, tourist information about the area, geology, inhabiting flora and fauna, and disaster prevention measures (hazard maps). The information may also include data on the history of the industry in the area represented on the map, the history of people and companies involved in the industry, and the history of innovation.

By providing such data (also referred to as "associated data" in this specification) to the driver of an autonomous vehicle depending on the driving situation of the vehicle, the driver can always know where he or she is currently located (where the vehicle is traveling). This means that you can continuously receive information regarding the location (on the way). In other words, by presenting the driver with accompanying data that is information related to the location where the self-driving vehicle is traveling, the driver's brain continues to be exposed to information that is relevant to the situation in which the driver is moving. It turns out. That is, the information that such ancillary data represents has no real-time direct relationship to the driving task itself, which the driver cannot engage in while autonomously driving, but is related to the affordance signals that the driver perceives while being carried by the autonomous vehicle. It turns out.

In summary, the accompanying data in this specification refers to information directly related to the driving task in real time, such as information provided by conventional car navigation systems, such as the weather, road congestion, and the driving status of nearby vehicles (in this specification, (also called "pilot guide navigation"). In other words, the accompanying data can be considered to be information that is not directly related to the driving task but is related to the location (map coordinates) where the self-driving vehicle is traveling. Navigation that uses such accompanying data is also referred to as "higher-order guide navigation" in this specification.

With this configuration of the present invention, the driver's readiness state (a term used in the technical field to describe the degree of awareness "readiness" for automated driving) can be maintained at a constant level or higher, and the driver does not become bored or drowsy. Nevertheless, the driver can maintain the readiness state as long as he or she is exposed to the information presented by the system, without necessarily having to keep gazing at the control space. In other words, even if the automated driving system requests the driver to take over driving, a driver who maintains such a high readiness state can smoothly transition to the driving task without feeling any discomfort.

Furthermore, the effects of the present invention are not limited to level 3, but can also be applied to passengers in autonomous driving at levels 4 and 5. That is, as described in the above-mentioned non-patent document 1, level 4 is fully autonomous driving under specific conditions, such as on highways, and level 5 is fully autonomous driving without specific conditions.

Even during these automated driving operations, the high-level guide navigation function of the present invention can be fully utilized to enjoy the time-space theater effect of many areas according to one's hobbies and tastes.

For example, it is possible to envisage the use of the present invention in level 4 automated driving in a limited area (that is, under specific conditions). While driving a short-distance self-driving bus from a railway station to a ferry terminal, which is already being planned for practical use, the function of the high-level guide navigation according to the present invention can be used to monitor the scenery around the travel route. History may be introduced in a timely manner to the occupants of the automated bus. Furthermore, the present invention may be similarly used in self-driving buses for supporting daily life on remote islands or in mountainous areas, to provide a service such as a simple local history guide to the passengers on a weekly basis. Moreover, by using such self-driving buses for sightseeing purposes and utilizing the functions of the high-level guide navigation according to the present invention, it is possible to expect the effect of improving the quality of sightseeing guidance. In this way, by providing map information and accompanying data to the passengers and passengers of self-driving vehicles (including large vehicles such as buses), they can learn about the area they are driving in without getting bored. It also becomes possible to do this.

After the filing of this application, the period during which Level 3 autonomous driving vehicles will be put on the market, which will be released soon, will be a very important period. By utilizing the present invention during this precious period and spreading the use of the high-level navigation guide function while avoiding driver distraction, many vehicles compatible with the high-level navigation guide will be on the road. In this way, the spatial data detection function will accumulate actual conditions such as operational data of autonomous vehicles in various places and behavioral data of occupants in the vehicles, and based on this, machine learning (deep learning etc.) can be carried out to improve It is believed that this will encourage the development of advanced systems and, as a result, create a scenario in which the appeal of autonomous driving services will be demonstrated to the market. In this way, it can be understood that the present invention can be utilized more widely with the aim of putting the present invention into practical use at Level 4, which will be several years from the date of this application, and Level 5, which will be even further in the future.

The example shown in FIG. 3A is a schematic diagram of the driver's seat of a passenger car (this figure illustrates a general passenger car, but the invention can also be applied to large vehicles and other special vehicles). In this example, in order to use the high-level guide navigation efficiently, a large display is placed in the center of the front of the vehicle, making it easy to view information from the driver's seat to the passenger's seat. This large display may be an integrated type, or a plurality of displays may be arranged side by side. With this configuration, when various service contents of the high-level guide navigation are provided, it becomes easier for the occupants to obtain the information, which promotes understanding and increases enjoyment. In addition to being easily recognizable by the system, it also makes it easier to understand the status of automatic navigation devices and take over operations in accordance with the revised Road Traffic Act mentioned above.

Utilizing the present invention in this way will enhance the cooperative relationship between people and cars, increase the enjoyment of driving, deepen knowledge, and eventually lead to the creation of a new mobile civilization. An excellent self-driving vehicle equipped with the high-level guide navigation according to the present invention can expand its field of view and expand the consciousness in the human brain while moving over large terrain. This should be understood in conjunction with the explanation of FIGS. 5 and 6, which will be described later.

Figure 3B shows a schematic diagram of the interior of a passenger car according to another embodiment of the present invention. In this example, a high-level guide navigation system (also called a high-level planning project system) incorporating a "planning" module, which will be described later with reference to Figure 8, etc., is applied. In this example, in addition to the effects obtained in the example of Figure 3A, it is also possible to display survey and photograph data from the location where the passenger car is traveling, and to present a simulation of the plan. The interior of the passenger car can also be used as a place to hold remote meetings. This has the effect of making space-time studies more efficient, so to speak.

When applying the example of Figure 3A or Figure 3B to a large vehicle, it will be appreciated that the information presented to the driver of the large vehicle can be presented to passengers as well. Such presentation of information to the driver and passengers can be via their individual displays, or can be provided using a large display (continuous display) installed to partially or completely cover the ceiling, walls, and windows of the vehicle.

On the other hand, even if the car navigation system only provides the driver with information related to the driving task itself, as in the past, the driver during automated driving is not/cannot be involved in the driving task, which is most of all boring. However, it has the opposite effect on maintaining readiness. In general, it is difficult for humans to maintain interest in things in which they cannot be involved.

As mentioned above, even if the driver is provided with information (such as television programs or games) that is unrelated to the driving task or affordance signals, as in the past, the readiness state cannot be maintained.

The system according to the embodiment of the present invention overcomes the drawbacks of the conventional technology described above and also provides an excellent user experience. In other words, drivers who are freed from driving for the first time in the age of autonomous driving can freely access the space outside the car, and can have the experience of learning while having fun about the wonderful values that fill that space. This type of user experience is something that has not been explored in the automotive field so far, and it can be said to be the creation of a new-age space-time theater in which the outside and inside of the car are a common moving stage that transcends time and space.

(Basic configuration of an autonomous driving system)
A system according to an embodiment of the present invention may include a storage means, a positioning means, an information presentation means, and a means for measuring the reliability of autonomous driving of a vehicle.

The storage means may be a storage used by a computer, a car navigation system (car navigation system), or the like, and is configured to store map image data associated with map coordinates and associated data. As described above, the associated data is information associated with map coordinates but not related to the driving task.

A navigation system such as a car navigation system can be arbitrarily used as the map image data, and when processed by a processor, it can be displayed as, for example, an overhead map, a planar map, or a three-dimensional map. Note that three-dimensional maps include, but are not limited to, three-dimensional maps that can be displayed using means such as holograms, 3D displays, and 3D printers.

Associated data relates to information associated with the above map coordinates, such as natural topography and geology data, past historical map image data, engineering technology data, data on historical figures or culture, data on literature and folklore, tourist information data, and habitat. Examples include data on animals and plants that are affected by natural disasters, and disaster countermeasure data. Further, the accompanying data may include data regarding "plan innovation simulation" which will be described later.

The geological information data may include data related to geology and geography, such as data on strata structure and the existence of large and medium terrain such as volcanoes, rivers, and lakes. For example, this system can determine when a self-driving vehicle approaches the western edge of Fossa Magna (Itoigawa-Shizuoka Tectonic Line) and present information about Fossa Magna to the driver.

Past historical map image data may include not only so-called old maps but also data on maps that were the setting for literary and artistic works. For example, it may be data about a map tracing the route of "Tokaido Chuuhizakurige." This system can determine that the location where the self-driving vehicle is traveling corresponds to the location where those works are set, and present information about this to the driver.

The engineering data may include data related to engineering such as architecture, civil engineering, bridges, and roads. For example, this system determines that a self-driving vehicle has entered the Seto Ohashi Bridge, or that the self-driving vehicle is in a position where the driver can see the Seto Ohashi Bridge, and provides information about the history, bridge structure, related local governments, etc. of the Seto Ohashi Bridge. may be presented to the driver.

Data on historical figures or culture can include data on people or cultures connected to the land. For example, the present system may determine that the self-driving vehicle approaches Sakuradamon and present data regarding Naosuke Ii to the driver.

Literature and folklore data can include data on literature and folklore related to the area (or on movies and manga related to them). For example, the system can determine that an autonomous vehicle is in a position where the driver can see Mount Hakkoda (an example of a medium-sized terrain), and present the driver with information (such as a bibliography and reviews) about Jiro Nitta's novel "Death Wandering on Mount Hakkoda."

The tourist information data may include data on tourism, travel, products, etc. in the area. For example, the system may determine that the autonomous vehicle is approaching the coast where Ibaraki's Goura Rokkakudo is located, or that the autonomous vehicle is in a position where the driver can see Goura Rokkakudo, and present tourist information (such as information on admission times and admission fees) to the driver. Or, for example, the system may determine that the autonomous vehicle is approaching a point where the Goura Rokkakudo is not yet visible to the occupant but is in its vicinity, and present information about Okakura Tenshin to the occupant.

Data on inhabiting animals and plants may include data on livestock and agricultural products at ranches, farms, etc., in addition to wild animals and plants. For example, this system determines that a self-driving vehicle approaches a fishing port in Onomichi City, Hiroshima Prefecture, or that the self-driving vehicle is located in a position where the driver can see the fishing port, and then sends information about stonefish that can be harvested at that fishing port. It may also be presented to the driver.

Disaster prevention data may include information about earthquakes, floods, rock falls, landslides, etc., related to the region. For example, the system could determine when the autonomous vehicle is in a position where the driver can see a monument in an area that was hit by a historical disaster in the past, and present information about the disaster and the monument to the driver.

The accompanying data is not limited to visual images (still images, moving images), and may also include audio information. It is also possible to include information regarding smell, touch, and taste through some kind of sensor or blowing means.

The positioning means included in this system determines the current vehicle position in map coordinates and may be, but is not limited to, a GPS or Quasi-Zenith Satellite System (QZS) that can be used by car navigation systems.

The information presentation means included in this system may be anything that can present information to the driver of the vehicle, such as a display (such as a car navigation system monitor, a television monitor, or a mobile device screen). Alternatively, it may be a projector that projects information onto the window or wall of an autonomous vehicle. When such a projector is used on a window, it is possible to obtain the effect of directly overlaying information onto the passing scenery seen outside the window. In this case, the scenery outside the window can be considered to be treated as data that reflects map image data (dominating as map image data). Alternatively, in such a case, the system may use both map image data and the scenery outside the window to overlay associated data.

Figure 4A shows an overview of an example of using a vehicle's windshield as part of the information presentation means (an example of a head-up display). The head-up display according to this embodiment includes a projector that can project information onto the windshield, an occupant position sensor that detects the driver's position, line of sight, etc., and a camera that photographs the scenery outside the windshield. and imaging means for. Such imaging means (cameras, etc.) take pictures of the scenery outside the vehicle, and the computer means performs image processing, and as described above, corresponds to the map image data, so that the topography, etc. (mountains, etc.) in the scenery can be seen on the map. It is possible to determine which coordinates correspond to which coordinates in the landscape (obtain the landscape of the afford point of the landscape). Then, accompanying data (mountain name, etc.) regarding the mountain, etc. can be associated with the mountain, etc. associated with the coordinates on the map.

The occupant position sensor can also calculate the line of sight of the driver or other occupant from the occupant's posture and physical characteristics, and simulate the scenery as seen by the occupant's eyes. The projector can then project the associated data so as to overlay it on the scenery that the occupant would be seeing. Preferably, the projector's projection can be automatically corrected according to changes in the vehicle's operation and the occupant's posture.

Means for measuring the reliability of autonomous driving of a vehicle may include means known in the technical field, such as human and object sensors, radar, lidar, and V2X (vehicle to X). Furthermore, the means may be a means that integrates the driving control system. These means allow the autonomous driving system to know the timing to switch between autonomous driving and manual driving. For example, the degree of reliability may be displayed on the information presentation means described above using graphs or numerical values.

When the vehicle is in a state where operation of the vehicle is judged not to require driver operation based on the reliability thus measured (i.e., an autonomous driving state), information in which map image data related to the current vehicle position in the map coordinates is overlaid with associated data related to the current vehicle position in the map coordinates is presented to the driver via the information presentation means.

In one embodiment, the system may further include an imaging means (such as a camera) for capturing images of the scenery outside the vehicle. The system may also include an editing means (such as computer software) for creating associated data based on images and/or videos of the scenery captured by the imaging means. Such an editing means can process and edit information stored in the database to automatically generate associated data.

For example, an image of a landscape obtained from an imaging means is associated with the position and track of a corresponding self-driving vehicle, and is classified through machine learning performed by a computer processor to generate accompanying data that is superimposed on map image data. It's okay. The acquisition of landscape images by such an imaging means is preferably continuous, but may be intermittent or discontinuous.

In some embodiments, a determination as to whether a driver of an autonomous vehicle can view a particular scene associated with accompanying data may be made based on map image data. This can be done, for example, by including in the map image data data about the elevation, shape, or other suitable geometry of existing terrain and relief, and performing calculations to simulate the driver's field of view. Preferably, the driver's physical characteristics (height, eye position, etc.) may also be factors used in the calculation of the simulation.

In another embodiment, the driver of the self-driving vehicle may determine whether or not the specific scenery associated with the accompanying data can be visually recognized using the imaging means described above. For example, such an imaging means may be installed near the driver's eyes to reproduce the scenery from the driver's viewpoint. Alternatively, as another method, images obtained by the imaging means (preferably a plurality of imaging means) are combined, taking into account the relative distance and angle between the imaging means and the position of the driver's eyes, to reproduce the scenery from the driver's perspective. You may.

In some embodiments, the system may include a voice transmission device such as a speaker. For example, (a part of) the associated data including the voice data may be output from the voice transmission device. The speaker may be mounted in the vehicle or may be a portable speaker carried by the driver.

In some embodiments, a smart speaker may be employed as the audio emitting device, and in addition to audio output as described above, it may also accept audio input from the driver and other occupants. The system may be operated by such voice input. For example, such smart speakers can search for and set locations for self-driving vehicles to navigate to, and adjust the brightness and volume of displayed content. Furthermore, adjustments may be made to suit the intellectual level of the user, such as by changing the level of expertise of the guide.

In one embodiment, the system may include a means for measuring the driver's readiness state. Examples of such means include a sensor (such as a camera) for measuring the driver's eye movement, a sensor for measuring the driver's posture, and a sensor for measuring the driver's brain waves. From the results of such measurements, the driver's readiness state may be evaluated and quantified based on some measure. The system can determine whether the quantified readiness state is below a threshold determined based on some standard (such as a standard that may be determined by a government agency or an industry group). The readiness state may also be displayed on a display means (which may be the information presentation means described above or another display) possessed by the system. Examples of such displays include, but are not limited to, numerical values, bar graphs, and progress bars.

In some embodiments, the information presenting means may provide the information with emphasis when it is determined that the readiness state of the driver is below a given threshold. Such enhancements include, for example, increasing the contrast or increasing the amount of light when displaying information visually, or increasing the volume when displaying information audibly.

FIG. 4B is a diagram showing a configuration near the driver's seat of an ordinary passenger car (regular car) incorporating a system according to an embodiment of the present invention. It is assumed that this ordinary car can be operated at any level of autonomous driving. (For example, in the case of Level 5 autonomous driving, the dotted line in the diagram indicates that the steering wheel can be retracted.)

In the example shown in FIG. 4B, information can be presented on information presentation means (such as displays) around the driver's seat (cockpit), for example, corresponding to *2 and *3 in FIG. 5 and FIG. 6 described below. In this example, multiple displays are arranged side-by-side across the space between the driver's seat and passenger seat of a right-hand drive vehicle, making it possible to present various information to the driver (and the passenger in the passenger seat, if necessary).

Figure 4C is a diagram showing the interior configuration of a standard passenger vehicle (standard car) incorporating a system according to an embodiment of the present invention. This standard car may be equipped with the driver's seat shown in Figure 4B.

Information presentation means (displays) such as those shown in Figure 4A or Figure 4B above are installed around the front seat (driver's seat) inside the vehicle, allowing the driver to view information and, if desired, providing information to passengers (users) in the passenger seat or rear seat. A separate information work desk is also provided in the rear seat, where information presentation means (displays, mobile PCs, etc.) can also be installed. The information presentation means in the front seat and the information presentation means in the rear seat can work in conjunction with each other via a network that uses an in-vehicle LAN or wireless communication lines, allowing each passenger, including the driver, to share information. In addition, users outside the vehicle and planners in the "planning" section described below can also remotely connect to such a network to share information.

In the example shown in Figure 4B and Figure 4C, information on the geological features of the river terrace group is presented to each passenger in the vehicle and shared. That is, as the passenger vehicle moves (from left to right in the figure), the high-level guide navigation system of the present invention built into the passenger vehicle can provide navigation with nearby geological features as accompanying data (based on a topographical and geological database, etc.). For such navigation, for example, a bird's-eye view image as shown in the lower part of Figure 4C can be provided, and the accompanying data can be displayed superimposed on the current position of the passenger vehicle. In this way, it becomes possible to provide information not only on the scenery that the passengers can see directly from the vehicle window, but also on geological features that cannot be seen directly or that would not be noticed without the necessary knowledge (such as the formation of river terraces).

By making it possible for this invention to provide and share such information, the time-space theater effect can be applied to multiple passengers in the car, and analysis, planning, and engineering of the area being traveled through can also be made possible. In other words, the interior of the car itself can become a mobile remote office, and a teleworking base where passengers can enjoy a rich natural space. This effect could not be achieved with conventional car navigation systems.

Furthermore, the system according to the present invention can also adjust how information is displayed on an information presenting means (display) that has a certain display area. For example, in the image shown at the bottom of Figure 4C, the regular car is moving from left to right on the screen, so the display on the left side of the center of the display group on the front seat side also shows the car's position. By displaying the vehicle in such a way that the vehicle is coming, it is possible to obtain the effect of making it easier to recognize and understand the entire driving space.

Figure 5 is a diagram showing a schematic overview of a system according to an embodiment of the present invention. The maneuvering space (*4) shown in the lower left of the figure is what the driver would have had to focus on in the past. With autonomous driving, this focus is no longer necessary. With this system, the driver senses (for example, through the vehicle's windshield) the entire space (*1) that surrounds the vehicle. This sensing is performed by the driver's brain function (*6b) via the driver's visual system and other sensory inputs (*6a).

While the vehicle is being driven autonomously by the autonomous driving system (*12), the reliability of that autonomous driving is continuously measured and evaluated (*11). During autonomous driving, the high-level guide navigation shown in the upper left of the figure (*2) functions to maintain the driver's state of readiness.

The high-level guide navigation is displayed superimposed on the map image data. At this time, the operation guide navigation (*5) according to the prior art and the higher-order guide navigation may be switched and displayed, or the operation guide navigation and the higher-order guide navigation may be overlapped. The driving guide navigation may display a parameter (STATUS) that quantifies the readiness state as described above, or may display a parameter that quantifies the reliability of automatic driving (*5).

In addition, the scenery outside the vehicle is continuously captured by the imaging means, and is also used to automatically generate data (*7) for use in high-level guide navigation. All such data may be automatically edited, or some or all of it may be saved and sent for offline editing (*8). The data created in this way can be linked to the vehicle's current location (coordinates on the map image data) (*9) and displayed as a high-level guide navigation.

Based on the feedback from the system for autonomous driving and the system for measuring the reliability of autonomous driving (*13a to 13d), the reliability of autonomous driving and the readiness state of the driver can be measured (*12b). If it is determined that the reliability of automated driving has deteriorated (driver intervention is required), a request to take over driving is made to the driver (*11a), and if the request is accepted (*11b). , can execute takeover (*11c). Furthermore, the output of the piloting guide navigation may be adjusted to emphasize and guide changes in the maneuvering environment (*10) based on such feedback (*12c).

This configuration is based on the fact that the driver's brain is richly formed with high-level thinking and behavioral programs based on spatial awareness. In other words, the driver's spatial awareness can easily transition to a dynamic maneuvering program within a limited maneuvering spatial awareness (*6c). Therefore, this maneuvering program cooperates with the basal nucleus loop involved in maneuvering and can increase the possibility of guiding the vehicle to the safe travel zone.

Figure 6 is a schematic diagram explaining the principle of a high-order guide according to an embodiment of the present invention. First, from the surrounding light conditions (*1) based on the time, season, weather, etc., the imaging means of this system (such as a landscape camera) can obtain information on the texture of the object (affordance signal) (*2).

Based on this information, the system's processor can classify and recognize (through machine learning, for example) the images of various objects (landforms, mountains, capes, inlets, etc.) that have been photographed(*6). By matching the recognized objects A, B, ... with coordinate data(*5) obtained from a spatial coordinate system database (map database), it is possible to generate data that positions each object in geometric space(*3).

It is also possible to associate the recognized objects A, B, ... with semantic data obtained from a map database (such as data that a certain mountain or a certain cape is located at a certain point)(*4) in order to recognize what those objects are.

In the manner described above, an Afford space system (i.e., data representing a spatial system in which semantic spaces are arranged in the landscape that the driver can see from the vehicle) can be calculated and constructed(*9). Then, by relating the Afford space system to data showing the vehicle's position on a map (car navigation data)(*10), a high-level guide that can be presented to the driver can be created(*8).

When the output from the Afford space system is passed to a high-level guide automatic generation system, the automatic generation system can continuously select guide targets (targets related to the information presented to the driver) according to the operation of the vehicle (*11a-11b). Guide targets can be extracted from a multi-layered time-space database (a database that overlays data related to nature, culture, literature, technology, topography, geology, history, etc. onto the corresponding map image data) (*12).

The data objects A, B, ... obtained in this way can be further organized into a hierarchy, and child objects corresponding to the existence level, knowledge level, emotion level, etc. can be associated with each other. Also, various contents such as images, icons, sounds, music, etc. can be received (*13) from an external server (*14) and used to edit such data objects.

Based on the data obtained in this way, or the output data obtained by passing that data to an offline editing system (*11c), the automatic playback system adjusts to changes in the landscape (the flow of the landscape as the vehicle moves). , the timing of high-level guide playback (presentation to the driver) can be controlled (*15). This allows the driver and other occupants to maintain a high level of readiness by synchronizing the real world with the stream of consciousness of the driver and other passengers. It also has the effect of satisfying the intellectual curiosity of passengers other than the driver (*15b). Information may be presented via a head-up display (*16).

Communication between the vehicle occupants and the system may be an interactive interface(*7), which may include, for example, a speaker and microphone(*17-19).

Figure 7 shows an example of car navigation that can be implemented by a system according to an embodiment of the present invention. In this example, Tokyo is the starting point and Niigata is the destination. The example route is shown by the dotted line.

Drivers in self-driving vehicles can set destinations and routes like a normal car navigation system. From this system, you may (easily) introduce the entire route system. Below, we mainly consider examples that target large and medium terrain.

When the vehicle departs, passes through the Kanto Plain, and the system determines that the driver can see the Kanto Plain, it can output characteristics of the Kanto Plain (such as geological characteristics) as a high-level guide, and can present information about the formation history of the alluvial lowlands, for example.

When the system determines that the vehicle is at a point where the driver can see Mount Asama, Mount Haruna, or Mount Akagi, it can present information about the principles of the formation of the volcanic fronts of those mountains, for example.

When the system determines that the vehicle has reached a point where the Tone River terraces are visible to the driver, it can present information about the geological characteristics of those river terraces, for example.

When the system determines that a vehicle is entering or has entered a tunnel area (near Gunma Prefecture) that passes through the Tanigawa Mountains, it can present information about tunnel excavation techniques and the construction history of the Kan'etsu Tunnel, for example.

When the system determines that the vehicle has reached a point where the driver can see Echigo Yuzawa or Hakkaisan, it can present information about each (such as information about folk tales related to the area).

When this system determines that the vehicle has arrived at a point where the driver can see the Uonuma Formation and river terraces, it can present information about each (such as information on history and geology).

When the system determines that the vehicle has reached a point where the driver can see the alluvial lowlands of Niigata, it can present information about the formation of raccoons, for example.

In another embodiment of the present invention, the system may further include a module (subsystem) that provides "innovation simulation" information. In this specification, "innovation simulation" information refers to information about urban planning or industrial development in a certain area, and is information that can explain the origins and history of so-called innovations.

Figure 8 shows an overview of such a planning module. For example, such a module may be placed alongside the time-space database (*12) in Figure 6 so that it works in cooperation with it. As an example, Figure 8 shows a planning module that divides innovation propositions into three hierarchical levels: large-scale, medium-scale, and small-scale. Note that Figure 8 shows examples of success, but it goes without saying that this is not limited to these and failures may also be included.

Large-scale issues refer to advanced, innovative regional development and smart city development that affect the entire region.

Information on the proposition of advanced innovation-based regional development could be, for example, data on innovation in time and space related to a certain region (e.g., information on the history of innovation in semiconductors and computer hardware/software in Silicon Valley). In other words, the planning module can store as "data on large-scale propositions" information on how industry has developed based on a certain region's social and cultural foundations (such as what kind of people moved to the region and how they started businesses), economic foundations (such as what kind of people provided support for starting businesses and how), and topographical structure (such as how rivers and mountains are arranged, and what facilities were built where in relation to them).

Information related to the proposition of smart city development includes, for example, how the layout and control methods of electric power and hydrogen energy supply networks, information communication networks, security systems, or automated transportation systems are related to the overall urban development of the area. May contain information about.

Medium-scale issues refer to issues related to the composition of parts of a region (districts), such as telework district development and relocation plans for river flood areas.

Information on the topic of developing teleworking districts could include, for example, information on where and how districts suitable for teleworking have been established. For example, information could be available on the fact that a certain company or local government has established such a district on a cool plateau so that teleworking can be done comfortably even in the summer.

Examples of information related to the proposition of river flood area relocation plans include information on the history or plans to relocate functions from areas that are judged to be at high risk of disasters based on hazard maps to other areas that are judged to be at low risk of disasters.

Small-scale propositions refer to disaster prevention measures, natural environment restoration plans, lifeline plans, etc., that are related to the composition of a small part (plot) of a region. Examples of information related to these include how water supply and sewerage systems were laid out or reforestation was carried out on a certain plateau or residential area.

The information described above can be stored in a storage means (server, storage, etc.) related to the "reference system" in Figure 8. This information is not limited to information about a specific region, but may systematically include data about regions around the world.

Based on the data possessed by the "reference system" and the data possessed by the spatio-temporal database (*12) in Figure 6, a certain area (not necessarily data stored by the "reference system") to which higher-level guides are provided is determined. (note that this does not have to correspond to the region concerned), it can provide a simulation of how development could proceed. This is referred to as a "proposal" in this specification. For example, by converting the location of rivers and mountains in region A, which is the target of high-level guides, into data and statistically comparing it with the data in another region B held by the "reference system," we can actually conduct research in region B. It is possible to simulate how the city plan should be customized if it were to be realized in this area A. This simulation can demonstrate a complex planning effect by moving the site vertically and horizontally using this high-level guide navigation and adding evaluations.

(Application examples for large, medium, and semi-medium vehicles)
The configurations shown in FIGS. 3A and 3B can also be applied to large vehicles such as buses, medium-sized vehicles such as microbuses, and semi-medium-sized vehicles such as vans. In general, drivers of large vehicles such as buses are professionals, and even if Level 3 autonomous driving is adopted in the future, drivers will still be responsible for monitoring their own vehicle's autonomous driving status, road conditions, readiness status, etc. Therefore, by using the system according to the present invention, the service status of the high-level guide navigation provided to passengers can be managed and adjusted in general, thereby efficiently preventing the driver from becoming distracted. becomes possible. That is, although the situation is slightly different from that in which a driver of a general passenger car maintains a state of readiness while utilizing a high-level guide navigation during automatic driving, it can be understood that the effects of the present invention can still be obtained.

For example, a high-level guide navigation integrated display (or a collection of multiple displays) can be installed near the driver's seat of a bus and used for information integration and management. The operation for management may be performed by the driver, or may be performed remotely by another manager or navigator. Furthermore, content can be provided to passengers by utilizing the system according to the present invention. Individual high-level guide navigation displays and voice transmitters are installed near each passenger seat, allowing information personalized to each passenger to be output. Such personalization may be done, for example, by analyzing the passenger's knowledge, interests and preferences based on information entered by the passenger, or based on input from some database about the passenger. good.

Further, a part or the whole of the ceiling surface, wall surface, or window of a large vehicle may constitute a large display so that information can be provided to a plurality of passengers at once. By doing this, it is possible to give passengers a sense of realism as if they were being shown information in a theater (also referred to as a ``spatio-temporal theater effect'').

In such large vehicles, each passenger seat may be a private (or semi-private) compartment. This will make it easier for each passenger to concentrate, help protect passenger privacy, and help prevent infectious diseases. When configuring such private rooms (or semi-private rooms), epidemic prevention shields and air conditioning may be used.

According to the disclosure of this specification, it will be understood that various operation routes and combinations of accompanying data are possible in addition to the examples described above.

In an embodiment of the present invention, a device (hardware) capable of implementing the system, a program, and a product (any medium, carrier wave, module, etc.) storing part or all of the program in a form executable by a user can also be provided. The system according to the embodiment of the present invention may be integrated or distributed. If it is distributed, the computation process may be performed by communicating with a remote resource (such as a server) outside the system.

As described above, the present invention can utilize various databases. In particular, the advantage of the present invention is that it is possible to use a valuable database that systematizes topography, altitude, aerial photographs, geological distribution, etc. such as national land infrastructure map information. Conventionally, such a database was used on a desk, but with the present invention, it can actually be used on-site, in a moving vehicle, or in a mobile office. With the system according to the present invention, it is possible to provide experiences based on such a database along with spatial and chronological axes, along with various expressions such as detected landscapes. Therefore, we believe that the present invention will enable users to acquire a more complex body of knowledge and will bring about new enjoyment, excitement, and creativity.

Claims (6)

1. A system for use in automated driving of a vehicle, comprising:
A storage means configured to store map image data associated with map coordinates based on a spatial coordinate system and associated data associated with the map coordinates and not related to a driving task;
positioning means for determining a current vehicle position and status in said map coordinates;
an information presentation means for presenting information to a driver and/or passengers of the vehicle;
and a means for measuring reliability of the autonomous driving of the vehicle;
The system is characterized in that, when the operation of the vehicle is judged to be in a state where the driver's operation is not required based on the measured reliability, the information presentation means is configured to present to the driver and/or passengers information that is displayed as an overhead map or a three-dimensional map and that is obtained by superimposing the map image data related to the current position of the vehicle in the map coordinates with the associated data related to the current position of the vehicle in the map coordinates, superimposed on the scenery outside the vehicle. Further, an imaging means for photographing the scenery outside the vehicle;
2. The system according to claim 1, further comprising: editing means for creating the accompanying data based on a landscape image and/or moving image photographed by the imaging means. The system according to claim 1 or 2, wherein the associated data includes one or more types selected from the group consisting of natural topography and geological data, historical map image data, engineering technology data, data on historical figures or culture, literature and folk tale data, tourist information data, data on inhabiting flora and fauna, and disaster prevention data, which are associated with the map coordinates. The system according to any one of claims 1 to 3, wherein the associated data includes voice, and the information presentation means includes a voice transmission device. The system according to claim 4, wherein the voice transmission device is a speaker mounted in the vehicle or a speaker carried by the driver. further comprising means for measuring the readiness state of the driver, and when the readiness state of the driver is determined to be below a given threshold, the information presentation means emphasizes when providing information. System according to any one of claims 1 to 5, characterized in that: