JP3876972B2 - GUIDE IMAGE GENERATION DEVICE, GUIDE IMAGE DISPLAY DEVICE, NAVIGATION DEVICE, AND PROGRAM - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a technique that contributes to appropriate guidance of a vehicle or the like.
[0002]
[Prior art and problems to be solved by the invention]
2. Description of the Related Art Conventionally, a car navigation apparatus has a display screen composed of a liquid crystal, a CRT, etc., and displays a plane map around the vehicle on this display screen, and displays the vehicle position on the plane map. Thereby, not only the route guidance according to the route to the destination but also when the route guidance is not performed, the user can confirm his current position, which is convenient.
[0003]
However, even though the position on the plane map can be confirmed, it is often the case that in actual driving, the vehicle will pass an intersection to turn left or right, or turn right or left at the wrong intersection. This is because it is not easy to associate the position on the map displayed on the navigation screen with the scenery (roads and buildings) actually seen through the windshield.
[0004]
For this reason, recently, a navigation device capable of displaying a building or the like in a three-dimensional manner has been developed, and is associated with a scene that can be actually seen.
For example, the area and height of a building are prepared as map information and displayed as CG, or 3D model information corresponding to the building is created in advance, and this 3D model information is displayed as CG. And so on. In the present specification, so-called computer graphics are expressed as “CG”.
[0005]
However, a building displayed in this way is generally displayed as a simple rectangular parallelepiped, for example, if it is a building, and is still different from an actually visible building. Therefore, although it is easier to associate with a plane map, it can still be said that it is difficult to associate with a landscape in a short time. Of course, if a detailed CG image can be generated, such a problem can be solved. However, if it is intended to display the buildings in various parts of the country in detail, the amount of data becomes enormous, and the display of the CG image is also possible. Processing takes a considerable amount of time, which is not realistic.
[0006]
In order to solve such a problem, Japanese Patent Application Laid-Open No. 11-108684 discloses a technique for capturing a scene in the traveling direction of a car and using the captured real image. In this technique, a mark indicating the position of the vehicle and an arrow indicating the direction of travel are superimposed on the live-action image.
[0007]
This technique is excellent in terms of associating with a scene that can actually be seen, but the following problems remain in terms of providing guidance.
That is, there is too much extra information in the live-action video. For example, roadside trees that are not necessary for guidance are reflected along with roads and buildings. In addition, shadows of buildings and subtle changes in color are also reflected. Therefore, even if an arrow indicating the direction of travel is displayed on the live-action image, there is a high possibility that it will be difficult to confirm an intersection to be bent. In other words, if a live-action video is used directly, even if it is easy to associate, a situation in which appropriate guidance is hindered by information unnecessary for guidance is caused.
[0008]
The present invention has been made to solve such a problem, and an object thereof is to generate a guide image that can be easily associated with a scene that can be actually seen and that realizes appropriate guidance.
[0009]
[Means for Solving the Problems and Effects of the Invention]
In the guidance image generation apparatus of the present invention made to achieve the above-described object, guidance contribution information that contributes to guidance is generated as CG images among the information in the live-action video, and this is combined with the real-life video for guidance. Generate an image. In other words, paying attention to the fact that information that contributes to guidance and information that does not do so are mixed in a live-action video, and only information that contributes to guidance is synthesized as a CG image, so that information that does not contribute to guidance is separated from information that does not contribute to guidance. It was made to be recognized separately.
[0010]
If such a guide image is used, since the base is a live-action video, it is easy to associate with the scenery that is actually seen, and the guidance contribution information is recognized separately from other information. realizable.
Note that the present invention can be used by being mounted on a vehicle, for example. In this case, on the assumption that the navigation device is mounted, the guide image may be displayed on the display means included in the navigation device. It can also be applied to mobile devices such as mobile phones, handtops and laptops.
[0011]
  It is conceivable that the CG image generating means for generating a CG image includes a geometric information extracting means and a CG converting means. At this time, the geometric information extraction unit extracts the geometric information corresponding to the guidance contribution information from the live-action image captured by the imaging unit, and the CG generation unit generates CG by converting the geometric information into CG.The
[0012]
  For example, the geometric information extraction means may extract geometric information based on map information and position related information.TheThe map information includes information on roads and buildings, and can be obtained from map information storage means such as a DVD-ROM. On the other hand, the position related information includes information for specifying the position and moving direction of the apparatus, and may be acquired from, for example, detection means configured using a GPS receiver. The same applies to the map information and position related information described below. In addition, if it is set as the structure which acquires map information and position relevant information from the outside, this apparatus does not need to be equipped with this means in particular.
[0013]
  When generating the guidance contribution information as a CG image, the CG converting means may color-code the geometric information based on the map information and the position-related information and convert it to CG.Yes.This is because it becomes easy to distinguish a plurality of information displayed as a CG image.
  By the way, as an example, the geometric information corresponding to the guidance contribution information may be road information including at least one of a road edge, a center line, a lane, and a pedestrian crossing.TheFor example, the travel path information is extracted by first searching a part of a live-action video where a change in pixel value or the like is large, and whether this part is a road edge, a center line, a lane, a pedestrian crossing, or the like. Can be realized by using map information including travel route information and a position on the map based on position-related information. At this time, the CG conversion means may color-code various information as the travel route information and convert it to CG.Yes.In this way, it is easy for the user to distinguish various types of information as travel route information, and more appropriate guidance can be realized.
[0014]
  The geometric information corresponding to the guidance contribution information may be building information indicating the outline of the building.TheAlso in this case, extraction is performed using the map information and the position related information in the same manner as the travel route information. Then, the CG conversion means converts the building information into a CG by color-coding according to the type of building.Yes.The type of building may be a function-specific type such as a building, a pedestrian bridge, or a traffic light. Color coding from such a viewpoint makes it easy to use a building as a landmark for guidance. However, from the viewpoint of enhancement of the guide surface, it is effective to color-code according to the facility type including the types of buildings, such as the types of facilities such as hotels, restaurants, and sports centers.
[0015]
  In consideration of extracting geometric information from a live-action video, it may be possible that geometric information is not appropriately extracted depending on the case. For example, it is assumed that a part of the road edge is not extracted because it is blocked by roadside planting or the like.
  Therefore, the CG image generation unit described above may further include a geometric information correction unit that corrects the geometric information extracted by the geometric information extraction unit. At this time, the CG converting means converts the geometric information corrected by the geometric information correcting means to CG.TheSpecifically, the geometric information correction means may correct the geometric information based on the map information and the position related information.TheThe geometric information may be corrected based on the imaging conditions.Yes.The imaging condition refers to, for example, the mounting position and angle of an imaging means embodied as a video camera, or exposure.
[0016]
By performing such correction, the geometric information repeatedly extracted from the live-action video becomes stable, and an appropriate CG image can be generated and an appropriate guide image can be synthesized.
In the above, the generation of the CG image by extracting the geometric information from the live-action video has been described. However, it is also conceivable to configure the CG image generation means as follows.
[0017]
  That is, the CG image generating means includes a model storing means, a model information reading means, and a CG converting means. The model storage means stores model information obtained by modeling the above-described guidance contribution information as a three-dimensional model, and the model information reading means reads model information from the model storage means. Then, the read model information is converted to CG by the CG conversion means to generate a CG image.1). This is a technical idea in which, for example, model information corresponding to a building or the like is stored in advance, and a CG image is generated from the model information to generate a guide image.
[0018]
  Of course, in addition to the above-described configuration for extracting geometric information, a configuration for generating a CG image from such model information may be adopted.Yes.In that case, the CG converting means generates a CG image by converting the geometric information and the model information into a CG. In this configuration, it is conceivable that the geometric information extraction means extracts geometric information as necessary based on the storage state of the model information in the model storage means. For example, when there is no model information corresponding to map information, geometric information is extracted. Also, for example, geometric information may be extracted even when there is only simple model information. If detailed model information is available, detailed CG images can be generated, so there is no need to extract geometric information. However, if there is only simple model information such as a simple rectangular parallelepiped, the geometric information is also extracted and converted to CG. This is because there is a high possibility that the CG image can be made more detailed.
[0019]
  Note that the model information reading means may read the model information based on the map information and the position related information.3). At this time, the CG conversion means may color-code the model information based on the map information and the position-related information and convert the model information into CG.4). This is because it becomes easy to distinguish a plurality of information displayed as a CG image.
[0020]
  By the way, the model information corresponding to the guidance contribution information may be, for example, building model information obtained by modeling a building.5). The building model information may be information obtained by modeling a building in detail. Or it is good also as what expressed the building using the rectangular parallelepiped.6). The use of building model information such as the latter is advantageous in that the data amount of model information can be suppressed. At this time, the CG conversion means may convert the building model information into a CG by color-coding according to the type of building (claims).7). As described above, color coding according to the type of building, such as building, pedestrian bridge, traffic light, etc. for use as a landmark for guidance, color coding by function, hotel, restaurant, sports center, etc. Color coding by facility type is considered. In addition, from the viewpoint of using a building as a guide for guidance, it is also important to make it easy to recognize the outer shape of the building. Therefore, the CG converting means may perform CG by emphasizing the outline of the building based on the building model information.8).
[0021]
  Although the building model information is mentioned here, for example, roads and the like can also be modeled and stored as a three-dimensional plane model, and can be generated as a CG image as well as the building model information described above. .
  Note that the CG image generating means may include a model position matching means for correcting the viewpoint for CG conversion.9). At this time, the CG converting means converts the model information to CG from the viewpoint corrected by the model position matching means. For example, the viewpoint for CG conversion is corrected based on the imaging conditions.0). The imaging conditions here mainly mean the mounting position and angle of the imaging means.
[0022]
As a result, a CG image indicating guidance contribution information can be combined with a live-action video in accordance with the driver's view. As a result, it becomes easier to associate with the scenery that can actually be seen, which contributes to appropriate guidance.
By the way, the guidance image is based on, for example, a live-action image from an imaging means attached to the vehicle body, and corresponds to the driver's view. This is because the focus is on the actual view and correspondence. However, it is needless to say that it is more convenient if the configuration can change the viewpoint as needed.
[0023]
  In view of this, it is conceivable to further comprise a viewpoint changing means for changing the viewpoint and generating a guide image.1). The viewpoint changing means uses, for example, a plurality of live-action videos from the imaging means constituted by a plurality of video cameras, or a plurality of live-action videos output from the imaging means before and after the time change. I do. That is, if there are a plurality of live-action images as two-dimensional information, three-dimensional information can be configured and the viewpoint can be changed.
[0024]
  Note that, when generating the guide image, the image composition means can change the composition ratio of the live-action video and the CG image.2Or after performing image processing on a live-action video, the composition is combined with a CG image (claim 1).3) Is also useful.
[0025]
  The change of the composition ratio is realized by changing the so-called blending ratio of pixel values. The image processing includes gradation processing, color reduction processing, and region integration / division processing in a live-action video.4, 15, 16). The purpose of image processing is to remove unnecessary information such as subtle color changes contained in live-action video. In other words, unnecessary information is made inconspicuous by emphasizing the necessary information called guidance contribution information, and here, unnecessary information is actively removed.
[0026]
By adopting these configurations, it is possible to provide a guide image that is easy for the user to see.
Up to this point, the description has been given of guidance performed by synthesizing guidance contribution information existing in a live-action video as a CG image. That is, making the user recognize the guidance contribution information existing in the live-action video is a kind of guidance.
[0027]
  And in order to implement | achieve more useful guidance, it is possible to employ | adopt the structure which presents the guidance information produced actively.
  In other words, the system may further comprise superimposing means, and the superimposing means may superimpose the guidance information in the guidance image.7). The guidance information here may be generated in this apparatus or may be acquired from the outside.
[0028]
  It is conceivable that the guidance information includes information on signs on the road and destination guidance information.8). Such information may be acquired as map information. Then, it is possible to superimpose at an appropriate timing based on the map information and the position related information.19). The appropriate timing may be, for example, when approaching a sign or approaching an intersection corresponding to the destination guidance. Further, it is preferable to superimpose the information on the sign and the destination guidance in association with the road in the guidance image.0). For example, a sign may be superimposed on the side of the road, or a destination guidance arrow may be superimposed on the road. In this way, the effectiveness of information presentation can be ensured.
[0029]
  Further, it is considered that the guidance information includes information on operation guidance for the user.1). The operation guidance information here includes switches, menus, and the like when the display screen is used as a touch panel. Such information may be superimposed on the lower part of the screen.2). This is because, in the guidance screen displayed corresponding to the driver's view, the screen upper image corresponding to the front in the traveling direction is highly important.
[0030]
  Furthermore, it is conceivable that the guidance information includes information on route guidance by the guidance means.3). Although the route guidance information may be generated as a configuration in which the present device itself includes guidance means, it may be acquired from an external navigation device, for example. Such route guidance information may be superimposed at the guidance timing by the guidance means.4).
[0031]
  Specifically, it is conceivable that the route guidance information includes an arrow image indicating the direction of the guidance route. In this case, the arrow image may be superimposed on the road in the guide image.5). For example, the arrow image is superimposed on the road so that the user can recognize straight, right and left turns at the intersection (claim 2).6). In this way, the effectiveness of route guidance is ensured.
[0032]
  By the way, when the map information is old, it is conceivable that buildings and roads that are not included in the map information are displayed in the live-action image. Therefore, it is desirable that the CG image generation means generates a CG image based on the actual video when the real video captured by the imaging means is different from the map information.Yes.This is because the guidance image corresponding to the scenery that can be actually seen becomes easy to associate with the scenery. In this case, the updating unit may further include an updating unit, and the updating unit may update the map information.Yes.Of course, in this case, it is necessary to use a rewritable recording medium for the map information storage means.
[0033]
  As described above, the CG image is generated using the position related information. However, when mounted on a vehicle, for example, when traveling through a tunnel or the like, radio waves from GPS may not reach and position related information may not be input. In view of the above, the estimation unit may be configured to estimate the position-related information based on the real image captured by the imaging unit when the position-related information is not input.7). For example, the vehicle position and the traveling direction are estimated. Further, on the assumption that the vehicle is mounted on a vehicle, the position related information may be estimated based on the travel related information including the vehicle speed and the steering angle.8). These configurations enable navigation information to be complemented when used in combination with a navigation device.
[0034]
  The configuration for superimposing route guidance information has already been described. However, on the premise of such route guidance, the image synthesizing means performs actual shooting as needed when route guidance is provided by the guidance means. The guide image may be generated by synthesizing a video and a CG image.29). This is because the present invention for facilitating the association with the scenery that is actually seen exhibits an extremely excellent effect particularly in confirming the position of the vehicle during route guidance. For example, normally, a CG image (two-dimensional image or bird view) is displayed, and a live-action image is synthesized only when necessary during route guidance. The case where it is necessary includes a case where it is assumed that it is difficult for the user to recognize the guide route. For example, intersections and three-dimensional intersection roads, particularly complicated intersections, highway interchanges and junctions. Also, a live-action image may be synthesized when arriving near the destination. This is because it is easy to associate with a target building.
[0035]
  Although the above has been described as the invention of the guide image generation device, it can also be realized as a guide image display device including the above-described guide image generation device and display means for displaying the guide image.0). Further, the present invention can also be realized as an invention of a navigation device including such a guide image display device, a map storage means, a detection means, and a guide means.1). Further, in this case, it is preferable that the communication unit for performing data communication is provided, and the map information stored in the map storage unit can be updated by data communication via the communication unit.2). In addition, in the configuration including the model storage unit by data communication via the communication unit, it may be possible to update the model information stored in the model storage unit. This makes it possible to easily update the map information and model information that are out of date and to generate an appropriate guide image.
[0036]
  Note that the present invention can also be realized as a program invention that causes a computer to function as at least a CG image generation unit and an image synthesis unit of the above-described guide image generation device.3). This is because at least the viewpoint changing means, the superimposing means, the updating means, and the estimating means can be realized as computer functions.
[0037]
In the case of such a program, for example, it is recorded on a computer-readable recording medium such as FD, MO, DVD-ROM, CD-ROM, hard disk, etc., and is used by being loaded into a computer and started up as necessary. it can. In addition, such a program may be recorded on a ROM or backup RAM as a computer-readable recording medium, and the ROM or backup RAM may be incorporated into a computer.
[0038]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram illustrating an overall configuration of a navigation device 1 according to the embodiment. The navigation device 1 includes a position detector 22, a map data input device 24, an operation switch group 26, a display device 28, a speaker 30, a memory 32, a communication device 34, a video camera 36, and an electronic control unit (ECU) 38. ing
The position detector 22 includes a known gyroscope 40, a vehicle speed sensor 42, and a GPS receiver 44 for GPS (Global Positioning System) that detects the position of the vehicle based on radio waves from a satellite. Since these sensors 40, 42, and 44 have errors of different properties, they are configured to be used while being interpolated by a plurality of sensors. Depending on the accuracy, a part of the above may be used, and a geomagnetic sensor, a steering rotation sensor, a wheel sensor of each rolling wheel, or the like may be used. The position detector 22 detects position data as position related information including the current position of the vehicle and the moving direction. Therefore, the position detector 22 corresponds to “detection means”.
[0039]
The map data input unit 24 is composed of a recording medium, and inputs map data stored in the recording medium. The map data includes information such as road width, number of lanes, intersections, presence / absence of pedestrian crossings, building information, signs, destination guidance, in addition to map information for the navigation function. By the way, as a recording medium for storing map data, a DVD-ROM is generally used in recent years because of the amount of data. In this embodiment, the map data can be rewritten for updating map data as will be described later. A hard disk device is used as a recording medium. This map data input device 24 corresponds to “map information storage means”.
[0040]
The operation switch group 26 includes various switches for operating the navigation device 1 and is a configuration for a user to input various information. In the present embodiment, as various switches constituting the operation switch group 26, a touch switch configured integrally with the display device 28 and a mechanical switch different from the display device 28 are used in combination. In addition, in order to enable operation from the vehicle rear seat, a receiving unit from a remote control terminal may be provided. Further, a voice recognition function may be provided to enable voice operation.
[0041]
The display device 28 is a color display device, and the vehicle current position mark input from the position detector 22 and the planar map data input from the map data input device 24 are displayed on the display screen in an overlapping manner. Can do. In the present embodiment, in particular, a guidance image as described later can be displayed. Therefore, the display device 28 corresponds to “display means”.
[0042]
The speaker 30 is configured to notify the user of information by voice. As a result, the route guidance is provided to the user by both the display by the display device 28 and the sound output from the speaker 30.
The memory 32 stores the calculated route and the memory point on the map. In addition, model information obtained by modeling the guidance contribution information as a three-dimensional model is stored. Guidance contribution information is information that contributes to guidance in live-action images. Specifically, it stores 3D models of buildings such as buildings, pedestrian bridges, traffic lights, and 3D plane models such as roads and pedestrian crossings. ing. A three-dimensional model of a building is modeled by a rectangular parallelepiped or the like in order to reduce the amount of data. Therefore, this memory 32 corresponds to “model storing means”. In addition, camera parameters including the mounting position and angle of the video camera 36 are stored. The memory 32 can be realized as a hard disk device or a semiconductor memory device, for example.
[0043]
The communication device 34 is configured by using a mobile phone or a mobile phone as a mobile communication device, and data communication is possible with the external center 50 via the communication device 34. Thereby, the map data stored in the map data input device 24 and the model information stored in the memory 32 can be updated by a request from the user, for example. Therefore, the user can easily replace the map data and model information with the latest one. The communication device 34 corresponds to “communication means”.
[0044]
As the video camera 36, a total of three cameras are attached to the vehicle body, and image the vehicle traveling direction. Specifically, two units are attached to both end portions of the front bumper, and the remaining one unit is attached to a mounting portion of the rearview mirror. The live-action video from the video camera 36 is continuously input to the electronic control unit 38 when the guide image is generated. Therefore, the video camera 36 corresponds to “imaging means”.
[0045]
The electronic control unit 38 is configured as a normal computer, and includes a well-known CPU, ROM, RAM, I / O, and a bus line for connecting these configurations.
The navigation device 1 of the present embodiment configured as described above is characterized in that it is possible to display a guidance image using a live-action video in addition to a general route guidance function.
[0046]
Next, the operation of the electronic control unit 38 that generates this guidance image will be described with reference to the functional block diagram of FIG.
As shown in FIG. 2, the functions of the electronic control unit 38 are a model information reading block 51, a geometric information extracting block 52, a collation / correction block 53, a viewpoint changing block 54, a CG conversion block 55, an image processing block 56, an image It can be shown as a composition block 57 and a superimpose block 58. Each block 51-57 shows the function of the computer system based on a program for convenience. The functions of the blocks 51 to 58 will be described below.
[0047]
The model information reading block 51 reads a three-dimensional model as model information stored in the memory 32 based on the map data and the position data. The memory 32 stores the three-dimensional model as described above. Therefore, the vehicle position on the map is specified based on the map data and the position data, and a three-dimensional model existing in a predetermined range in the moving direction of the vehicle is read out. The three-dimensional model read out here is output to the collation / correction block 53.
[0048]
A real image from the video camera 36 is input to the geometric information extraction block 52. The geometric information extraction block 52 functions when the 3D model corresponding to the map data is not stored in the memory 32 and the 3D model cannot be read by the model information reading block 51. In this geometric information extraction block 52, first, a portion in which a change in pixel value is large in a live-action image is searched, and further, based on map data and position data, a road edge, a center line, a lane, a pedestrian crossing, a building Is extracted as geometric information. Then, the extracted geometric information is output to the collation / correction block 53.
[0049]
When the 3D model is output from the model information reading block 51, the collation / correction block 53 reads the camera parameters from the memory 32, and performs viewpoint correction according to the driver's view based on the map data and the position data. To do. In other words, since the mounting position and angle of the video camera 36 can be known from the camera parameters, when the 3D model is a CG image, the CG image is created so that the user can correspond to an actual building viewed through the windshield. The viewpoint is corrected. This is positioned as a pre-process for accurately comparing a CG image with a live-action video.
[0050]
Further, when the geometric information is output from the geometric information extraction block 52, the collation / correction block 53 reads the camera parameters from the memory 32 and corrects the geometric information based on the map data and the position data. Since the geometric information is extracted based on the live-action video, it may be impossible to extract appropriately. For example, there may be a case where a road edge is blocked by planting and cannot be partially extracted. Therefore, the collation / correction block 53 corrects the geometric information using information other than the live-action image.
[0051]
The three-dimensional model and geometric information corrected in this way are output to the viewpoint change block 54. In the following description, the three-dimensional model and geometric information are collectively referred to as three-dimensional data.
The viewpoint change block 54 receives the live-action image and the three-dimensional data from the collation / correction block 53. In the viewpoint change block 54, a viewpoint change process is performed. Such a viewpoint change is possible because a total of three video cameras 36 are provided at both ends of the front bumper of the vehicle and the rear mirror mounting portion, and three-dimensional spatial information can be obtained from a two-dimensional live-action image. Because you can. Even in a configuration having only one video camera 36, it is still possible to convert a two-dimensional real image into a three-dimensional image based on a real-time image that is continuously input from the video camera 36 while the vehicle is traveling. Space information can be obtained, and the viewpoint can be changed.
[0052]
The viewpoint change block 54 reconstructs a two-dimensional live-action picture based on the acquired three-dimensional space information to generate a picture with a changed viewpoint. Note that this video with a different viewpoint is also referred to as a live-action video for convenience. This viewpoint can be set by the user. For example, a configuration in which the user can arbitrarily set within a range in which the viewpoint can be changed may be used, or a configuration in which several viewpoints are prepared and the user can selectively set from among the viewpoints. In addition, the viewpoint change block 54 also changes the viewpoint of the input three-dimensional data according to the live-action video. The live-action video after the viewpoint change is output to the image processing block 56, and the three-dimensional data is output to the CG conversion block 55. When the viewpoint change setting is not made by the user, the viewpoint change block 54 outputs the live-action video and the three-dimensional data as they are.
[0053]
The CG conversion block 55 generates a two-dimensional CG image from the three-dimensional data. At this time, the CG conversion block 55 specifies the type of the three-dimensional data based on the map data and the position data, and generates a CG image by color coding.
The outline of a building as a three-dimensional model or a building as geometric information is color-coded according to the type of the building. For example, color coding according to functions of buildings such as buildings, pedestrian bridges, traffic lights, as well as color classification according to facility types such as hotels, restaurants, and sports centers. Also, road edges, center lines, lanes, pedestrian crossings, and the like are color-coded and converted to CG.
[0054]
Further, the CG conversion block 55 can also generate a CG image by emphasizing the outline of the three-dimensional model. Whether or not to perform such contour enhancement is determined based on user settings.
The image processing block 56 is a configuration for performing image processing on the inputted real video. Image processing includes gradation processing, color reduction processing, and region division / integration processing. It is assumed that the user can select execution of these processes from a menu or the like. In particular, when the execution of image processing is not instructed, the image processing block 56 outputs the actual video as it is. Here, it is conceivable that the gradation processing is processing for reducing contrast, for example. Further, the color reduction processing is processing for, for example, converting a live-action image into 256 colors or 16 colors, thereby removing subtle color changes. The area dividing / integrating process may be a process of grouping small areas in a live-action video into a large area by a filter operation such as cutout. That is, it is possible to perform processing for making unnecessary information that does not contribute to guidance actively inconspicuous.
[0055]
The image composition block 57 synthesizes the CG image from the CG conversion block 55 and the real shot image from the image processing block 56 to generate a guide image. The image composition block 57 performs image composition by determining a composition ratio between the CG image and the live-action video based on the setting by the user.
[0056]
Here, image composition by the image composition block 57 will be described with reference to a screen display example.
FIG. 3A is an example of a CG image generated by the CG conversion block 55 based on the three-dimensional model read from the memory 32. On the other hand, FIG. 3B is an example of a live-action video output from the image processing block 56. The real image is not subjected to the image processing as described above. At this time, the image synthesis block 57 synthesizes two images as shown in FIGS. 3A and 3B to generate the guide image shown in FIG.
[0057]
As described above, the image composition block 57 determines the composition ratio of the CG image and the live-action video based on the setting by the user. Specifically, the composition ratio is changed by changing the blending ratio of the pixel values. For example, when the composition ratio of the real image is relatively high, the result is as shown in FIG. On the other hand, when the composition ratio of the CG image is relatively high, the result is as shown in FIG.
[0058]
Furthermore, the CG conversion block 55 can generate a CG image by emphasizing the contour of the three-dimensional model. FIG. 6 shows a guide image synthesized with the CG image with the contour enhanced in this way.
By the way, in this embodiment, when the 3D model corresponding to the map data is not stored in the memory 32, the geometric information extraction block 52 uses the road edge, center line, lane, and crossing as the geometric information. The outline of the sidewalk and the building is extracted.
[0059]
FIG. 7 shows a composite image of a CG image generated from such geometric information and a live-action video. In this case, if the image processing block 56 outputs a live-action image subjected to gradation processing and reduced contrast, the composite image becomes as shown in FIG.
Returning to the description of the functional block diagram shown in FIG.
[0060]
The superimpose block 58 superimposes the guide information on the guide image output from the image composition block 57. The guidance information includes information for signs and destination guidance based on map data, operation guidance for the user, and route guidance. The image output from the superimpose block 58 is output via the display device 28.
[0061]
For example, FIG. 9 shows a state where the guidance information is superimposed on the guidance image shown in FIG.
In FIG. 9, destination guidance is displayed at the top of the screen. In this embodiment, when approaching an intersection corresponding to the destination guidance, such destination guidance is superimposed.
[0062]
In FIG. 9, an arrow for route guidance is displayed in the approximate center of the screen. Such an arrow is superimposed at the guidance timing by the route guidance function of the navigation device 1. In this embodiment, such an arrow is displayed on the road of the guide image.
[0063]
Further, in FIG. 9, an operation panel as an operation guide for the user is displayed at the lower end of the screen. On the operation panel, buttons indicated as “home”, “special memory”, “1”, “2” are displayed, which corresponds to the above-described operation switch group 26 as a touch panel. Specifically, when the user touches the screen with a finger, it is determined that the corresponding button has been pressed, and a predetermined function is realized. Such an operation panel display may be always superimposed below the guidance image, or may be superimposed only when a user operation is performed.
[0064]
The model information reading block 51 corresponds to “model information reading means”, the geometric information extraction block 52 corresponds to “geometric information extraction means”, and the collation / correction block 53 includes “geometric information correction means” and The viewpoint change block 54 corresponds to the “viewpoint change means”, the CG conversion block 55 corresponds to the “CG conversion means”, and the image processing block 56 and the image composition block 57 correspond to “model position matching means”. The superimposing block 58 corresponds to “superimposing means”. The blocks 51, 52, 53, 55, 56, and 57 correspond to “CG image generating means”.
[0065]
Next, the effect which the navigation apparatus 1 of a present Example exhibits is demonstrated.
In the navigation device 1 of the present embodiment, as shown in FIG. 2, the model information corresponding to the map data is read by the model information reading block 51, or if there is no model information corresponding to the map data, the geometric information Geometric information is extracted from the live-action video by the extraction block 52. Then, these three-dimensional model or geometric information is generated as a CG image by the CG block 55 and is combined by the image combining block 57 to generate a guide image.
[0066]
When a CG image as shown in FIG. 3A is used as a guide image as it is, a building or the like is shown in a simplified manner, so that an actually visible scene such as a live-action image shown in FIG. Is difficult to associate. On the other hand, when a live-action image as shown in FIG. 3B is directly used, on the contrary, information that is not necessary for guidance increases, and there is a high possibility that appropriate guidance is hindered. For example, planting of a median strip (center line) indicated by symbol A and street trees indicated by symbol B are necessary information in terms of association with actual scenery, but are not necessary in the sense of providing guidance. Information.
[0067]
Therefore, in this embodiment, the guide images as shown in FIG. 4 are generated by combining the images. Since this guide image is based on a live-action video, it can be easily associated with an actual scene. On the other hand, the composition of the CG image makes it easy to recognize the building and road that are the guidance contribution information, and can be distinguished from unnecessary information that does not contribute to the guidance. In particular, in the guidance image shown in FIG. 4, unnecessary information is not noticeable due to the synthesis of the CG image. For example, it can be seen that the planting indicated by the symbol C and the roadside tree indicated by the symbol D are not noticeable. Thereby, appropriate guidance can be realized.
[0068]
In the present embodiment, the composition ratio between the CG image and the live-action video can be changed by the image composition block 57. Thereby, a guide image close to a real image as shown in FIG. 5A is generated, or a guide image close to a CG image based on a three-dimensional model as shown in FIG. 5B is generated. You can also. In the former case, the effect of facilitating the association with the actual scenery becomes relatively large, and in the latter case, the effect that unnecessary information that impedes appropriate guidance becomes inconspicuous becomes relatively large.
[0069]
By the way, in this embodiment, when the three-dimensional model corresponding to the map data is not stored in the memory 32, the geometric information is extracted from the photographed video and the CG image is generated. A guide image obtained by synthesizing the CG image based on the geometric information is as shown in FIG. Also in this case, the outline of the building, road edges, lanes, center lines, and pedestrian crossings are synthesized as CG images, so it is easy to recognize the guidance contribution information in the live-action video, and the correspondence with the actually visible scenery Attaching is facilitated and appropriate guidance is realized.
[0070]
As described above, since the CG image can be created also by extracting the geometric information, appropriate guidance is provided even when traveling in an area where a three-dimensional model is not prepared. In other words, it is not necessary to store the three-dimensional models in various parts of the country in the memory 32. Therefore, the data amount can be reduced. Furthermore, in this embodiment, since the building is configured with a simple model using a rectangular parallelepiped or the like, the data amount can be reduced in this respect as well.
[0071]
In the present embodiment, the image processing block 56 performs image processing on the live-action video according to user settings. This image processing is processing for making unnecessary information that does not contribute to guidance inconspicuous, and it is possible to positively eliminate unnecessary information that appears in a live-action video. For example, FIG. 7 will be described as an example. By reducing the contrast of a live-action image to be synthesized, unnecessary information becomes inconspicuous and a CG image based on the extracted geometric information becomes clear as shown in FIG. Therefore, appropriate guidance is possible.
[0072]
Furthermore, in the present embodiment, the CG conversion block 55 performs CG by color-coding the guidance contribution information. As a result, the guidance contribution information in the live-action video can be easily recognized, and a plurality of guidance contribution information can be easily distinguished and recognized. Specifically, because the colors of the buildings, pedestrian bridges, traffic lights, etc., are classified, it is possible to clearly recognize the landmark building during traveling. In addition, since color classification is performed according to the type of facility such as a hotel, a restaurant, or a sports center, the user can search for the target facility on the guidance screen, and can instantly grasp which building is actually the target building. Furthermore, since the road edges, center lines, lanes, pedestrian crossings, and the like are also color-coded, the center lines, lanes, pedestrian crossings, etc. can be recognized instantaneously while driving, which can contribute to safe driving. Further, when performing CG based on the three-dimensional model, the CG conversion block 55 can also generate a CG image by emphasizing the outline of the three-dimensional model. In this case, for example, the outline of a building as a landmark is emphasized (see FIG. 6), and more appropriate guidance can be realized.
[0073]
Further, in this embodiment, the collation / correction block 53 performs viewpoint correction on the three-dimensional model to create a CG image. As a result, the CG image based on the three-dimensional model can correspond to the driver's view, and the correspondence with the actual scenery of the CG image, that is, the matching with the live-action video is accurate. The possibility that the association can be performed instantaneously increases. In addition, the collation / correction block 53 appropriately corrects the extracted geometric information when, for example, a part of the geometric information is hidden and not extracted. As a result, the outline of a building can be reliably extracted even from a live-action image, and appropriate guidance can be realized.
[0074]
Furthermore, in this embodiment, the viewpoint change block 54 can generate not only the guide image corresponding to the driver's view but also a guide image from another viewpoint. Therefore, for example, when a right or left turn road is grasped in an urban area with many intersections, a guide image corresponding to a bird's-eye view can be displayed, which is convenient for the user.
[0075]
In this embodiment, the superimpose block 58 superimposes guidance information on the guidance image. Specifically, as shown in FIG. 9, arrows for destination guidance and route guidance, and an operation panel are superimposed.
The sign and the destination guidance are superimposed when approaching the sign or approaching the intersection corresponding to the destination guidance, so that the correspondence with the actual road can be easily grasped.
[0076]
The arrow for route guidance is superimposed on the road of the guidance image at the guidance timing by the route guidance function. Therefore, the effectiveness of route guidance is ensured.
Furthermore, since the operation panel is superimposed on the lower end of the screen, guidance is not hindered. This is because there is no important information at the lower end of the guidance screen indicating the portion closest to the vehicle.
[0077]
As described above, the present invention is not limited to such embodiments, and can be implemented in various forms without departing from the spirit of the present invention.
(A) In the above embodiment, when there is no 3D model corresponding to the map data, the geometry information extraction block 52 is configured to extract the geometry information from the live-action video, but there is a 3D model. Alternatively, the geometric information may be extracted, and the CG block 55 may generate a CG image from both the three-dimensional model and the geometric information. In particular, when the three-dimensional model is configured as a simple model as in the present embodiment, the contour of the CG image is emphasized and becomes an appropriate guide image by extracting the geometric information and converting it to CG. It is.
[0078]
Of course, as a configuration not including the geometric information extraction block 52, a configuration in which a CG image is generated from only a three-dimensional model may be employed. On the other hand, as a configuration not including the model information reading block 51, a configuration in which a CG image is generated only from geometric information can be employed.
[0079]
(B) In the above embodiment, an arrow for route guidance is displayed on the road of the guidance image. At this time, it is good to superimpose in association with the road so that the user can recognize straight turn, right turn, and left turn at the intersection. For example, as shown in FIG. 10A, in the case of indicating straight ahead, it is conceivable to display an arrow (indicated by symbol Y1) that crosses an intersection. For example, as shown in FIG. 10B, when a left turn is indicated, it is conceivable to display an arrow (indicated by symbol Y2) that bends at an intersection. In this way, more appropriate route guidance becomes possible.
[0080]
In the above embodiment, the destination guidance is superimposed (see FIG. 9). It is conceivable that such destination guidance display is also performed in correspondence with the intersection as in the above-described arrow display for route guidance. For example, as shown in FIG. In this way, reliable destination guidance is provided.
[0081]
(C) By the way, it has already been described that the map data of the map data input device 24 can be updated by data communication with the center 50 via the communication device 34 at the request of the user.
However, if such updates are not performed and the map data becomes old, it is possible that buildings and roads that are not in the map data will be reflected in the live-action video, or that buildings and roads that are in the map data will not be reflected. It is done. At this time, a live-action image may be preferentially used to generate a CG image. This is because the guidance image corresponding to the scenery that can be actually seen becomes easy to associate with the scenery. Further, in such a case, the map data of the map data input device 24 may be automatically updated based on the photographed video. In this case, the electronic control unit 38 corresponds to “update means”.
[0082]
(D) Further, there may be a case where the GPS receiver 44 of the position detector 22 cannot receive a signal from a satellite, such as when a vehicle travels through a tunnel. In view of this, it is conceivable to adopt a configuration in which the above-described position data is estimated based on a live-action video. Alternatively, the vehicle speed from the vehicle speed sensor 42 and a configuration including a steering sensor may be used, and the position data may be estimated using the steering angle from the steering sensor. Thereby, the navigation information is interpolated, and as a result, appropriate guidance can be performed. In this case, the electronic control unit 38 corresponds to “estimating means”.
[0083]
(E) In the above embodiment, in addition to the general route guidance function, it is possible to display a guidance image using a live-action video. The guidance image display using the photographed video may be performed as necessary when route guidance is performed by the navigation device 1. This is because the guidance image display using the live-action video that facilitates the association with the scenery that is actually seen exhibits an extremely excellent effect particularly in confirming the position of the vehicle during route guidance. For example, it is conceivable to adopt a configuration in which a CG image (a two-dimensional image or a bird view) is displayed during normal driving and a live-action image is synthesized only when necessary during route guidance. The case where it is necessary includes a case where it is assumed that it is difficult for the user to recognize the guide route. For example, intersections and three-dimensional intersection roads, particularly complicated intersections, highway interchanges and junctions. Also, a live-action image may be synthesized when arriving near the destination. This is because it is easy to associate with a target building.
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating an overall configuration of a navigation device according to an embodiment.
FIG. 2 is a functional block diagram of an electronic control device.
3A is an explanatory diagram showing a CG image based on a three-dimensional model, and FIG. 3B is an explanatory diagram showing a live-action video.
FIG. 4 is an explanatory diagram showing a guide image generated using a CG image based on a three-dimensional model.
FIG. 5 is an explanatory diagram showing a guide image when the composition ratio is changed.
FIG. 6 is an explanatory diagram showing a guide image when an outline of a CG image based on a three-dimensional model is emphasized.
FIG. 7 is an explanatory diagram showing a guide image generated using a CG image based on geometric information.
FIG. 8 is an explanatory diagram showing a guide image generated by performing gradation processing on a live-action image.
FIG. 9 is an explanatory diagram showing a guide image obtained by superimposing guide information.
FIG. 10 is an explanatory diagram showing guidance information display corresponding to an intersection.
[Explanation of symbols]
1 ... Navigation device
22 ... Position detector
24 ... Map data input device
26 ... Operation switch group
28 ... Display device
30 ... Speaker
32 ... Memory
34. Communication device
36 ... Video camera
38 ... Electronic control unit
40 ... Gyroscope
42 ... Vehicle speed sensor
44 ... GPS receiver
50 ... Center
51 ... Model information reading block
52 ... Geometric information extraction block
53. Verification / correction block
54 ... Viewpoint change block
55 ... CG block
56. Image processing block
57. Image composition block
58 ... Superimpose block

Claims (33)

An imaging means for capturing a live-action image;
CG image generation means for generating, as a CG image, guidance contribution information that contributes to guidance in information in a live-action video;
Image synthesizing means for synthesizing a live-action image captured by the imaging means and a CG image generated by the CG image generating means to generate a guide image;
The CG image generating means
From the captured photographed image by the imaging means, the geometrical information extraction means for extracting geometrical information as the contour of a region corresponding to the guidance contribution information,
Model storage means for storing model information obtained by modeling the guidance contribution information as a three-dimensional model;
Model information reading means for reading the model information from the model storage means;
Geometric information extracted by said geometrical information extraction unit, and a feature that turned into CG model information read by the model information reading means and a CG means for generating the CG image A guide image generating device. In the guidance image generation device according to claim 1 ,
The geometric information extraction unit extracts the geometric information based on a storage state of the model information stored in the model storage unit. In the guidance image generation device according to claim 1 or 2 ,
The model information reading means reads the model information based on map information and position related information. In the guidance image generation device according to any one of claims 1 to 3 ,
The guide image generating apparatus characterized in that the CG converting means color-codes the model information based on map information and position related information. In the guidance image generation device according to any one of claims 1 to 4 ,
The guide image generating apparatus, wherein the model information is building model information obtained by modeling a building. In the guidance image generation device according to claim 5 ,
The building model information is a representation of the building using a rectangular parallelepiped. In the guidance image generation device according to claim 5 or 6 ,
The guide image generating apparatus, wherein the CG conversion unit converts the building model information into a CG by color-coding according to the type of building. In the guidance image generation device according to any one of claims 5 to 7 ,
The said CG conversion means emphasizes the outline of a building based on the said building model information, and converts it into CG. The guide image generation apparatus characterized by the above-mentioned. In the guidance image generation device according to any one of claims 1 to 8 ,
The CG image generating means
Model position matching means for correcting a viewpoint for converting the model information into CG,
The guide image generating apparatus characterized in that the CG converting means converts the model information to CG from the viewpoint corrected by the model position matching means. In the guidance image generation device according to claim 9 ,
The model position collating unit corrects the viewpoint for the CG based on the imaging condition. In the guidance image generation device according to any one of claims 1 to 10 ,
Further, a guide image generating apparatus comprising a viewpoint changing means for changing the viewpoint and generating the guide image. In the guidance image generation device according to any one of claims 1 to 11,
The guide image generating apparatus, wherein the image synthesizing unit is capable of changing a synthesis ratio of the photographed video and the CG image. In the guidance image generation device according to any one of claims 1 to 12 ,
The said image synthetic | combination means synthesize | combines with the said CG image, after performing an image process with respect to the said photographed image. In the guide image generating apparatus according to claim 1 3,
The guide image generating apparatus, wherein the image processing is gradation processing. In the guide image generating apparatus according to claim 1 3 or 1 4,
The guide image generating device, wherein the image processing is a color reduction processing. In the guide image generating apparatus according to any one of claims 1 3 to 1 5,
The guide image generating apparatus, wherein the image processing is region integration / division processing in a live-action video. In the guidance image generation device according to any one of claims 1 to 16 ,
Furthermore, a guide image generating device characterized by comprising superimposing means for superimposing guide information in the guide image. In the guidance image generation device according to claim 17 ,
The guidance information generating apparatus characterized in that the guidance information includes information on signs and destination guidance on a road. The guide image generating device according to claim 18 ,
A guide image generating apparatus characterized by superimposing the sign and destination information at an appropriate timing based on map information and position related information. The guide image generation device according to claim 18 or 19 ,
The guide image generating apparatus characterized in that the information on the sign and the destination guidance is superimposed on the road in the guide image in association with the road. In the guide image generating apparatus according to claim 1 7-2 0,
The guidance information generating apparatus characterized in that the guidance information includes information on operation guidance for a user. In the guide image generating apparatus according to claim 2 1,
A guide image generating apparatus characterized in that the operation guide information is superimposed on the lower part of the screen. In the guide image generating apparatus according to claim 1 7-2 2,
The guidance information generating apparatus characterized in that the guidance information includes information on route guidance by guidance means. In the guide image generating apparatus according to claim 2 3,
A guide image generating apparatus characterized in that the route guidance information is superimposed at a guide timing by a guide means. In the guide image generating apparatus according to claim 2 3 or 2 4,
The route guidance information includes an arrow image indicating the direction of the guidance route, and the arrow image is superimposed on the road in the guidance image. The guide image generating device according to claim 25 ,
A guide image generating apparatus, wherein the arrow image is superimposed on a road so that the user can recognize straight, right and left turns at an intersection. In the guidance image generation device according to any one of claims 1 to 26 ,
Furthermore, when position related information is not input, it is provided with the estimation means which estimates position related information based on the real image image imaged by the said imaging means. The guide image generation apparatus characterized by these. In the guide image generating apparatus according to claim 2 7,
Assuming that this device is installed in a vehicle,
The estimation unit estimates the position related information based on travel related information including at least a vehicle speed or a steering angle. In the guide image generating apparatus according to any one of claims 1-2 8,
The image synthesizing unit generates the guide image by synthesizing the live-action video and the CG image as necessary when route guidance is performed by the guidance unit. . A guide image generating device according to any one of claims 1 to 29 ;
A guide image display device comprising: display means for displaying the guide image. A guidance image display apparatus according to claim 3 0,
Map storage means for storing map information;
Detection means for detecting position-related information including at least information for specifying the current position;
A navigation device comprising: guidance means for generating a route to a set destination and performing route guidance. The navigation apparatus according to claim 3 1,
Furthermore, a communication means for performing data communication is provided,
The navigation apparatus characterized in that the map information stored in the map storage means can be updated by data communication via the communication means. A program that causes a computer to function as at least a CG image generation unit and an image synthesis unit of the guide image generation device according to any one of claims 1 to 29 .

Images