JP6424100B2 - NAVIGATION SYSTEM, NAVIGATION DEVICE, GLASS TYPE DEVICE, AND METHOD FOR LINKING THE DEVICE - Google Patents

Description

  The present invention relates to a system including a navigation device and a glass device, and an apparatus and method used in the system.

  Patent Document 1 discloses an invention related to a navigation device provided with a glasses-type display device. The navigation device detects a user's position with a GPS receiver, and recognizes an object at a predetermined position on the user's line of sight with a line-of-sight detection sensor. Then, additional information related to the position detected by the GPS receiver and the object recognized by the line-of-sight detection sensor is displayed on the eyeglass-type display device. In this way, displaying additional information on the glasses-type display device can display additional information about the recognized object in the vicinity of the recognized object existing ahead of the user's line of sight. The necessary information can be provided in an easy-to-understand manner.

JP 2005-037181 A

  In the navigation device described in Patent Document 1 described above, although the eyeglass-type display device is provided with a line-of-sight detection sensor, it is mainly used as a display device. For this reason, it is the structure controlled by the navigation apparatus. In recent years, glass (glasses) -type devices called glass-type smart devices have been provided. Recent glass-type devices, for example, collaborate with high-function mobile phone terminals called smartphones, etc. to display personal attribute information by recognizing faces, and by displaying translation results by recognizing characters. Functions can be realized.

  Glass-type devices are still regarded as important as a display device, but some glass-type devices are equipped with a CPU (Central Processing Unit) and can perform complicated discrimination processing. For this reason, it is desired to realize a more convenient navigation device by cooperating with the navigation device.

  In view of the above, an object of the present invention is to seamlessly link a glass-type device and a navigation apparatus to realize an unprecedented function.

In order to solve the above problem, the navigation system of the invention according to claim 1 provides:
A navigation system formed by a navigation device and a glass-type device,
The navigation device
Navigation-side communication means for communicating with the glass-type device;
Instructing to generate message data for the user of the glass-type device and to output guidance information corresponding to the message data in a mode other than voice output when a predetermined event occurs, at least according to the event output instruction to both, and the navigation-side command control means for controlling to transmit to said glass-type device through the navigation device side communication means,
After transmitting the message data and the output instruction, when receiving a photographed video from the glass-type device through the navigation side communication means, video processing means for processing the photographed video;
Using a processing result in the video processing means, and a process control means for controlling to execute a desired process ,
The glass-type device is
A glass side communication means for communicating with the navigation device;
Imaging means;
Glass that controls to output guidance information according to the message data in a mode other than voice output according to the output instruction when the output instruction and the message data are received through the glass side communication means Side instruction control means ;
And providing control means for controlling to transmit a captured image obtained by imaging through the imaging means to the navigation device through the glass side communication means .

  According to the present invention, the glass-type device and the navigation device can be seamlessly linked to realize an unprecedented function.

It is a figure for demonstrating the whole structure of the navigation system of embodiment. 2 is a block diagram for explaining a configuration example of a navigation device 1. FIG. 4 is a block diagram for explaining a configuration example of a control mechanism unit 21 provided in an ear hooking part (temple part) of the glass device 2. FIG. 6 is a diagram illustrating an example of an output message prepared in the storage device 103. FIG. It is a figure which shows the example of the three-dimensional image of an intersection. 4 is a flowchart for explaining navigation processing executed by a control unit 102 of the navigation device 1. It is a flowchart for demonstrating the navigation cooperation process performed with the glass-type device 2 at the time of a navigation process. 4 is a diagram illustrating an example of an output message displayed on a lens unit of the glass device 2. FIG. 4 is a diagram illustrating an example of an output message displayed on a lens unit of the glass device 2. FIG. It is a flowchart for demonstrating the route confirmation process mainly performed by the cooperation control part 102C of the navigation apparatus 1 at the time of a navigation process. It is a figure which shows the example of the message output through the glass-type device 2 at the time of a route confirmation process. It is a figure which shows the example of the operation | movement which the user of the glass-type device 2 performs at the time of a route confirmation process, and the example of the message output through the glass-type device 2. FIG. It is a flowchart for demonstrating the information sharing process mainly performed by the cooperation control part 102C of the navigation apparatus 1 at the time of a navigation process. It is a figure which shows the example of the message output through the glass-type device 2 at the time of an information sharing process. It is a flowchart for demonstrating the information update process mainly performed by the cooperation control part 102C of the navigation apparatus 1 at the time of a navigation process. It is a figure which shows the example of the message output through the glass-type device 2 at the time of an information update process. It is a flowchart for demonstrating the process performed by the cooperation control part 102C when it becomes GPS lost in the navigation apparatus 1. FIG. It is a flowchart following FIG. FIG. 4 is a diagram illustrating an example of a display message displayed on the lens unit of the glass-type device 2 and a scene seen through the lens unit of the glass-type device 2.

  Hereinafter, an embodiment of a system, apparatus, and method of the present invention will be described with reference to the drawings.

[Overall configuration of navigation system]
FIG. 1 is a diagram for explaining the overall configuration of the navigation system of this embodiment. As shown in FIG. 1, the navigation system of this embodiment includes a navigation device 1 and a glass smart device (hereinafter referred to as a glass device) 2. In this embodiment, the navigation apparatus 1 and the glass-type device 2 can communicate with each other by short-range wireless communication.

  The navigation device 1 is realized, for example, by mounting navigation application software, map data, network data, etc. on an information processing device called a high-function mobile phone terminal called a smartphone or a tablet PC (Personal Computer). The navigation apparatus 1 can be used as a navigation apparatus for pedestrians, a navigation apparatus for bicycles, and a navigation apparatus for vehicles such as motorcycles and automobiles.

  The navigation device 1 includes a GPS unit, a display unit, an operation unit, a voice processing unit, and the like, as in the case of a general navigation device. Then, the navigation device 1 searches for a route according to an instruction input from the user and presents it to the user, sets a route selected according to the user's instruction, and displays a map display and a voice message. A function to perform route guidance is realized.

  The appearance of the glass-type device 2 is glass (glasses), but the ear hooking portion (temple portion) includes a control unit, a switch 204A, a camera unit 209, a display processing unit and a projection unit 206, a bone conduction vibration unit 208, and the like. The control mechanism unit 21 is configured. The glass-type device 2 includes a control unit, so that the control unit can execute application software and operate in cooperation with the navigation apparatus 1.

  In this embodiment, the switch 204 </ b> A is operated when a request is transmitted from the glass device 2 to the navigation apparatus 1. Specifically, the switch 204 </ b> A is operated when a route confirmation is requested when navigation processing is being performed in the navigation device 1, or when information sharing processing or information updating processing described later is executed in detail. Is done. The projection unit 206 displays various display information on, for example, a specially coated lens portion. Thereby, the user wearing the glass device 2 can visually recognize the display information displayed on the lens unit.

  Moreover, in the glass type device 2 of this embodiment, in order to simplify the description, the control mechanism unit 21 has an ear hooking portion (temple portion) on the left side (the right side of the user when worn by the user). It is the composition provided in. However, it is not limited to this. For example, it is also possible to display the display information in a range that the user can visually recognize with both eyes by providing the projection unit 206 on both the left and right ear hooks and displaying the display information on both the left and right lenses.

  Similarly, the camera unit 209 can be provided in both the left and right ear hooks, and an image equivalent to the image visually recognized by the user with both eyes can be formed from the captured images taken by both the left and right camera units. It is also possible to provide the bone conduction vibration unit 208 in both the left and right ear hooks so that both the left and right bone conduction vibration units act to transmit information to the user. Of course, the switch 204A may be provided in both the left and right ear hooks, and any switch 204A may be operated.

  And the navigation apparatus 1 and the glass-type device 2 cooperate (cooperate) and cooperate, and implement | achieve the function which is not in the past. There are the following three linkage patterns. (1) A pattern in which information or the like is provided from the navigation device 1 to the glass device 2 and cooperates, (2) A pattern in which information or the like is provided from the glass device 2 to the navigation device 1 and cooperates, (3) Navigation This is a pattern in which the apparatus 1 and the glass-type device 2 provide information and the like in a bidirectional manner and perform a cooperative operation.

  For this reason, each of the navigation apparatus 1 and the glass-type device 2 includes a control unit including a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. Application software can be executed and linked to each other. Below, after demonstrating the structural example of the navigation apparatus 1 and the glass-type device 2 which comprise the navigation system of this embodiment, the operation | movement (process) performed by each is demonstrated concretely.

[Configuration Example of Navigation Device 1]
FIG. 2 is a block diagram for explaining a configuration example of the navigation device 1 used in the navigation system of this embodiment. In FIG. 1, a short-range transmission / reception antenna 101 </ b> A and a short-range communication unit 101 are for communicating with the glass-type device 2. For this reason, in the near field communication part 101, the same communication system as the near field communication part provided in the glass type device 2 side mentioned later is used. For example, various communication methods such as a Bluetooth (registered trademark) standard that can maintain a communication distance of at least several meters can be used.

  The control unit 102 realizes a function of controlling each unit of the navigation device 1. The control unit 102 also functions as a provision control unit 102A, a process control unit 102B, and a cooperation control unit 102C. The provision control unit 102 </ b> A functions when the above-described (1) navigation apparatus 1 provides information to the glass-type device 2 and performs a cooperative operation. Further, the process control unit 102B functions when the above-described (2) the glass-type device 2 provides information to the navigation device 1 and performs a cooperative operation. Further, the cooperation control unit 102C functions when (3) the navigation apparatus 1 and the glass device 2 described above provide information and the like bidirectionally to perform a cooperative operation.

  The storage device 103 implements an information storage holding function. The storage device 103 stores various programs, various data necessary for processing, output messages, and the like. The storage device 103 is also used as a work area for temporarily storing intermediate results of various processes. FIG. 4 is a diagram for explaining an example of an output message stored and held in a predetermined storage area of the storage device 103. As shown in FIG. 4, the storage device 103 stores various messages such as various guidance messages used in the navigation process and a message for notifying the analysis result of the captured video from the glass-type device 2 as will be described later. Is retained. The operation unit 104 includes a power on / off switch, several function buttons, and the like.

  As shown in FIG. 2, the navigation device 1 includes a map DB 105, a network DB 106, a 3D image DB 107, a timetable DB 108, and a POI (point of interest) DB. “DB” is an abbreviation for database. The map DB 105 stores and holds map data for displaying a map used for navigation processing. The network DB 106 stores and holds node data including a node ID, latitude / longitude, and node type, and link data including a link ID, node IDs at both ends of the link, link cost, link type, and the like. The network data is used in route search processing. The map DB 105 and the network DB 106 store map data and network data according to the purpose (purpose) such as for pedestrians, bicycles, and automobiles.

  The 3D image DB 107 stores and holds three-dimensional image data for rendering a real world state as a three-dimensional image and a position (latitude, longitude) in the real world in association with each other. FIG. 5 is a diagram illustrating an example of a three-dimensional image of an intersection existing in the real world. In the 3D image DB 107, 3D image data for rendering a 3D image as shown in FIG. 5 is stored and held corresponding to the position (latitude, longitude) in the real world. The stored data of the 3D image DB 107 is used for easily showing the route at an intersection that is difficult to understand during navigation processing.

  The stored data of the 3D image DB 107 is obtained by matching the captured video of the intersection existing on the user's moving route with the 3D image data of the intersection stored and held in the 3D image DB 107, so that the moving route is determined. It is also used when determining whether or not it is appropriate. The stored data in the 3D image DB 107 is also used when detecting changes in road signs, intersections, signal installation conditions, cityscapes, landmarks, and the like by comparing with captured images from the glass-type device 2.

  The timetable DB 108 stores and holds information that summarizes operation times of public transportation such as railways, route buses, air routes, and regular routes. Thereby, for example, in the case of a railway, if the boarding station and the boarding time are separated, the arrival time at the target getting-off station (target station) can be known. In addition, when transfer is necessary, it is possible to grasp the transfer station, transfer platform, departure time of the train to be transferred, and the like.

  The POIDB 109 stores and holds base information, which is information about various bases such as various stores and various facilities specified by latitude and longitude, or feature points on the topography. For example, if there are bases such as landmark buildings or various facilities on the map, detailed information such as what kind of base the base is and what features it has is arranged Yes.

  The sensor unit 110 includes various sensors such as a six-axis sensor and a geomagnetic sensor (orientation sensor) in order to realize an autonomous navigation function. The time information providing unit 111 provides information related to time such as the current date, current day, and current time. The GPS unit 112 and the GPS antenna 112A are parts that realize a function of accurately acquiring (positioning) the current position of the own device by receiving and analyzing transmission signals (positioning information) from a plurality of artificial satellites. .

  The touch panel 115 includes a display unit 113 such as an LCD (Liquid Crystal Display) and a touch sensor 114, and realizes a function as an information display unit and a function as an information input unit. When the navigation function is executed, a guidance map is displayed on the display unit 113, the route obtained as a result of the route search and the current position of the own device are displayed on the map, and route guidance (navigation). Is done.

  The voice processing unit 116 and the speaker 117 or the voice output terminal 118 realize a function of emitting an alarm sound or a voice message according to the control of the control unit 202. An earphone is connected to the audio output terminal 118. For example, when the navigation function is executed, voice message data is supplied to the voice processing unit 116 through the control unit 102, converted into an analog voice signal, and a voice message is received from the earphone connected to the speaker 117 or the voice output terminal 118. Sound is emitted.

  When an earphone is connected to the audio output terminal 118, the audio processing unit 116 supplies an audio signal only to the earphone via the audio output terminal 118 without supplying an audio signal to the speaker 117. For this reason, the audio processing unit 116 can detect whether an earphone is connected to the audio output terminal, and also notifies the control unit 102 of the detection result.

  When the navigation function is executed, the route search unit 119 performs route search, and performs processing for displaying the result on the display unit 113 using the map data of the map DB 105. Specifically, the route search unit 119 displays on the display unit 113 a screen display for receiving information necessary for route search such as a departure place, a waypoint, and a destination. Then, the route search unit 119 receives necessary information such as a departure point, a stopover point, and a destination via the touch panel 115, refers to the stored data in the network DB 106 based on the information, and performs a route search. Is displayed on the display unit 113 using the map data of the map DB 105.

  When the user confirms the display and wants to change the searched route, the user performs an operation for performing the route search again through the touch panel 115. In this case, the route search unit 119 functions and the route search process is performed again. In addition, when the user confirms the display and traces the searched route, the user performs an operation of starting navigation to follow the route through the touch panel 115.

  Thereafter, the control unit 102 causes the route guide unit 120 to function, displays a map of the area including the current position on the display unit 113, and displays the route obtained by the route search on the map and the current position of the own device. Display and navigate. In this case, the route guide unit 120 displays a map of an area including the current position of the own device on the display unit 113 by map data read from the map DB 105. Then, the route guidance unit 120 performs navigation so that the own device can follow the route set on the network according to the network data of the network DB 106.

  The 3D image comparison unit 121 performs a matching process between the captured video of the intersection existing on the moving route of the user provided from the glass device 2 and the 3D image data of the intersection stored in the 3D image DB 107. The result of the matching process is notified to the control unit 102. As a result, in the control unit 102, the processing control unit 102B functions to determine whether or not the set route has deviated, and when it is deviated, the route search unit 119 is controlled to perform route search. And correct the route.

  The image recognizing unit 122 receives provision of a photographed image photographed through the camera unit of the glass device 2, analyzes this, and notifies the control unit 102 of the analysis result. For example, the image recognizing unit 122 recognizes information such as a place name, a station name, and a building name included in the captured video and notifies the control unit 102 of this information, congestion of a moving route (road), weather, accident, It recognizes the construction status and notifies the control unit 102 of this. As a result, in the control unit 102, the processing control unit 102B functions to determine whether or not the route has been deviated during execution of pedestrian navigation or car navigation, and performs reroute processing as necessary or execution of train navigation. Sometimes it is possible to correct the arrival time and prevent overpassing. It is also possible to post the status of the moving route to a predetermined server and share information with other drivers.

  Also, the POI comparison unit 123 receives provision of a photographed image photographed through the camera unit of the glass-type device 2, and automatically compares the photographed image with a 3D image of a portion corresponding to the photographed image in the 3D image DB 107. . Then, if there is a different location, the POI comparison unit 123 updates the update information in which position information (latitude, longitude) indicating the different location and information indicating the image recognition result of the location are associated with each other. Formed and recorded in the storage device 103. For example, the update information can be transmitted to a predetermined server device and used for update processing of 3D image data.

  Although the POI comparison unit 123 has been described as comparing the captured video from the glass-type device 2 with the 3D image data of the 3D image DB 107, the present invention is not limited to this. The POI comparing unit 123 recognizes (identifies) various POIs and their location (latitude, longitude) existing in the photographed video by analyzing the photographed video from the glass-type device 2 and identifies them. The result and the stored data of POIDB 109 may be matched. Even in this way, it is possible to identify a POI that has newly existed or a POI that no longer exists, and the identification result can be used as data indicating the difference between the two.

  In addition, the transmission / reception antenna 124A and the wireless communication unit 124 realize a communication function through a mobile phone network or the Internet. Thereby, for example, the control unit 202 accesses a predetermined map server on the Internet in response to an instruction from the user, and receives provision of the latest map data, the latest network data, the latest 3D image data, and the like. It is done. The control unit 202 can store and use the latest map data and the latest network data provided in the map DB 105, the network DB 106, and the 3D image DB 107. The navigation device 1 can also transmit various information acquired by the navigation device 1 to a target server or the like through the wireless communication unit 124 and the transmission / reception antenna 124A.

[Configuration Example of Control Mechanism 21 of Glass Device 2]
FIG. 3 is a block diagram for explaining a configuration example of the control mechanism unit 21 provided in the ear hooking portion (temple portion) of the glass-type device 2 having the appearance shown in FIG. In FIG. 3, the short-range transmission / reception antenna 201 </ b> A and the short-range communication unit 201 are for communicating with the navigation device 1. For this reason, the near field communication unit 201 has the same communication method as the near field communication unit 101 of the navigation device 1 described above.

  The control unit 202 realizes a function of controlling each unit of the glass device 2. The control unit 202 also functions as a process control unit 202A, a provision control unit 202B, and a cooperation control unit 202C. The process control unit 202A functions when the above-described (1) navigation apparatus 1 provides information to the glass-type device 2 and performs a cooperative operation. The provision control unit 202B functions in the case where the above-described (2) the glass-type device 2 provides information to the navigation device 1 and performs a cooperative operation. Also, the cooperation control unit 202C functions when the above-described (3) navigation apparatus 1 and the glass-type device 2 provide information or the like bidirectionally to perform a cooperative operation.

  The storage device 203 implements an information storage holding function. The storage device 203 stores various programs and various data necessary for processing. The storage device 203 is also used as a work area for temporarily storing intermediate results of various processes. The operation unit 204 includes a power on / off switch, the above-described switch 204A, several function buttons, and the like.

  Under the control of the control unit 202, the display processing unit 205 forms a display signal to be supplied to the projection unit 206 from the display data provided thereto, and supplies this to the projection unit 206. The projection unit 206 performs a process of projecting display information corresponding to the display signal from the display processing unit 205 onto the lens portion of the glass device 2. Thereby, display information is displayed on the lens part of the glass-type device 2, and a user who wears the glass-type device 2 on the head so that the lens part is positioned in front of the eyes can be seen.

  The bone conduction drive unit 207 and the bone conduction vibration unit 208 realize a bone conduction function. Bone conduction is transmitted through the air to vibrate the eardrum (middle ear) and transmitted to the auditory nerve (inner ear) (air conduction sound), whereas for example, vibration of the vocal cords is transmitted through the skull and directly to the auditory nerve (bone conduction sound ) For this reason, the bone conduction drive unit 207 forms a drive signal to be supplied to the bone conduction vibration unit 208 from the display data and audio data which are output data supplied to the bone conduction drive unit 207 under the control of the control unit 202. Is supplied to the bone conduction vibration unit 208. The bone conduction vibration unit 208 vibrates according to the drive signal from the bone conduction drive unit 207. As a result, the vibration according to the output data, which is display data and audio data, is transmitted directly to the auditory nerve via the skull, and the content according to the output data is the same as when listening to the emitted sound. It is recognized by the user as a bone conduction sound.

  As shown in FIG. 1, the camera unit 209 is provided at the front end portion (end portion on the lens unit side) of the ear hooking portion (temple portion) of the glass device 2. Then, when the user wears the glass device 2 on the head so that the lens portion is located in front of the eyes, the camera unit 209 captures an image in the user's line-of-sight direction and captures it as video data. Realize the function. And the picked-up image | video (video data) image | photographed through the camera part 209 can be transmitted to the navigation apparatus 1 through the near field communication part 201 and the near field transmission / reception antenna 201A.

  The GPS unit 210 and the GPS antenna 210A are parts that realize a function of accurately detecting (positioning) the current position of the own device by receiving and analyzing transmission signals (positioning information) from a plurality of artificial satellites. . The sensor unit 211 includes at least a geomagnetic sensor (orientation sensor) for specifying the orientation of the camera unit 209. In addition to the geomagnetic sensor, for example, a 6-axis sensor may be provided.

[Details of processing performed in the navigation system]
Next, processing executed in the navigation system according to this embodiment including the navigation device 1 and the glass device 2 having the above-described configuration will be described. Here, each of the navigation processing executed by the navigation device 1 and the navigation cooperation processing executed by the glass device 2 when the navigation processing is executed in the navigation device 1 will be described.

[Navigation processing executed in the navigation device 1]
FIG. 6 is a flowchart for explaining the navigation process executed by the navigation device 1. The process shown in FIG. 6 is executed by the control unit 102 of the navigation device 1 when an operation for performing a route search is performed after the navigation device 1 is powered on. First, the control unit 102 controls the route search unit 119 to display a route search condition input screen on the display unit 113, and accepts a navigation mode (navigation mode) selection input and route search information input via the touch panel 115. (Step S101). The navigation mode indicates a navigation processing mode to be executed, such as walking navigation, bicycle navigation, car navigation, and train navigation. The route search information is information necessary for route search processing such as a departure place, a waypoint, and a destination.

  Next, the route search unit 119 refers to the network data corresponding to the selected navigation mode in the network DB 106 based on the navigation mode and the route search information received in step S101 under the control of the control unit 102, and determines the route. A search is executed (step S102). Then, the route search unit 119 displays the result of the route search in step S102 on the display unit 113, and accepts route selection and setting operations (step S103).

  Although not shown in FIG. 6, if the result of the route search displayed on the display unit 113 in step S103 is not satisfactory, the processing from step S101 is repeated to display the navigation mode and route search information. In addition, the route search process can be performed again.

  After the processing of step S103, the control unit 102 performs short-range communication with the glass-type device 2 through the short-range communication unit 101 and the short-range transmission / reception antenna 101A, thereby forming a state in which mutual communication is possible (step S104). Thereafter, the control unit 102 controls the route guide unit 120 and starts navigation processing in cooperation with the glass device 2 (step S105). In step S105, the route guidance unit 120 extracts map data of the corresponding area from the map DB 105, and displays a map on the display unit 113 according to the extracted map data. In addition, the route guidance unit 120 displays the route set on the displayed map and the current position of the own device acquired through the GPS unit 112.

  Further, in step S105, the route guidance unit 120 reads a guidance message as necessary from the storage device 103 under the control of the provision control unit 102A, and supplies the guidance message to the glass device 2 to provide the glass device. 2 is output. Further, as will be described later, in step S105, under the control of the processing control unit 102B, the image recognition unit 122 analyzes the captured video from the glass-type device 2, and recognizes, for example, a signal state and a road sign. The recognition result is supplied to the glass device 2 and output through the glass device 2 is also performed. In this way, the navigation process is started.

  The control unit 102 performs so-called event-driven processing that is executed when there is a user instruction or an instruction from the glass-type device 2 such as route confirmation processing, information sharing processing, and information update processing, which will be described in detail later. An execution process is performed (step S106). In step S106, each unit mainly functions under the control of the cooperation control unit 102C, and a route confirmation process, an information sharing process, an information update process, and the like are performed. Thereafter, it is determined whether or not a predetermined event for ending the navigation process, such as arrival at the destination or a predetermined process for ending the navigation process, has occurred (step S107).

  When it is determined in the determination process in step S107 that the predetermined event for ending the navigation process has not occurred, the control unit 102 repeats the process from step S106. Further, in the determination process of step S107, when it is determined that a predetermined event that ends the navigation process has occurred, the control unit 102 ends the navigation process and returns the navigation device 1 to the state before the navigation process is executed. A navigation end process is executed (step S108), and the navigation process shown in FIG. 6 is ended.

  In step S105 shown in FIG. 6, the provision control unit 102A and the process control unit 102B of the control unit 102 function. In step S105, (1) the navigation apparatus 1 provides information and the like to the glass device 2 to perform a cooperative operation. In step S105, (2) the glass-type device 2 provides information and the like to the navigation apparatus 1 to perform a cooperative operation. In step S106, the cooperation control unit 202C functions, and (3) the navigation apparatus 1 and the glass device 2 provide information and the like in a bidirectional manner and perform cooperative operation.

[Navigation cooperation processing executed by the glass-type device 2]
FIG. 7 is a flowchart for explaining the navigation cooperation process executed by the glass device 2 when the navigation process shown in FIG. 6 is executed in the navigation device 1. When the glass device 2 is powered on, the control unit 202 of the glass device 2 executes the navigation cooperation process shown in FIG.

  First, the control unit 202 performs communication according to a request from the navigation device 1 through the short-range communication unit 201 and the short-range transmission / reception antenna 201A, and forms a state in which communication is possible (step S201). The processing in step S201 corresponds to the processing executed in step S104 of the navigation processing executed in the navigation device 1 shown in FIG.

  Then, the control unit 202 causes the camera unit 209 to function under the control of the provision control unit 202B, and navigates the video (captured video) captured through the camera unit 209 through the short-range communication unit 201 and the short-range transmission / reception antenna 201A. The process of transmitting to the apparatus 1 is started (step S202).

  Thereafter, the control unit 202 receives an instruction or the like from the navigation device 1 (step S203), and determines whether an instruction or the like from the navigation device 1 has been received (step S204). When it is determined in the determination process of step S204 that an instruction or the like from the navigation device 1 has been received, the control unit 102 executes a process according to the instruction or the like from the navigation device 1 (step S205).

  In step S205, under the control of the processing control unit 202A, a message is displayed and output according to an instruction from the navigation device 1 or bone conduction is output. Further, in step S205, processing such as providing information indicating the current position acquired through the GPS unit 210 is performed under the control of the cooperation control unit 202C.

  After the process of step S205, for example, it is determined whether or not an operation to turn off the power has been performed (step S206). When it is determined that an operation to turn off the power has been performed, a predetermined end process is performed. This is done (step S207), and the process shown in FIG. If it is determined in step S206 that no power-off operation has been performed, the processing from step S203 is repeated.

  On the other hand, assume that it is determined in the determination process in step S204 that an instruction or the like from the navigation device 1 has not been received. In this case, the control unit 202 receives an operation input to the switch 204A (step S208), and determines whether or not an operation to the switch 204A has been received (step S209).

  If it is determined in step S209 that an operation for the switch 204A has been received, a request corresponding to the operation is formed and transmitted to the navigation device 1 under the control of the cooperation control unit 202C (step S210). For example, when the switch 204A is operated once, a route confirmation request is formed, and when the switch 204A is operated twice continuously, an information sharing request is formed, and the switch 204A is continued three times. If it is operated, an information update request is formed and transmitted to the navigation device 1.

  After the process of step S210 and in the determination process of step S209, when it is determined that an operation for the switch 204A is not accepted, the process proceeds to step S206. Then, as described above, it is determined whether or not an operation to turn off the power has been performed (step S206). When it is determined that an operation to turn off the power has been performed, a predetermined termination process is performed ( Step S207), the processing shown in FIG. If it is determined in step S206 that no power-off operation has been performed, the processing from step S203 is repeated.

  In this way, in the glass-type device 2, the navigation cooperation process that cooperates with the navigation process executed by the navigation device 1 is performed. Then, processing for outputting message information, processing for providing information such as the current position, and processing for forming and transmitting various requests to the navigation device 1 are performed.

[(1) Information is provided from the navigation device 1 to the glass-type device 2 to cooperate]
Next, processing in a case where information is provided from the navigation device 1 to the glass device 2 to perform a cooperative operation will be described. Here, the case where the voice guidance generated in the navigation device 1 is displayed as characters on the glass-type device 2 or transmitted by bone conduction will be described.

[Processing of the navigation device 1]
The navigation device 1 of this embodiment outputs voice guidance only through the earphone connected to the voice output terminal 118 in consideration of the influence on the surroundings when the pedestrian navigation process is being executed. However, when the earphone is detached from the user's ear, the user cannot listen to the voice guidance. Therefore, in the navigation apparatus 1 of this embodiment, information corresponding to the voice guidance is provided to the user even through the glass-type device 2.

  Assume that some kind of voice guidance occurs when the pedestrian navigation mode is selected, and the pedestrian navigation processing is executed using the pedestrian map data and the network data. In this case, in the process of step S105 illustrated in FIG. 6, the provision control unit 102A determines whether the earphone is connected to the audio output terminal 118 based on the detection result of the presence or absence of the earphone connection of the audio processing unit 116. Is determined.

  If it is determined that the earphone is connected, the provision control unit 102A obtains instruction information for instructing only display of guidance information according to voice guidance and voice data of the voice guidance. It transmits to the glass type device 2. Assume that it is determined that the earphone is not connected. In this case, the providing control unit 102A transmits to the glass device 2 instruction information for instructing to perform both display of guidance information according to voice guidance and bone conduction output, and voice data of the voice guidance. To do.

[Processing of glass-type device 2]
In the glass type device 2, in the process of step S205 shown in FIG. 7, when the instruction received from the navigation device 1 instructs only the display of the guidance information, the guidance information is displayed according to the voice guidance. Execute output processing. In this case, the processing control unit 202 </ b> A functions and converts the voice data corresponding to the voice guidance transmitted together to display data, and supplies this to the display processing unit 205. The display processing unit 205 forms a display signal to be supplied to the projection unit 206 from the display data supplied thereto, and supplies this to the projection unit 206. As a result, the projection unit 206 displays guide information corresponding to the supplied display signal on the lens unit of the glass-type device 2.

  FIG. 8 is a diagram illustrating an example of guidance information displayed on the lens unit of the glass-type device 2 by the display processing unit 205 and the projection unit 206 functioning. As shown in FIG. 8, guidance information “Please turn right at the next intersection” which is originally output as voice guidance is displayed (projected) on the lens unit of the glass-type device 2 as a display message. In this embodiment, the process control unit 202A forms display data to be displayed with an arrow mark instructing a right turn displayed in response to the message “turn right” in the guidance information. This is supplied to the processing unit 205. Thereby, even if the earphone connected to the audio output terminal 118 is disconnected from the user's ear, the user can surely recognize the guidance information generated in the walking navigation process.

  Further, in the process of step S205 shown in FIG. 7, it is assumed that the instruction received from the navigation apparatus 1 is an instruction to execute a display output process of guidance information and a bone conduction output process according to voice guidance. . In this case, as described above, the processing control unit 202A causes the display processing unit 205 and the projection unit 206 to function, and displays guidance information on the lens unit of the glass-type device 2 in the manner illustrated in FIG.

  Furthermore, the process control unit 202A supplies voice data corresponding to the voice guidance transmitted together to the bone conduction driving unit 207. The bone conduction drive unit 207 forms a drive signal to be supplied to the bone conduction vibration unit 208 from the audio data supplied thereto, and supplies this to the bone conduction vibration unit 208. The bone conduction vibration unit 208 vibrates according to the drive signal from the bone conduction drive unit 207. As a result, the vibration according to the voice data for voice guidance is directly transmitted to the auditory nerve via the skull, and the content according to the generated voice guidance is the bone conduction sound as in the case of listening to the emitted voice. Recognized by the user.

  As described above, the navigation device 1 and the glass device 2 cooperate with each other, so that the guidance information corresponding to the voice guidance generated in the navigation device 1 during the execution of the navigation process for pedestrians is reliably transmitted through the glass device 2. Can be provided to the user. Moreover, the glass-type device 2 not only simply outputs the guidance information, but can output the guidance information by display or by display and bone conduction sound in accordance with a request from the navigation apparatus 1 and is necessary. You can also add information.

[(2) Information is provided from the glass-type device 2 to the navigation device 1 to cooperate]
Next, processing in a case where information is provided from the glass-type device 2 to the navigation device 1 to perform a cooperative operation will be described. Here, as described above, a case will be described in which image recognition is performed on captured images sequentially provided from the glass-type device 2 and the recognition result is notified.

[Processing of the navigation device 1]
The navigation apparatus 1 of this embodiment can cooperate with the glass-type device 2 as described with reference to the flowcharts of FIGS. 6 and 7 when the navigation process is executed. Shooting images are provided in order. In this case, the image recognition unit 122 functions under the control of the processing control unit 102B of the navigation apparatus 1, performs image recognition by analyzing the captured video from the glass-type device 2, and the result of the image recognition is determined by the control unit 102. Can be notified.

  For example, it is assumed that a traffic light is present in a photographed image from the glass-type device 2 and that the lighting state of the traffic light is recognized, it can be recognized that it is in a red signal state. In this case, the processing control unit 102B functions and the fact that the front traffic light is in red is displayed on the map displayed on the display unit 113, or the audio processing unit 116 and the speaker 117 as audio information, or The sound can be emitted from the earphone connected to the audio output terminal 118.

  The result of image recognition can be notified to the glass-type device 2 and output through the glass-type device 2. In this case, as described above, only the display can be used, or the display and bone conduction can be used. FIG. 9 is a diagram illustrating an example of a case where the result of image recognition is displayed and output through the glass-type device 2. As a result of recognizing the situation on the route by analyzing the captured image from the glass device 2, it is assumed that the signal at the intersection in the traveling direction is a red signal.

  In this case, as shown in FIG. 9A, the recognition result “red signal” can be displayed, or bone conduction output can be performed. Further, as a result of recognizing the situation on the route by analyzing the captured video from the glass-type device 2, it is recognized that the road at the intersection in the traveling direction is prohibited from entering. In this case, as shown in FIG. 9B, the recognition result “Invasion prohibited” can be displayed, or bone conduction output can be performed.

[(3) Information is provided by the navigation device 1 and the glass-type device 2 to cooperate with each other]
Next, processing when the navigation apparatus 1 and the glass-type device 2 provide information and perform a cooperative operation will be described. Here, a route confirmation process, an information sharing process , and an information update process performed during the navigation process will be described. In addition, when the information indicating the current position cannot be acquired through the GPS unit 112 of the navigation device 1, the cooperative operation in the case where the navigation processing is continued by the cooperation of the navigation device 1 and the glass device 2 is also performed. explain.

[Route confirmation processing]
FIG. 10 is a flowchart for explaining the route confirmation process executed by the control unit 102 in step S106 of the navigation process shown in FIG. When the route confirmation process shown in FIG. 10 is executed in step S106 of FIG. 6, an instruction input for executing the route confirmation process is accepted (step S301). Here, the instruction input for executing the route confirmation processing is, for example, an instruction input for a display button for executing the route confirmation processing displayed on the display unit 113 or predetermined hardware provided in the navigation device 1. For example, an instruction input to a key.

  Next, the control unit 102 determines whether or not an instruction input for executing the route confirmation process has been accepted (step S302). When it is determined that the instruction has been accepted, an instruction to be focused is formed on the glass device 2. (Step S303). In step S303, the current position, the navigation mode being executed, the object that can be compared at the current position, and the like are comprehensively considered to determine what kind of captured image is necessary and provide the desired captured image. This is a process of requesting the user wearing the glass-type device 2 so that the device can be received.

  FIG. 11 is a diagram for explaining an instruction of a target of interest provided from the navigation device 1 to the glass device 2 and an example of a route determination result and the like. It is assumed that the navigation mode is a train navigation mode. In this case, as shown in the left column of FIG. 11, for example, when the current position is a departure station or an arrival station, an attention instruction such as “Please pay attention to the station name” is displayed. In some cases, a notice instruction such as “Please pay attention to the number line display” is transmitted to the glass-type device 2.

  Further, it is assumed that the navigation mode is other than the train navigation mode. In this case, as shown in the left column of FIG. 11, depending on the current position and the comparison target that exists, “Please pay attention to the intersection name”, “Please pay attention to the building name”, “ Attention instructions such as “Please pay attention” are transmitted to the glass-type device 2. In any navigation mode, the direction in which the objective lens of the camera unit 209 is directed, such as “please look right” or “please look left”, depends on the position to be compared according to the current position. It can be transmitted to the glass-type device 2 as an attention instruction. The attention instruction is displayed on the lens unit of the glass-type device 2 or bone conduction output, and is provided to the user wearing the glass-type device 2, and an image necessary for route confirmation is transmitted through the camera unit 209. Enable to shoot. Note that the example of the instruction of interest shown in FIG. 11 is merely an example, and there are various other instructions.

  After the processing in step S303, the processing control unit 102B controls the image recognition unit 122, and after transmitting the attention instruction to the glass-type device 2, whether or not a captured video that can be used for route confirmation has been transmitted. Is confirmed (step S304). In the confirmation, the image recognition unit 122 recognizes the captured image transmitted from the glass-type device 2 under the control of the cooperation control unit 102C, and confirms whether the target object is included. It is processing to do.

  And the control part 102 discriminate | determines in the confirmation process of step S304 whether the target object is contained in the picked-up image from the glass-type device 2 (it is OK) (step S305). If it is determined in step S305 that the target object is not included in the captured image from the glass-type device 2 (not OK), the processing from step S303 is repeated.

  Assume that it is determined in the determination process in step S305 that the target object is included in the captured image from the glass-type device 2 (it is OK). In this case, the image recognition unit 122 functions under the control of the cooperation control unit 102C to extract a captured video image to be recognized (step S306), and performs recognition processing of the extracted captured video image and movement route identification processing. Execute (Step S307). In step S307, the user of the navigation apparatus 1 and the glass-type device 2 is accurately identified where and in which direction it is moving.

  Then, the cooperation control unit 102C compares the current position and moving direction of the user of the navigation device 1 and the glass device 2 with the route set by searching for the route, identifies the guidance content, and forms a guidance message. Processing such as performing is performed (step S308). In this step S308, for example, as shown in the right column of FIG. 11, “go to the right, south ticket gate”, “move to the next home”, “return to the previous intersection”, [entrance It forms a guidance message that matches the specified guidance content.

  Thereafter, the guidance message formed in step S308 is notified to the glass device 2 (step S309), the route confirmation processing routine shown in FIG. 10 is terminated, and the next execution timing is awaited. Further, it is assumed that in the determination process in step S302, it is determined that an instruction input for executing the route confirmation process is not accepted. In this case, the control unit 102 receives a request from the glass-type device 2 (step S310), and determines whether a route confirmation request from the glass-type device 2 has been received (step S311).

  FIG. 12 is a diagram for explaining an operation of transmitting a route confirmation request from the glass device 2 and an example of a route determination result and the like. In the glass-type device 2, when an operation of pressing the switch 204 </ b> A only once is performed, a route search request is formed and transmitted to the navigation device 1. For this reason, as shown in the left column of FIG. 12, after paying attention to the object of interest, such as “see the station name and press the switch 204A” or “see the intersection name and press the switch 204A”, the switch 204A is When an operation of pressing only once is performed, a captured image including a target of interest and a route search request can be transmitted to the navigation device 1.

  For this reason, it is assumed that it is determined that the route search request is received from the glass-type device 2 in the determination process in step S311. In this case, under the control of the cooperation control unit 102C, the image recognition unit 122 functions to perform processing for specifying a captured video to be recognized (step S312). The identification process in step S312 comprehensively considers the navigation mode being executed, the object that can be compared at the current position, etc., and determines what kind of captured image is necessary, and determines the determined captured image. It is specified by the image recognition unit 122.

  That is, in the process of step S312, the image recognizing unit 122 includes a buffer memory having a predetermined capacity for temporarily storing captured images from the glass-type device 2, and necessary image capturing from the captured images temporarily stored in the buffer memory. Identify the video. Note that in step S312, information indicating the shooting direction added to the shot video is also taken into consideration, for example, when there is a shot video when the right is viewed continuously and a shot video when the left is viewed. Are both identified as captured images to be recognized. The same applies to a case where the front and back are viewed continuously and a case where the front and rear and the left and right are viewed continuously.

  After the process of step S312, the process from step S306 is performed. That is, the above-described extraction of the captured video (step S306), recognition processing of the captured video and movement route identification processing (step S307), message formation processing (step S308), and notification of the formed message to the glass device 2 Processing (step S309) is performed. As described above, these processes are executed by the function of the image recognition unit 122 under the control of the cooperation control unit 102C.

  As described above, the route confirmation process can be performed from the navigation device 1 to the glass device 2 in accordance with an operation on the navigation device 1. Further, the route confirmation process can be performed by acting on the navigation device 1 from the glass-type device 2 in accordance with an operation on the glass-type device 2. In other words, the navigation apparatus 1 and the glass-type device 2 can cooperate with each other by providing information and the like bidirectionally.

[Intersection discrimination processing]
Note that the navigation device 1 of this embodiment stores and holds the 3D image data of the intersection in the 3D image DB 107. Route confirmation can also be performed using the three-dimensional data of the intersection.

  Assume that the image of the intersection is included in the captured image extracted in step S306 shown in FIG. In addition, the captured image is acquired through information indicating the shooting direction (direction in which the objective lens of the camera unit 209 is directed) detected by the sensor unit 211 of the glass device 2 or the GPS unit 210 of the glass device 2. It is assumed that information indicating the current position is added.

  In this case, in step S307, the cooperation control unit 102C of the navigation device 1 supplies the captured video from the glass-type device 2 to the 3D image comparison unit 121 and starts a process of extracting the intersection feature included in the captured video. Control to do. Further, the cooperation control unit 102 </ b> C controls the 3D image comparison unit 121 so as to start the matching process between the extracted feature of the intersection and the 3D image data of the intersection in the 3D image DB 107.

  Here, an intersection that exists in the vicinity of the current position acquired by the GPS unit 112 and matches with the 3D image data of the intersection when viewed from the imaging direction added to the captured image from the glass-type device 2. Is done. In this case, when information indicating the current position is added to the captured image from the glass-type device 2, the current position from the glass-type device 2 is replaced with the current position acquired by the GPS unit 112 of the navigation device 1. Information indicating the position may be used.

  Thereby, the intersection where the user passes (is about to pass) can be specified, and the place where the user is moving can be specified. Then, in step S307, the cooperation control unit 102C starts a process of determining whether or not the route set route is traced based on the matching result. When the route is out of the route set route, , Process to correct the route. The process of correcting the route is a process in which the cooperation control unit 102C controls the route search unit 119 to search for a route that returns to the set route from the current position acquired through the GPS unit 112.

  Thereafter, in step S309, the cooperation control unit 102C forms a processing result notification and performs processing for transmitting the notification to the glass device 2. As a result, a guidance message for returning to the setting route is formed on the lens portion of the glass-type device 2, for example, “You are off the setting route. Please turn right at the next intersection”. This can be provided to the user through the glass-type device 2.

[Train navigation cooperation processing]
In the train navigation mode, the navigation apparatus 1 according to this embodiment can grasp the delay state, change the arrival time at the destination, and adjust the overpass prevention function. .

  For example, it is assumed that an image of a station name signboard is included in the captured video extracted in step S306. The photographed video also includes information indicating the photographing direction (direction in which the objective lens of the camera unit 209 is directed) detected by the sensor unit 211 of the glass device 2 and the current position acquired through the GPS unit 210 of the glass device 2. Is added. In the process of step S307, it is assumed that the station name is extracted by the image recognition unit 122 under the control of the cooperation control unit 102C of the navigation device. In this case, the cooperation control unit 102C performs the correction process of the arrival time at the destination station in the process of step S307 based on the extracted station name and the current time provided by the time information providing unit 111.

  That is, the cooperation control unit 102C specifies the time of arrival at the station specified by the station name from which the train on which the user is going to get on the set route is extracted based on the information in the timetable DB 108. . By comparing this specified arrival time with the current time provided by the time information providing unit 111, it can be determined whether or not the train on board is delayed. And when it is late, cooperation control part 102C considers the time required from the station which recognized the station name to the destination station, and performs processing which corrects the arrival time to the destination station. When there is a transfer, the arrival time at the transfer station is also corrected.

  Then, the cooperation control unit 102C performs a process of correcting the overpass prevention setting based on the correction result in step S307. The transit prevention setting is notified by vibrating a vibrator (not shown) of the navigation device 1 a predetermined time before the arrival time at the transfer station or the destination station (for example, 5 minutes before) or through the glass-type device 2. Correct the arrival time setting for notification by display message or bone conduction output.

  Thereafter, in step S308, the cooperation control unit 102C forms a result notification for the arrival time correction process and the overpass prevention setting correction process, and notifies this to the glass device 2 in step S309. In this way, the arrival time correction process and the overpass prevention setting correction process in the train navigation mode are also performed without any complicated operation by using the captured image including the station name signboard from the glass-type device 2. Easy to do.

[Information sharing process]
FIG. 13 is a flowchart for explaining the information sharing process executed by the control unit 102 in step S106 of the navigation process shown in FIG. When the information sharing process shown in FIG. 13 is executed in step S106 of FIG. 6, an instruction input for executing the information sharing process is accepted (step S401). Here, the instruction input for executing the information sharing process is, for example, an instruction input for a display button for executing the information sharing process displayed on the display unit 113 or predetermined hardware provided in the navigation device 1. For example, an instruction input to a key.

  Next, the control unit 102 determines whether or not an instruction input for executing the information sharing process has been accepted (step S402). When it is determined that the instruction has been received, the control unit 102 instructs the glass-type device 2 to pay attention to the posting target. Is formed and transmitted (step S403). FIG. 14 is a diagram showing an example of an output message used in this information sharing process. In step S403, as shown in FIG. 14B, an instruction message such as “Please pay attention to the situation you want to post” is formed, which is notified to the glass-type device 2 and displayed or outputted. Will be.

  Then, the captured images sequentially transmitted from the glass device 2 are received, and the image recognition unit 122 performs image recognition on the received captured images from the glass device 2 under the cooperation control unit 102C (step S404). . Then, the cooperation control unit 102C detects a state (situation) on the route such as congestion, weather, an accident, and a construction situation (step S405). Note that, as described above, the shooting direction (direction in which the objective lens of the camera unit 209 is directed) detected by the sensor unit 211 of the glass type device 2 is applied to the shot image transmitted from the glass type device 2. Information indicating the current position acquired through the GPS unit 210 of the glass-type device 2 is added.

  Here, the congestion state can be detected based on a change state of the automobile existing in the traveling direction. The weather condition can be detected by the brightness of the image, the color of the sky, the presence of a puddle or an umbrella. In addition, the accident situation can be detected by the stop position of the automobile on the road, the shape of the automobile, the presence or absence of a police car or an ambulance. The construction status can be detected by the presence of construction signs and construction vehicles (including trucks, bulldozers, excavators, etc.).

  Thereafter, the cooperation control unit 102C executes processing for posting the detected state on the route to a predetermined server on the Internet through the wireless communication unit 124 and the transmission / reception antenna 124A (step S406). Posting of the state on the route is performed by pairing the detected state with the current position acquired through the GPS unit 112. Thereby, for example, when a traffic jam occurs, it can be notified where the traffic jam has occurred, and when an accident has occurred, it can be notified where the accident has occurred.

  For example, when there is a traffic jam, the processing control unit 102B measures the travel time per unit distance using the current time provided by the time information providing unit 111, and how fast it moves. It is also possible to notify information indicating whether or not there is. Through the processing in step S406, the information posted to a predetermined server can be shared with drivers of other automobiles that can use the server.

  Thereafter, the cooperation control unit 102C forms a result notification regarding the notification of the state on the route, and notifies this to the glass-type device 2 (step S407). As a result, as shown in FIG. 14B, the posted fact and the posted content are displayed on the lens portion of the glass-type device 2 as shown in FIG. 14B, and the user is notified. it can. Then, the process shown in FIG. 13 is terminated and the next execution timing is awaited.

  Further, it is assumed that in the determination process in step S402, it is determined that an instruction input for executing the information sharing process is not accepted. In this case, the control unit 102 receives a request from the glass-type device 2 (step S408), and determines whether an information sharing request from the glass-type device 2 has been received (step S409).

  If it is determined in step S409 that the information sharing request has been received, the processing from step S404 is performed. Thereby, even in response to an information sharing request from the glass-type device 2, the navigation device 1 can execute the information sharing process. If it is determined in step S409 that the information sharing request has not been received, the process shown in FIG. 13 is terminated and the next execution timing is awaited.

  The information sharing request from the glass-type device 2 is formed and transmitted to the navigation device 1 when, for example, the switch 204A is operated twice in succession or a predetermined function key of the operation unit 204 is operated. It has been made possible.

  As described above, the information sharing process can be performed from the navigation device 1 to the glass device 2 according to an operation on the navigation device 1. Further, the information sharing process can be performed by acting on the navigation device 1 from the glass device 2 in accordance with an operation on the glass device 2. In other words, the navigation apparatus 1 and the glass-type device 2 can cooperate with each other by providing information and the like bidirectionally.

[Information update processing]
FIG. 15 is a flowchart for explaining the information update process executed by the control unit 102 in step S106 of the navigation process shown in FIG. When the information update process shown in FIG. 15 is executed in step S106 of FIG. 6, an instruction input for executing the information update process is accepted (step S501). Here, the instruction input for executing the information update process is, for example, an instruction input for a display button for executing the information update process displayed on the display unit 113 or predetermined hardware provided in the navigation device 1. For example, an instruction input to a key.

  Next, the control unit 102 determines whether or not an instruction input for executing the information update process has been received (step S502). When it is determined that the input has been received, the control unit 102 moves to a place where the glass-type device 2 is desired to detect a change. Is generated and transmitted (step S503). FIG. 16 is a diagram showing an example of an output message used in this information update process. In step S503, as shown in FIG. 16A, an instruction message such as “Please pay attention to the direction in which a change is to be detected” is formed, which is notified to the glass-type device 2 and displayed or displayed. Will be output.

  Then, the photographed images sequentially transmitted from the glass device 2 are received, and the photographed images from the glass device 2 are supplied to the POI comparing unit 123 under the cooperation control unit 102C, and the 3D image DB 107 is positioned. Comparison processing with corresponding 3D image data is started (step S504). Note that, as described above, the shooting direction (direction in which the objective lens of the camera unit 209 is directed) detected by the sensor unit 211 of the glass type device 2 is applied to the shot image transmitted from the glass type device 2. Information indicating the current position acquired through the GPS unit 210 of the glass-type device 2 is added.

  For this reason, in the comparison process in step S504, information indicating the current position from the GPS unit 112 or the current position added to the captured image from the glass-type device 2 is taken into account. The shooting direction added to the shot video is also considered. That is, in step S504, the captured video and the 3D image are compared when the same direction is viewed from the same position.

  Then, the cooperation control unit 102C controls the POI comparison unit 123 to specify the difference between the captured video and the 3D image (step S505). That is, it identifies what is in the captured video but not in the 3D image, and not in the captured video but in the 3D image. Then, the POI comparison unit 123 records the identified difference in a predetermined recording area of the storage device (step S506). In the recording process in step S506, “in which position”, “in which direction”, “what”, the captured image that is not in the 3D image, “in which position”, “which direction”, Information indicating “what” is not in the captured video but in the 3D image is recorded.

  The information “of which position” indicates the latitude and longitude corresponding to the information indicating the current position provided from the GPS unit 112 or the glass device 2. The information indicated by “which direction is seen” is an orientation based on east, west, south, and north. Accordingly, various identifiable orientations such as the southeast direction and the northeast direction are included. The information indicated by “What” is information indicating the object of the difference itself, such as “building”, “traffic light”, “road sign”, etc. This is information indicating an object that can be identified by analyzing and the like.

  Thereafter, the process control unit 102B forms a result notification related to the information update process, and notifies this to the glass-type device 2 (step S507). As a result, as shown in FIG. 16B, for example, “which position”, “what direction”, “what”, and what is in the captured image and not in the 3D image are used through the glass-type device 2. Can be notified. Of course, it is also possible to notify information indicating “in which position”, “in which direction”, “what”, and what is in the 3D image but not in the captured image. Then, the process shown in FIG. 15 is terminated and the next execution timing is awaited.

  Also, assume that it is determined in the determination process in step S502 that an instruction input for executing the information update process is not accepted. In this case, the control unit 102 receives a request from the glass-type device 2 (step S508), and determines whether an information update request from the glass-type device 2 has been received (step S509).

  If it is determined in step S509 that an information update request has been received, the processing from step S504 is performed. Thereby, even in response to an information update request from the glass-type device 2, the information update process can be executed in the navigation device 1. If it is determined in step S509 that the information update request has not been received, the process shown in FIG. 15 is terminated and the next execution timing is awaited.

  The information update request from the glass device 2 is formed and transmitted to the navigation device 1 when, for example, the switch 204A is operated continuously three times or a predetermined function key of the operation unit 204 is operated. It has been made possible.

  In this way, the information update process can be performed from the navigation device 1 to the glass device 2 in accordance with an operation on the navigation device 1. Further, the information update process can be performed by working the navigation device 1 from the glass device 2 according to the operation on the glass device 2. In other words, the navigation apparatus 1 and the glass-type device 2 can cooperate with each other by providing information and the like bidirectionally.

  That is, in the navigation system of this embodiment, the navigation apparatus 1 can receive information from the glass-type device 2 and process it to perform information update processing. When information update processing is performed, it is possible to update existing information by detecting changes in information (existing information) for various POIs already known, and forming and storing information about the changes. Can be useful.

[Cooperative processing when the current position cannot be determined by the navigation device 1]
Next, when the information indicating the current position cannot be acquired through the GPS unit 112 of the navigation device 1, the cooperative operation when the navigation device 1 and the glass device 2 cooperate to continue the navigation process. explain.

[Processing of the navigation device 1]
17 and 18 are executed by the cooperation control unit 102C when information indicating the current position cannot be acquired through the GPS unit 112 of the navigation device 1 when the navigation process is executed (when the GPS position is lost). It is a flowchart for demonstrating the process performed.

  When it becomes GPS lost, the cooperation control unit 102C forms a request for provision of GPS data, and performs processing to transmit the request to the glass-type device 2 (step S601). And cooperation control part 102C performs processing which receives a response from glass type device 2 (Step S602). In the process of step S602, the process waits until a response from the glass device 2 is received.

  When the response from the glass device 2 is received through the response reception process in step S602, the cooperation control unit 102C indicates that the response from the glass device 2 indicates the current position acquired through the GPS unit 210 of the glass device 2. It is determined whether or not it has (GPS data) (step S603). Assume that it is determined in the determination process in step S603 that the response from the glass-type device 2 includes GPS data. In this case, the cooperation control unit 102C controls the route guide unit 120, and continues the navigation process using the GPS data from the glass device 2 as the current position of the own device (step S604).

  Then, the cooperation control unit 102C determines whether or not the information indicating the current position can be acquired through the GPS unit 112 of the own device (step S605). In the determination process of step S605, when it is determined that it is not possible to acquire information indicating the current position through the GPS unit 112, the process from step S604 is repeated, and the GPS data supplied from the glass device 2 is used. Continue the navigation process.

  In the determination process of step S605, when it is determined that the information indicating the current position can be acquired through the GPS unit 112, the cooperation control unit 102C forms a GPS data transmission stop request and sends it to the glass device 2. Transmit (step S606). Thereafter, the cooperation control unit 102C controls the route guide unit 120 to perform navigation processing using information (GPS data) indicating the current position from the GPS unit 112 of the own device (step S607). Then, the processes shown in FIGS. 17 and 18 are terminated, and the normal navigation process is resumed.

  Further, it is assumed that it is determined in the determination process in step S603 that the response from the glass-type device 2 does not include GPS data. In this case, the cooperation control unit 102C confirms the operation state of the route guide unit 120 and determines whether or not position confirmation is necessary (step S608). Specifically, in the determination processing in step S608, the route guidance unit 120 knows the current position of the own device, and determines whether navigation by autonomous navigation is possible according to the sensor output from the sensor unit 110. It is processing to do.

  In the determination process of step S608, when it is determined that position confirmation is necessary (the current position of the own device is not grasped), the cooperation control unit 102C forms a provision request for a video such as a signboard and an address display, This is transmitted to the glass-type device 2 (step S609). Then, the cooperation control unit 102C performs reception processing of the captured video transmitted from the glass device 2 (Step S610). In step S610, the process waits until a captured video from the glass-type device 2 is received.

  In step S610, when the captured video from the glass device 2 is received, the cooperation control unit 102C performs processing for specifying the current position based on the captured image (step S611). In step S611, the cooperation control unit 102C supplies the photographed video from the glass device 2 to the image recognition unit 122 to execute the image recognition process, and displays the company name and address described on the signboard included in the photographed video. To extract attention information (character information). If the attention information is an address display, the position indicated by the address display is the current position of the navigation device 1.

  When the attention information is a company signboard or the like, the linkage control unit 102C is formed by the map data in the map DB 105 with reference to the position indicated by the latest GPS data acquired before the acquisition becomes impossible. The position of the company or the like indicated by the recognized attention information is specified on the map. Next, the cooperation control unit 102C calculates the distance from the shooting position to the subject on which the attention information is displayed (described) according to the position indicated by the attention information and the performance of the camera unit 209 of the glass-type device 2. . Furthermore, the shooting position is specified in consideration of the information indicating the shooting direction added to the shot video from the glass-type device 2. This specified shooting position becomes the current position of the navigation device 1.

  In the determination process of step S608, when it is determined that position confirmation is not necessary (the current position of the own device is grasped) and after the process of step S611, the cooperation control unit 102C performs the process of FIG. move on. In this case, the cooperative control unit 102C uses the current position of the navigation device 1 identified in step S611 or the current position of the grasped navigation device 1 as a reference, and uses the sensor output from the sensor unit 110 as a stand-alone navigation method. The navigation process is executed by (step S612). Then, the cooperation control unit 102C determines whether or not the information indicating the current position can be acquired through the GPS unit 112 of the own device (step S613).

  If it is determined in step S613 that information indicating the current position cannot be acquired through the GPS unit 112 of the own device, the processing from step S608 shown in FIG. 17 is repeated. Also, in the determination process in step S613, it is determined that information indicating the current position can be acquired through the GPS unit 112 of the own device. In this case, the cooperation control unit 102C controls the route guide unit 120 to return to normal navigation processing using information (GPS data) indicating the current position acquired through the GPS unit 112 of the own device (step S614). ). Then, the processing shown in FIGS. 17 and 18 is terminated.

[Processing of glass-type device 2]
On the other hand, as described with reference to the flowchart of FIG. 7, the glass-type device 2 receives an instruction or the like from the navigation device 1. And in the process of step S205 of FIG. 7, the process according to the request | requirement from the navigation apparatus 1 is performed. That is, in step S205 in FIG. 7, processing such as starting and ending provision of information indicating the current position acquired through the GPS 210 to the navigation device 1 and displaying a message indicating the target of interest is performed.

  FIG. 19 shows a case where the GPS device is lost in the navigation device 1 and the current position is unknown in the navigation device 1 and is displayed on the lens unit of the glass device 2 in response to an instruction from the navigation device 1. FIG. 19A is a diagram showing an example of a display message (FIG. 19A) and a scene (FIG. 19B) seen through the lens unit of the glass-type device 2.

  If the navigation device 1 is GPS lost and the current position is unknown in the navigation device 1, the navigation device 1 instructs the attention to the signboard and address display as described above. . For this reason, the cooperation control unit 202C first supplies an instruction message to pay attention to the signboard and address display to the display processing unit 205. As a result, the display processing unit 205 and the projection unit 206 function, and an instruction message for paying attention to a signboard and address display as shown in FIG. 19A is displayed on the lens unit of the glass-type device 2.

  In this case, according to the instruction message displayed in the mode shown in FIG. 19A, the user wearing the glass device 2 on the head, for example, as shown in FIG. Pay attention to (see) signs that can be used to identify your location. As a result, a video that can be used to specify the location of the user is captured and transmitted to the navigation device 1. As a result, the navigation apparatus 1 extracts information that contributes to the identification of the current position, which is good for analyzing the video from the glass-type device 2, and uses the extracted information to identify the current position as described above. The autonomous navigation function can function properly.

  As described above, the navigation apparatus 1 and the glass-type device 2 provide information and cooperate to operate the navigation process appropriately even when the current position cannot be measured in the navigation apparatus 1. .

[Effect of the embodiment]
According to the navigation system of the embodiment described above, (1) the navigation apparatus 1 can provide information and the like to the glass-type device 2 to perform a cooperative operation. Specifically, when voice guidance occurs in the navigation device 1, guidance information corresponding to the voice guidance is displayed on the lens unit of the glass-type device 2 and notified by bone conduction. be able to.

  In addition, (2) information can be provided from the glass-type device 2 to the navigation apparatus 1 to perform a cooperative operation. Specifically, as described above, by using the captured image from the glass-type device 2, for example, it is possible to determine various states such as the state of a signal in the traveling direction and whether or not the road is prohibited from entering. Users can be notified.

  In addition, (3) the navigation apparatus 1 and the glass device 2 can provide information and the like bi-directionally to perform a cooperative operation. Specifically, a route confirmation function, an information sharing function, and an information update function can be realized. Further, in the navigation device 1, when the GPS unit 112 cannot measure the current position, the GPS data from the glass device 2 is used or the captured video from the glass device 2 is used. The navigation process can be continued. In this case, it is possible to provide instructions to each other and perform appropriate processing.

[Modifications, etc.]
In the above-described embodiment, in the current position specifying process using the captured image from the camera unit 209 of the glass device 2, a case where a signboard or address display is photographed and used is described. It is not limited to. For example, it is possible to identify the current position by identifying two landmarks that can be seen on the map by photographing two landmarks existing on the map from the same place. Of course, both are not limited to landmarks, one can be the sun or the moon, and the other can be a landmark.

  It is also possible to identify the current position by identifying a position where two or more landmarks can be seen on the map by photographing two or more appropriate numbers of landmarks from the same place. it can. In this case, the current position is accurately specified in consideration of the rotation angle of the user when viewing different landmarks by mounting a 6-axis sensor on the sensor unit 211 of the glass-type device 2. It is also possible.

  In addition, an instruction to pay attention to the target subject is notified from the navigation device 1 by display output through the glass device 2 or bone conduction output. In response to this, the user wearing the glass device 2 pays attention to the target subject and operates the shutter of the operation unit provided in the ear hooking portion of the glass device 2. A still image of a subject to be photographed can be taken and transmitted to the navigation device 1. In this case, the navigation apparatus 1 may analyze the still image from the glass-type device 2 and perform a process for recognizing characters or a target object. That is, the captured video transmitted from the glass device 2 to the navigation device 1 can be configured as either a moving image or a still image, and the processing method may be determined on the navigation device 1 side.

  Further, for example, when a noise sensor is provided in the navigation apparatus 1 and the ambient noise becomes a predetermined level or more and voice guidance is generated, guidance notification by bone conduction is performed through the glass-type device 2. You may do it.

[Others]
As can be seen from the above-described embodiment, the function of the navigation side communication means of the navigation device is realized by the short-range transmission / reception antenna 101A and the short-range communication unit 101 of the navigation device 1, and the navigation side instruction control means of the navigation device The function is realized by the provision control unit 102A of the navigation device 1. The function of the video processing means of the navigation device is realized by the 3D image comparison unit 121, the image recognition unit 122, and the POI comparison unit 123 of the navigation device 1, and the function of the processing control means of the navigation device is the processing of the navigation device 1. The control unit 102B is realized. The function of the current position acquisition means of the navigation device is realized by the GPS unit 112 and the GPS antenna 112A of the navigation device 1. Further, the functions of the first and second cooperation control means of the navigation device are realized by the cooperation control unit 102C of the navigation device 1. In addition, the function of the extraction unit of the navigation device is realized by the image recognition unit 122 of the navigation device 1.

  The function of the glass side communication means of the glass type device is realized by the short-range transmission / reception antenna 201A and the short-range communication unit 201 of the glass type device 2, and the function of the glass side instruction control means of the glass type device is the glass type device. Two processing control units 202A are realized. The function of the display unit of the glass type device is realized by the display processing unit 205 and the projection unit 206 of the glass type device 2, and the function of the bone conduction unit of the glass type device is the bone conduction driving unit 207 of the glass type device 2. And the bone conduction vibration part 208 is implement | achieved. Further, the camera unit 209 of the glass-type device 2 realizes the function of the imaging device of the glass-type device, and the provision control unit 202B of the glass-type device 2 realizes the function of the provision control unit of the glass-type device. Moreover, the function of the 1st, 2nd cooperation control means of a glass-type device is implement | achieved by the cooperation control part 202C of the glass-type device 2. FIG.

  Further, the navigation apparatus 1 constituting the navigation system of the above-described embodiment is the one to which the embodiment of the navigation apparatus of the present invention is applied. Further, the glass type device 2 constituting the navigation system of the above-described embodiment is one to which an embodiment of the glass type device of the present invention is applied.

  In addition, the method for performing the processing shown in the flowcharts of FIGS. 6 to 7, 10, 13, 15, and 17 to 18 is applied with the embodiment of the inter-device cooperation method according to the present invention. It is.

DESCRIPTION OF SYMBOLS 1 ... Navigation apparatus, 101A ... Short distance transmission / reception antenna, 101 ... Short distance communication part, 102 ... Control part, 102A ... Provision control part, 102B ... Processing control part, 102C ... Cooperation control part, 103 ... Memory | storage device, 104 ... Operation 105: Map DB, 106 ... Network DB, 107 ... 3D image DB, 108 ... Timetable DB, 109 ... POIDB, 110 ... Sensor unit, 111 ... Time information providing unit, 112 ... GPS unit, 112A ... GPS antenna, DESCRIPTION OF SYMBOLS 113 ... Display part, 114 ... Touch sensor, 115 ... Touch panel, 116 ... Sound processing part, 117 ... Speaker, 118 ... Audio | voice output terminal, 119 ... Route search part, 120 ... Route guidance part, 121 ... 3D image comparison part, 122 ... image recognition unit, 123 ... POI comparison unit, 124 ... wireless communication unit, 124A ... transmission / reception antenna, 2 ... Las-type device, 201A ... near field transmission / reception antenna, 201 ... near field communication unit, 202 ... control unit, 202A ... processing control unit, 202B ... providing control unit, 202C ... cooperation control unit, 203 ... storage device, 204 ... operation unit 205 ... Display processing unit 206 ... Projection unit 207 ... Bone conduction drive unit 208 ... Bone conduction vibration unit 209 ... Camera unit 210 ... GPS unit 210A ... GPS antenna 211 ... Sensor unit

Claims (9)

A navigation system formed by a navigation device and a glass-type device,
The navigation device
Navigation-side communication means for communicating with the glass-type device;
Instructing to generate message data for the user of the glass-type device and to output guidance information corresponding to the message data in a mode other than voice output when a predetermined event occurs, at least according to the event output instruction to both, and the navigation-side command control means for controlling to transmit to said glass-type device through the navigation device side communication means,
After transmitting the message data and the output instruction, when receiving a photographed video from the glass-type device through the navigation side communication means, video processing means for processing the photographed video;
Using a processing result in the video processing means, and a process control means for controlling to execute a desired process ,
The glass-type device is
A glass side communication means for communicating with the navigation device;
Imaging means;
Glass that controls to output guidance information according to the message data in a mode other than voice output according to the output instruction when the output instruction and the message data are received through the glass side communication means Side instruction control means ;
A navigation system comprising: a providing control unit configured to control a captured image obtained by imaging through the imaging unit to be transmitted to the navigation device through the glass side communication unit . The navigation system according to claim 1,
The glass-type device is
Display means;
Bone conduction means, and
According to the output instruction, the glass side instruction control means controls to output guide information according to the message data through one or both of the display means and the bone conduction means. Navigation system. The navigation system according to claim 1 ,
In the navigation device,
The navigation-side instruction control means forms the message data instructing a target to be photographed when an instruction to execute route confirmation processing is received as the predetermined event.
The video processing means performs processing for identifying a moving route by performing recognition target extraction and recognition processing from the captured video from the glass device ,
The navigation system according to claim 1, wherein the processing control unit performs a process of forming a guidance message based on a result of the process of specifying the moving route and providing the guidance message to the glass device . The navigation system according to claim 1 ,
The navigation device includes current position acquisition means for acquiring a current position,
In the navigation device,
The navigation-side instruction control unit forms the message data instructing attention to a posting target when an instruction to execute information sharing processing is received as the predetermined event.
The video processing means can identify the status of the travel route by analyzing the video shot from the glass device ,
The processing control means transmits the specified result in the video processing means together with the current position acquired by the current position acquisition means to a predetermined transmission destination through the navigation side communication means, and is shared with other automobile drivers. A navigation system characterized by being controlled to be able to. The navigation system according to claim 1 ,
The navigation device includes current position acquisition means for acquiring a current position,
In the navigation device,
The navigation-side instruction control means forms the message data instructing attention to a place where a change is to be detected when an instruction to execute an information update process is received as the predetermined event.
The image processing means compares the existing information apparatus itself held corresponding to the captured image and the captured image provided from the glass type device, which can be detected according to the difference in position,
The navigation system according to claim 1, wherein the processing control means controls to accumulate the difference detected by the video processing means. The navigation system according to claim 1,
The navigation device includes current position acquisition means for acquiring a current position,
In the navigation device,
The navigation-side instruction control means, as the predetermined event, provides an image that contributes to specifying the current position when the current position cannot be acquired by the current position acquisition means and the current position needs to be confirmed. Forming the message data to indicate,
The video processing means specifies an image to be noted from the captured video from the glass-type device, and performs a process of specifying the current position based on the specified image.
The navigation system according to claim 1, wherein the processing control means performs navigation processing by autonomous navigation using the identified current position. The navigation device of a navigation system formed by communication connection between a navigation device and a glass device,
When the glass-type device receives an output instruction and message data through the glass-side communication means for communicating with the navigation device, the imaging means, and the glass-side communication means, according to the output instruction, Glass side instruction control means for controlling the guidance information according to the message data to be output in a mode other than voice output, and the navigation device through the glass side communication means Providing control means for controlling to transmit to
Navigation-side communication means for communicating with the glass-type device;
When a predetermined event occurs, at least according to the event, forming the message data for the user of the glass-type device, and outputting guidance information according to the message data in a mode other than voice output Navi side instruction control means for controlling to transmit to the glass device through the navigation side communication means together with the output instruction to instruct;
After transmitting the message data and the output instruction, when receiving a photographed video from the glass-type device through the navigation side communication means, video processing means for processing the photographed video;
A process control means for controlling to execute a desired process using a result of the process in the video processing means;
A navigation device comprising: The navigation device and the glass device of the navigation system formed by communication connection of the glass device,
The navigation device forms a message data for a user of the glass device at least according to the event when a predetermined event occurs with a navigation side communication means for communicating with the glass device, A navigation-side instruction control means for controlling to transmit to the glass device through the navigation-side communication means, together with an output instruction that instructs to output the guidance information according to the message data in a mode other than voice output, After the message data and the output instruction are transmitted, when a captured video is received from the glass-type device through the navigation side communication unit, a video processing unit that processes the captured video, and a process in the video processing unit And a process control means for controlling to execute a desired process using the result of
A glass side communication means for communicating with the navigation device;
Imaging means;
Glass that controls to output guidance information according to the message data in a mode other than voice output according to the output instruction when the output instruction and the message data are received through the glass side communication means Side instruction control means;
Providing control means for controlling to transmit a captured image obtained by imaging through the imaging means to the navigation device through the glass side communication means;
A glass-type device comprising: An inter-device cooperation method used in a navigation system formed by communication connection between a navigation device and a glass-type device,
In the navigation device,
When a predetermined event occurs, the navigation-side instruction control means forms message data for the user of the glass-type device at least according to the event, and guide information according to the message data other than voice output A navigation side instruction control step for controlling to transmit to the glass type device through a navigation side communication means for communicating with the glass type device together with an output instruction for instructing output in an aspect,
After transmitting the message data and the output instruction, when receiving a captured video from the glass-type device through the navigation-side communication unit, the video processing unit processes the captured video,
A process control step in which a process control means controls to execute a desired process using a result of the process in the video processing step;
Have
In the glass type device,
When the output instruction and the message data are received through the glass side communication means for communicating with the navigation device, in response to the output instruction, the glass side instruction control means provides guidance information according to the message data. A glass-side instruction control process for controlling to output in a manner other than voice output,
A providing control step of controlling the providing control means to transmit a captured image obtained by imaging through the imaging means to the navigation device through the glass side communication means;
An inter-device cooperation method characterized by comprising:

Images