JP6120751B2 - Information processing apparatus, information terminal, information communication system, and information providing method - Google Patents

Description

  The present invention relates to an information processing apparatus, an information terminal, an information communication system, and an information providing method.

  In recent automobiles, a plurality of display devices may be mounted. For example, the display device is disposed in front of the driver's seat, between the driver's seat and the front passenger seat, and in front of the front passenger seat. In addition, a display device for the rear seat may be disposed on the headrest of the front seat.

  Patent Document 1 discloses a technique for sharing the same virtual space video between different display devices. Specifically, a virtual space image (a virtual car is exemplified) is drawn from a viewpoint corresponding to the position and posture of the person wearing the head mounted display (HMD), and the virtual space image is displayed as HMD and non-HMD. Display on the display device for the wearer. When an instruction from the non-HMD wearer to the HMD wearer is input (for example, when the non-HMD wearer touches a certain part of the virtual car with the touch panel), the instruction is displayed on the HMD. Display on the space image.

JP 2007-42073 A

  By the way, in a mobile body such as an automobile, it may be difficult to communicate between people. For example, a person who enjoys AV content may not notice even if he / she talks. In addition, for example, it is better to refrain from talking to the driver and largely turning the driver's line of sight.

  Also, for example, in a moving body such as a large vehicle, a microbus, a train, an airplane, a ship, etc., it is difficult for people who are away from each other to talk. Even if they are close to each other, the voices of other people may get in the way.

  The factors that hinder communication are not limited to the above examples. In addition, the environment in which people communicate is not limited to the mobile body.

  An object of the present invention is to provide a technology that is useful for people's communication, for example, a technology that enables communication in a form that is easier for the other party to understand than verbal explanation.

The information processing apparatus according to the present invention includes a first display control unit and a second display control unit. The first display control unit controls display on the first information terminal. The first information terminal exists in the moving body together with the second information terminal, and the first information terminal or the second information terminal is a driver terminal for the moving body. The second display control unit performs a partial designation operation when a partial designation operation for designating a part in an operation side terminal image that is an image displayed on the first information terminal is detected by the first information terminal. The display of the second information terminal is controlled so that the notification information regarding the performed part is displayed on the second information terminal.
In particular, the operation side terminal image includes an image captured by a camera mounted on a moving body.
Alternatively, the second display control unit configures the notification information according to the type of gesture operation performed as the part designation operation, and performs a surrounding gesture operation that draws a trajectory so that the part designation operation surrounds the part of the operation-side terminal image. In some cases, the second display control unit configures the notification information with a configuration in which a range object having a design representing the range is added to the portion of the operation-side terminal image.
Alternatively, the second display control unit configures the notification information according to the content displayed in the portion where the partial designation operation is performed, and the portion where the partial designation operation is performed is a display of the moving speed of the mobile object. In some cases, the second display control unit configures the notification information with a configuration having a content that calls attention regarding the moving speed.
Alternatively, the second display control unit configures the notification information according to the content displayed in the part where the part designation operation is performed, and the operation side terminal image is an image captured by a camera mounted on the moving body And the second display control unit designates a movement caution object for calling attention regarding movement as a partial designation of the operation-side terminal image when the part on which the part designation operation is performed is outside the lane in which the moving object is located. The notification information is configured by the configuration added to the part where the operation is performed.
Alternatively, the second display control unit configures the notification information according to the content displayed in the part where the part designation operation is performed, and the operation side terminal image is an image captured by a camera mounted on the moving body When the part on which the part designation operation is performed is a facility, the second display control unit configures the notification information with a configuration that indicates the position of the facility on the map image.
Or a 2nd display control part comprises notification information according to the situation where a mobile body is located.
Alternatively, when the position of the part where the part designation operation is performed in the operation side terminal image changes with the change of the operation side terminal image, the second display control unit performs the part designation operation in the notification information The display position of the object associated with the part is made to follow the position change of the part.
Or a 1st display control part adds the object linked | related with the part in which the part designation | designated operation was performed in notification information to an operation side terminal image.
Alternatively, the first information terminal and the second information terminal share the display screen and display each image constituting the split view on the display screen.
The information providing method according to the present invention includes steps (a) and (b). In step (a), the display of the first information terminal is controlled. The first information terminal exists in the moving body together with the second information terminal, and the first information terminal or the second information terminal is a driver terminal for the moving body. In step (b), when the first information terminal detects a partial designation operation for designating a part in the operation-side terminal image that is an image displayed on the first information terminal, the partial designation operation is performed. The display of the second information terminal is controlled so that the notification information regarding the part is displayed on the second information terminal.
In particular, the operation side terminal image includes an image captured by a camera mounted on a moving body.
Alternatively, step (b) includes a step (b1) of configuring notification information according to the type of gesture operation performed as the part designation operation, and step (b1) includes a part designation operation in which a part of the operation side terminal image is selected. In the case of an enclosing gesture operation that draws a trajectory so as to enclose, there is a step of configuring notification information by a configuration in which a range object having a design representing a range is added to the portion of the operation side terminal image.
Alternatively, step (a) includes a step of adding an object associated with the part where the part designation operation is performed in the notification information to the operation side terminal image.

According to the present invention , visual information can be sent from the user of the first information terminal to the user of the second information terminal. The visual information may be easier for the receiving user to understand than verbal explanation. In such a case, smooth communication can be realized.

  The objects, features and advantages of the present invention will become more apparent from the following detailed description and the accompanying drawings.

1 is a block diagram illustrating a first configuration example of an information communication system according to Embodiment 1. FIG. FIG. 3 is a block diagram illustrating a second configuration example of the information communication system according to the first embodiment. 3 is a block diagram illustrating a first configuration example of an information terminal according to Embodiment 1. FIG. 5 is a diagram for describing a first example of information transmission between information terminals in the first embodiment. FIG. 5 is a diagram for describing a first example of information transmission between information terminals in the first embodiment. FIG. 6 is a diagram for describing a second example of information transmission between information terminals in the first embodiment. FIG. FIG. 10 is a diagram illustrating a third example of information transmission between information terminals in the first embodiment. 3 is a block diagram illustrating an exemplary configuration of an information processing unit according to Embodiment 1. FIG. FIG. 10 is a diagram illustrating object tracking control by a second display control unit in the first embodiment. 5 is a flowchart illustrating an operation example of an information processing unit in the first embodiment. 5 is a flowchart illustrating an operation example of an information processing unit in the first embodiment. FIG. 10 is a diagram for explaining an example of an object in the second embodiment. FIG. 16 is a diagram illustrating a first example in which notification information is configured according to the display content of a part for which a part specifying operation has been performed in the third embodiment. FIG. 25 is a diagram illustrating a second example in which notification information is configured according to the display content of a part where a part specifying operation has been performed in the third embodiment. FIG. 24 is a diagram illustrating a third example in which notification information is configured according to the display content of a part where a part specifying operation has been performed in the third embodiment. 10 is a block diagram illustrating a second configuration example of an information terminal according to Embodiment 4. FIG. FIG. 10 is a block diagram illustrating a third configuration example of the information communication system according to the fifth embodiment.

<Embodiment 1>
<Information communication system>
FIG. 1 is a block diagram illustrating a first configuration example of an information communication system 10 according to the first embodiment. As shown in FIG. 1, the information communication system 10 includes a plurality of information terminal apparatuses 11. However, the number of information terminal apparatuses 11 is not limited to the example of FIG. In addition, the information terminal device 11 may be called the information terminal 11, the terminal 11, etc.

  The information terminal 11 provides the user with visual information and audio information. Further, the information terminal 11 accepts information input by the user. Each information terminal 11 has a communication function and communicates with other information terminals 11.

  The communication function can be realized by wireless communication, wired communication, or a combination thereof. Below, the example which all the information terminals 11 have both a transmission function and a reception function is given. However, the information communication system 10 may include a transmission-only information terminal 11 or may include a reception-only information terminal 11.

  Since the information terminal 11 has both the transmission function and the reception function as described above, the information terminal 11 may operate as the information terminal 11 on the transmission side, or may operate as the information terminal 11 on the reception side. The information communication system 10 employs both one-to-one communication and one-to-many communication. However, it is also possible to adopt only one of the communication forms.

  In addition, the information terminal 11 when operating as a transmission side may be referred to as a transmission side information terminal 11, a transmission side terminal 11, a first information terminal 11, a first terminal 11, and the like. Similarly, the information terminal 11 when operating as a receiving side may be referred to as a receiving side information terminal 11, a receiving side terminal 11, a transmission destination terminal 11, a second information terminal 11, a second terminal 11, and the like.

  The information communication system 10 is arranged in the moving body, and thus the information terminal 11 exists in the moving body. Below, a motor vehicle is illustrated as a moving body. However, the moving body may be a vehicle other than an automobile (for example, a railway vehicle) or a moving body other than a vehicle (for example, an airplane or a ship).

  In addition, it is assumed that at least one of the information terminals 11 is provided for a driver. Note that the driver information terminal 11 may be referred to as a driver terminal 11, for example.

  Each information terminal 11 may be either a built-in type or a portable type. A built-in information terminal 11 is provided for each seat, for example. As will be described later, for example, the information terminal 11 for the driver's seat is embodied as an integrated instrument panel. Here, the main user of the information terminal 11 provided for the driver's seat is a driver. Therefore, the driver's seat information terminal 11 is the driver's information terminal 11.

  The portable information terminal 11 (hereinafter also referred to as a portable terminal 11) is, for example, a PDA (Personal Digital Assistant) such as a mobile phone, a smartphone, or a tablet terminal. The mobile terminal 11 may be a device for a specific application such as a PND (Portable Navigation Device).

  Here, as shown in FIG. 1, the information communication system 10 is connected to an image supply source 15 mounted on an automobile by wire or wirelessly. In the example of FIG. 1, the image supply source 15 supplies an image to each information terminal 11. However, the image supply source 15 may be provided so that an image is supplied only to a part of the information terminals 11. The image supplied by the image supply source 15 may be a still image or a moving image.

  The image supply source 15 is, for example, a television receiver, an AV content playback unit, or an imaging unit. The imaging unit includes one or more cameras, and supplies the information communication system 10 with captured images in one or more directions outside the vehicle. Hereinafter, an imaging unit is mainly exemplified as the image supply source 15, and for this reason, the image supply source 15 may be referred to as an imaging unit 15, a camera 15, and the like.

  FIG. 2 is a block diagram illustrating a second configuration example of the information communication system 10. In the example of FIG. 2, an overall system control device 12 that controls the information communication system 10 as a whole is added. The overall system control device 12 has a communication function and communicates with each information terminal 11. The communication function can be realized by wireless communication, wired communication, or a combination thereof.

  The overall system control device 12 functions as, for example, an information communication relay device that relays communication between the information terminals 11. That is, each information terminal 11 communicates via the system overall control device 12. Such a relay function can be realized by a general routing function.

  The overall system control device 12 is also arranged in the automobile. As long as communication with the information terminal 11 is possible, the arrangement position of the overall system control device 12 is not particularly limited.

  In the example of FIG. 2, the imaging unit 15 supplies an image to the overall system control device 12. However, an image supply source 15 may be provided so that an image is supplied to each information terminal 11 as in FIG. 1, or an image is supplied to each information terminal 11 and the overall system control device 12. .

  Here, each information terminal 11 and the overall system control device 12 is given unique identification information (hereinafter also referred to as ID) in advance. The ID is used, for example, to specify the source and destination of information. In the following, in order to simplify the description, the ID is shared not only for communication but also for other processing.

<Information terminal>
FIG. 3 is a block diagram illustrating a configuration example of the information terminal 11. According to the example of FIG. 3, the information terminal 11 includes an information processing unit 21, a storage unit 22, and an information output unit 23. Here, the information output unit 23 includes a display unit 24 and an audio output unit 25. The information terminal 11 further includes an information input unit 26, an external communication unit 27, and an external input unit 28.

  The information processing unit 21 is a device that performs various processes in the information terminal 11. For this reason, the information processing unit 21 may be called the information processing device 21. The various processes include, for example, control of the storage unit 22, the information output unit 23, the information input unit 26, the external communication unit 27, and the external input unit 28. The various processes include, for example, generation of data to be supplied to the information output unit 23 and analysis of a user operation acquired through the information input unit 26.

  Here, the information processing unit 21 includes a central processing unit (for example, configured with one or a plurality of microprocessor circuits) and a main storage unit (for example, one or a plurality of storage devices such as a ROM, a RAM, and a flash memory). The case where it is comprised is illustrated. According to this example, various processes are executed by the central processing unit executing various programs stored in the main storage unit. Various processes can be executed in parallel. Various functions corresponding to the various processes are realized.

  The program executed by the central processing unit may be stored in advance in the main storage unit, or may be read from the auxiliary storage unit (here, the storage unit 22) during execution and stored in the main storage unit. Good. The main storage unit is used not only for storing programs but also for storing various data. In addition, the main storage unit provides a work area when the central processing unit executes the program. The main storage unit provides an image holding unit for writing an image to be displayed on the display unit 24. The image holding unit may be called a video memory or a graphic memory.

  Here, various functions of the information processing unit 21 are realized by software, but all or a part of the various functions are realized by hardware (for example, an arithmetic circuit configured to perform a specific arithmetic operation). Also good.

  The storage unit 22 stores various information. Here, the storage unit 22 is provided as an auxiliary storage unit used by the information processing unit 21. The storage unit 22 can be configured using one or more storage devices such as a hard disk device, a rewritable and nonvolatile semiconductor memory, and an optical disk device, for example.

  Examples of information stored in the storage unit 22 include image data (objects such as icons, map data, etc.), audio data (operation sounds, notification sounds, sound effects, guidance sounds, etc.), AV data, and the like.

  The display unit 24 provides visual information to the user. Here, a case where the display unit 24 is configured by a liquid crystal display device is illustrated, and the display unit 24 may be referred to as a liquid crystal display device 24 or a display device 24 below. However, the display unit 24 may be configured by other types of display devices. For example, the display unit 24 performs a display operation based on the image data supplied from the information processing unit 21.

  The audio output unit 25 provides auditory information to the user. The audio output unit 25 can be realized by an audio output device including a speaker, an audio amplifier, and the like. For example, audio data supplied from the information processing unit 21 is converted into an analog electric signal by an audio amplifier and output as audio from a speaker.

  The information input unit 26 (hereinafter also referred to as the input unit 26) receives an input operation performed on the information terminal 11, and the input operation is processed by a signal that can be processed by the information processing unit 21. Hand over to 21.

  Here, the case where the input part 26 is comprised with a contact-type touch panel is illustrated. For this reason, hereinafter, the input unit 26 may be referred to as a touch panel 26. However, it is also possible to employ a non-contact type touch panel. The touch panel may be called a touch pad. Further, the contact type is sometimes called a two-dimensional (2D) type, and the non-contact type is sometimes called a three-dimensional (3D) type.

  Further, a structure in which the touch panel 26 is overlaid on the screen of the liquid crystal display device 24 is illustrated. According to this, the user can perform an input operation with a sense of touching an icon or the like displayed on the screen. However, it is possible to adopt a structure in which the touch panel 26 is not superimposed on the screen of the liquid crystal display device 24. In this case, the touch panel 26 is used in a place different from the screen of the liquid crystal display device 24.

  Moreover, a user's finger (more specifically, a fingertip) is illustrated as an instruction | indication which inputs with respect to the touch panel 26. FIG. However, a tool such as a stylus pen (also called a touch pen) may be used as an indicator.

  Note that a pointing device such as a mouse may be used as the input unit 26 instead of or in addition to the touch panel. In addition, a so-called motion capture device that recognizes a user's operation by various sensors, cameras, or the like may be used.

  The external communication unit 27 is a circuit that mediates a communication signal between the information processing unit 21 and an external device existing outside the terminal. The external device corresponds to the other information terminal 11 in the example of FIG. 1, and corresponds to the overall system control device 12 in the example of FIG.

  Specifically, the external communication unit 27 acquires data to be transmitted from the information processing unit 21, converts the transmission data into a transmission signal of a communication standard adopted in advance, and outputs the transmission signal to the outside of the information terminal 11. To do. On the contrary, the external communication unit 27 receives a signal arriving from the outside of the information terminal 11, extracts data included in the received signal in accordance with a communication standard adopted in advance, and obtains the received data obtained as an information processing unit Deliver to 21.

  The external input unit 28 is a part that captures image data supplied from the image supply source 15. Specifically, when the image supply source 15 is connected by wire, the external input unit 28 includes, for example, a connection terminal and wiring, and may further include a signal shaping circuit. When the image supply source 15 is wirelessly connected, the external input unit 28 includes, for example, a wireless communication circuit. When the image supply source 15 is wirelessly connected, the external input unit 28 and the external communication unit 27 may be integrated.

  As described above, in the example of FIG. 2, the image supply source 15 is connected to the overall system control device 12. In that case, the information terminal 11 may not have the external input unit 28. Further, the system overall control device 12 is provided with the same elements as the external input unit 28.

  The configuration of the information terminal 11 is not limited to the example of FIG. For example, when the output of auditory information is not used, the audio output unit 25 may be omitted.

  In addition, a position information acquisition unit that acquires information on the current position of the information terminal 11 may be provided. The position information acquisition unit can be configured by, for example, a GPS (Global Positioning System) receiving antenna and a GPS receiving device that detects the current position of the information terminal 11 based on information from GPS satellites received by the GPS receiving antenna. is there. However, the configuration of the position information acquisition unit is not limited to this example.

<Outline of information transmission>
Before describing a more specific configuration example, an outline of information transmission between the information terminals 11 will be described. In order to make the explanation easy to understand, information transmission between the information terminal 11 for the driver's seat and the information terminal 11 for the passenger's seat will be exemplified. The driver's seat information terminal 11 may be referred to as a driver's seat terminal 11, and the passenger's seat information terminal 11 may be referred to as a passenger's seat terminal 11. Since the driver's seat terminal 11 is provided for the driver, the driver's seat terminal 11 is the driver's terminal 11.

  Note that the display screen of the driver's seat terminal 11 is arranged in the front portion of the driver's seat on the dashboard, and is embodied as an integrated instrument panel, for example. The integrated instrument panel is, for example, a meter (speedometer, tachometer, etc.), warning light, indicator light, navigation image, operation status of various devices (AV (Audio-Visual) equipment, etc.), It is a display panel that can display captured images in an integrated manner. The integrated instrument panel is sometimes referred to as a meter cluster. Further, the display screen of the passenger seat terminal 11 is arranged in the front part of the passenger seat of the dashboard.

  4 and 5 are diagrams for explaining an example of information transmission between the information terminals 11. In the example of FIGS. 4 and 5, the car is moving backward into the parking space, the captured image 101 behind the vehicle is displayed on the driver's seat terminal 11, and the same captured image 101 is also displayed on the passenger's seat terminal 11. Yes. The driver confirms the reverse state with the captured image 101 of the driver terminal 11 (that is, the terminal 11) while paying attention to the real space.

  Assume that, for example, an infant is found in a real space in a range where the passenger in the passenger seat is not displayed on the information terminal 11 during reverse travel.

  Therefore, the passenger in the passenger seat touches, for example, a portion that is considered to be closest to the infant in the captured image 101 displayed on the passenger seat terminal 11 (that is, the own terminal 11) (see FIG. 4). Note that the touched part may be, for example, a part that an infant is supposed to appear in the captured image 101.

  When the passenger seat terminal 11 is touched, a preset object 102 is displayed on the touched portion of the passenger seat terminal 11 (see FIG. 4). At the same time, the same object 102 appears in the same position in the captured image 101 displayed on the driver's seat terminal 11 (see FIG. 4). At that time, the driver's seat terminal 11 may output sound (alarm sound, utterance of “attention”, etc.).

  According to this, information that is not displayed in the captured image 101 but exists in the real space can be notified to the driver by visual information called an image. That is, the information obtained by the passenger is also shared with the driver. Rather than explaining the above-mentioned part in the captured image 101 verbally, it is considered that it is easier for the driver to understand the image, and smooth communication can be realized.

  Here, the information transmitted from the passenger's seat terminal 11 to the driver's seat terminal 11 is information 100 relating to the portion designated by the passenger in the passenger seat. Hereinafter, the information 100 may be referred to as notification information 100. Note that the notification information 100 is not limited to the above example (an example of notifying that an infant has been found). In the example of FIG. 4, the notification information 100 includes a captured image 101 and an object 102.

  When the object 102 displayed on the passenger seat terminal 11 is touched again (see FIG. 5), the object 102 disappears from the display screens of the passenger seat terminal 11 and the driver seat terminal 11 (see FIG. 5). The touch operation here is an operation for ending the display of the notification information 100 and is referred to as a notification information end operation. However, a gesture operation other than the touch operation may be assigned to the notification information end operation.

  4 and 5, transmission and deletion of the notification information 100 are operated by the passenger seat terminal 11. The operation is performed on an image (captured image 101 in the above) displayed on the passenger seat terminal 11. In view of this point, the information terminal 11 on the side that operates the transmission and deletion of the notification information 100 may be referred to as the operation side terminal 11, and the image displayed on the operation side terminal 11 may be referred to as the operation side terminal image.

  The design of the object 102 is not limited to the example of FIG. Further, the user may be able to set in advance from a plurality of types of designs.

  Further, the design of the object 102 may be associated with an operation for designating a part in the captured image 101 (hereinafter also referred to as a part designating operation).

  For example, as shown in FIG. 6, when the part specifying operation is a gesture operation (for example, dragging or flicking) that draws a locus in a straight line, an arrow-shaped object 102 facing the same direction as the gesture direction is displayed. May be. According to the design representing the directionality like the arrow shape, for example, it is also possible to notify the information on which direction the infant is approaching.

  For example, as illustrated in FIG. 7, when a gesture operation surrounding a portion in the captured image 101 is performed, an object 102 having a design representing a range may be displayed. A design that represents a range such as a hollow circular shape illustrated in FIG. 7 is suitable when a portion related to the notification information 100 is relatively large.

  Here, the arrow-shaped design illustrated in FIG. 6 represents the direction of the gesture operation, and the object 102 having the design representing such directionality may be referred to as the direction object 102.

  On the other hand, although the design illustrated in FIG. 7 does not have directionality, it can be said that the range is represented by the hollow shape. For this reason, the object 102 having such a design may be called a range object 102.

  Also, the design illustrated in FIG. 4 does not seem to be intended for directionality and range. However, it can be understood that such a design represents a position because attention is directed to the position itself where the object 102 is displayed. Therefore, the object 102 having the design representing the position may be called the position object 102.

  In this way, various types of notification information 100 can be configured by using a plurality of types of objects 102. Further, by properly using the position object 102, the direction object 102, and the range object 102, the notification information 100 indicates the intention of the user on the transmission side of the notification information 100 (intention of which position, direction, and range should be noticed). Can be included.

  In addition, although the transmission destination of the notification information 100, that is, the receiving side terminal 11, is set in advance, the setting may be changeable by the user. Further, an operation for setting a transmission destination may be performed when the notification information 100 is transmitted.

<Notification information transmission function>
FIG. 8 is a block diagram illustrating a configuration example of the information processing unit 21 regarding transmission of the notification information 100. According to the example of FIG. 8, the information processing unit 21 includes a notification information transmission function unit 200, and the notification information transmission function unit 200 includes a first display control unit 201 and a second display control unit 202. Yes.

<First display control unit>
The first display control unit 201 controls the display of the information terminal 11 (here, the own terminal 11 on which the first display control unit 201 is mounted) on the transmission side of the notification information 100.

  For example, when the partial designation operation is performed on the own terminal 11, the first display control unit 201 adds the object 102 on the captured image 101 being displayed on the own terminal 11 (see FIG. 4). At that time, the first display control unit 201 sets the display position of the object 102 to the part where the part designation operation has been performed.

  For example, when the notification information end operation is performed, the first display control unit 201 erases the object 102 being displayed (see FIG. 5).

<Second display control unit>
The second display control unit 202 instructs the information terminal 11 (in this case, the other terminal 11) that is the reception side of the notification information 100 to control the display.

  Specifically, when a part specifying operation is performed on the terminal 11, the second display control unit 202 displays the notification information 100 regarding the part on which the part specifying operation is performed on the other terminal 11. In addition, the display of the other terminal 11 is controlled.

  In that case, according to the example of FIGS. 4-7, the 2nd display control part 202 comprises the notification information 100 according to the kind of gesture operation performed as partial designation | designated operation.

  That is, depending on whether the part specifying operation is a touch operation, a directional gesture operation that draws a trajectory with directionality, or an enclosing gesture operation that draws a trajectory so as to surround, the type of the object 102 Change (in other words design). As a result, the configuration of the notification information 100 changes depending on the type of the object 102.

  The second display control unit 202 instructs the receiving side terminal 11 so that the notification information 100 configured as described above is displayed on the receiving side terminal 11. However, the control instruction may take various forms. It is done.

  For example, there is an instruction form in which the second display control unit 202 itself generates and transmits the notification information 100. Specifically, the object 102 is added to each frame image of the captured image 101, and the frame image generated in this way is transmitted to the receiving terminal 11. Note that all the frame images may be processed, or the processing targets may be reduced by thinning out the frame images.

  Further, for example, the second display control unit 202 may transmit only the information necessary for generating the notification information 100 to the receiving terminal 11 and cause the receiving terminal 11 to generate the notification information 100.

  Specifically, in the examples of FIGS. 4 to 7, the reception side terminal 11 (the driver's seat terminal 11 in the above) displays the same captured image 101 as the transmission side terminal 11 (the passenger's seat terminal 11 in the above). Therefore, only information related to the object 102 may be transmitted to the receiving terminal 11. The information regarding the object 102 is, for example, the image data of the object 102 and the display position of the object 102. When the receiving terminal 11 has image data of the object 102, information specifying the type of the object 102 may be transmitted instead of the image data.

  Here, assuming that the receiving terminal 11 does not display the captured image 101 constituting the notification information 100, an input source switching instruction for acquiring the captured image 101 from the imaging unit 15 may also be transmitted. .

  Note that the transmission-side user may be instructed to transmit the input source switching instruction to the reception-side terminal 11 by, for example, performing a partial designation operation with two fingers.

  In addition, the second display control unit 202 changes the position of the part where the part designation operation is performed in the captured image 101 (in other words, the object displayed in the part) as the captured image 101 changes. In this case, the display position of the object 102 associated with the part in the notification information 100 is made to follow the position change of the part (see FIG. 9). Detection of a change in the position of the part, that is, extraction of an object existing in the part and motion detection of the object can be performed by using an existing image processing technique.

  When the notification information end operation is performed on the terminal 11, the second display control unit 202 ends the display of the notification information 100 on the reception side terminal 11.

  Specifically, the second display control unit 202 ends the transmission of the notification information 100. Alternatively, the second display control unit 202 ends transmission of information necessary for generating the notification information 100. At that time, the second display control unit 202 may transmit a notification notifying the end of the display of the notification information 100 to the receiving terminal 11.

<Operation example>
10 and 11 are flowcharts for explaining an example of the operation of the notification information transmission function unit 200. 10 and 11 are connected by connectors C1 and C2. 10 and 11 also show an operation flow ST200 of the reception side terminal 11 together with an operation flow ST100 of the transmission side terminal 11 in which the notification information transmission function unit 200 operates.

  According to the operation flow ST100 of the transmission side terminal 11, it waits for a partial designation operation to be performed in step ST101.

  When the partial designation operation is detected by the input analysis unit of the information processing device 21, the first display control unit 201 of the transmission side terminal 11 displays the object 102 on the own terminal 11 in step ST <b> 102. At that time, the type of the object 102 is selected in accordance with the type of the part specifying operation.

  In step ST <b> 103, the second display control unit 202 of the transmission side terminal 11 instructs the reception side terminal 11 to display the notification information 100. Note that step ST103 may be executed before step ST102 or may be executed in parallel with step ST102.

  The receiving terminal 11 instructed to display the notification information 100 displays the notification information 100 in step ST201.

  When the display position of the object 102 changes (see FIG. 9), for example, steps ST102, ST103, and ST201 are repeated.

  The second display control unit 202 of the transmission side terminal 11 waits for the notification information end operation to be performed in step ST104.

  When the notification information end operation is detected by the input analysis unit of the information processing device 21, the first display control unit 201 of the transmission side terminal 11 erases the object 102 displayed on the own terminal 11 in step ST105.

  Then, in step ST106, the second display control unit 202 of the transmission side terminal 11 ends the display of the notification information 100 on the reception side terminal 11. Note that step ST106 may be executed before step ST105 or may be executed in parallel with step ST105.

  In response to step ST106, the receiving terminal 11 ends the display of the notification information 100 in step ST202.

<Effect>
According to the first embodiment, visual information such as an image can be sent from the user of the transmitting terminal 11 to the user of the other terminal 11. The visual information may be easier for the receiving user to understand than verbal explanation. In such a case, smooth communication can be realized.

<Embodiment 2>
Even for the same type of object 102, a plurality of designs may be prepared and used appropriately. An example of this is shown in FIG. According to the example of FIG. 12, it is possible to change the impression of alerting given to the viewer, in other words, the degree of alerting, depending on the design. Note that it is possible to change the alerting degree of the direction object 102 and the range object 102 by design.

  For example, the configuration of the notification information 100 may be changed by changing the design of the object 102 according to the situation where the automobile is located. For example, the design of the object 102 is changed depending on whether the car is located on a parking lot, a general road, or a highway.

  Generally, the vehicle speed on a general road is larger than the vehicle speed on a parking lot. For this reason, the driver may be less likely to notice the presence of the object 102 when traveling on a general road. For this reason, a design with a greater alerting degree is preferable when traveling on a general road. In addition, a design with a higher degree of alerting is preferable on an expressway.

  Here, the design of the object 102 may be selected according to the speed of the own vehicle. However, especially on expressways, it is likely that the driver will need to pay attention not only to his own vehicle but also to other vehicles. Then, it may be difficult to notice the presence of the object 102 regardless of the speed of the own vehicle. In view of this point, the rule of changing the design of the object 102 according to the situation where the automobile is located is considered preferable.

  Information on whether the vehicle is located in a parking lot, a general road, or an expressway, that is, information on the situation where the vehicle is located can be obtained by, for example, comparing the vehicle position with map data. It is. Such collation can be performed using a general navigation function.

  According to the second embodiment, the configuration of the notification information 100 can be diversified.

<Embodiment 3>
In the third embodiment, an example in which the notification information is configured according to the content displayed in the part where the part specifying operation has been performed will be described with reference to FIGS. In the examples of FIGS. 13 to 15, the transmission terminal 11 displays an image in which a speedometer and a tachometer are superimposed on a captured image 101 in front of the vehicle. However, the display image of the transmission side terminal 11, that is, the operation side terminal image is not limited to this example.

  According to the example of FIG. 13, when the part designation operation is performed on the vehicle speed display part, the second display control unit 202 receives the notification information 100 </ b> B configured to alert the vehicle speed. The reception side terminal 11 is instructed to be displayed on the side terminal 11.

  Unlike the examples of FIGS. 4 to 7 described above, the notification information 100B illustrated in FIG. 13 includes only a speed warning object 102B that is a text object “speed attention!”. Note that the speed warning object 102B may be configured by a figure, as in the first embodiment.

  FIG. 13 illustrates the vehicle speed display displayed on the captured image 101. On the other hand, the speed display that is the target of the partial designation operation may be superimposed on another type of image, or may not be superimposed on another image (that is, no background). Further, in the example of FIG. 13, the vehicle speed is displayed as a number, but it may be a pointer-type speed display, for example.

  According to the example of FIG. 14, when the partial designation operation is performed on the outside of the lane in which the automobile (that is, the own vehicle) is located in the captured image 101, the second display control unit 202 is careful to move. The receiving side terminal 11 is instructed so that the notification information 100C having the configuration in which the arousing object 102C is added to the portion where the part specifying operation is performed in the captured image 101 is displayed on the receiving side terminal 11.

  Here, the movement warning object 102C is an object having a design that can call attention regarding movement, and may be called a driving warning object 102 in the case of an automobile. The design of the movement alerting object 102C is not limited to the illustrated example.

  The determination as to whether or not the part where the part designation operation has been performed in the captured image 101 is outside the lane in which the host vehicle is located can be performed using existing technology, for example, image processing technology. For example, an image processing technique for detecting a white line on a road from a captured image is known, and a portion sandwiched between white lines can be estimated as a lane. Further, for example, since the relationship between the imaging direction by the camera 15 and the forward direction (traveling direction) of the automobile is known, the forward direction of the automobile can be estimated in the captured image 101. For this reason, the lane in which the own vehicle is located can be estimated on the captured image 101, and as a result, the portion outside the lane can also be estimated.

  Further, for example, from the position of various objects in the captured image 101 (peripheral scenery, trees, people, fences, steps between roadways and sidewalks, steps made of snow removed, etc.) in the captured image 101, etc. It is possible to estimate the lane in which the vehicle is located and the outer portion thereof. According to this example, even if the white line detection technique is not used (including the case where it is difficult to detect a white line on a snowy road, for example), the portion of the captured image 101 where the partial designation operation has been performed is It can be determined whether the vehicle is outside the lane in which it is located.

  In that case, for example, the position of the own vehicle is specified from the GPS positioning information, and information on the road where the own vehicle is located (number of lanes, distinction between one-way road and two-way road, presence / absence of sidewalk, width of lane and sidewalk etc. ) From the map database, and the acquired information can be combined with various information obtained from the captured image 101, the estimation accuracy can be improved. In addition, for example, the part outside the lane where the vehicle is located (that is, the part in which the part designation operation is performed in the captured image 101) is an adjacent lane, a sidewalk, It is also possible to estimate in detail such as whether it is a part.

  Specifically, according to the example of FIG. 14, it can be estimated from the captured image 101 that the road is a two-lane road, but by using a map database, the two-lane road is a two-way road and each direction It is possible to acquire information such as that there is one lane per road, that there is no sidewalk. Further, in Japan, since the automobile is left-hand traffic, it can be estimated that the portion where the partial designation operation is performed in the example of FIG. 14 is on the shoulder side.

  If more accurate GPS positioning information is obtained using the quasi-zenith satellite, it is also possible to obtain the vehicle position on the order of 10 cm. By comparing the vehicle position with the map database, It is also possible to estimate which range is the lane of the own vehicle.

  According to the example of FIG. 15, when the partial designation operation is performed on the facility shown in the captured image 101, the second display control unit 202 has a configuration in which the location of the facility is indicated on the map image. The receiving terminal 11 is instructed to display the notification information 100 </ b> D it has on the receiving terminal 11.

  In the example of FIG. 15, the notification information 100 </ b> D includes a map image 103 </ b> D and a pointing object 102 </ b> D <b> 1 that is an object that points to the location of the facility that is the part where the partial designation operation is performed. Further, the notification information 100D further includes a text object 102D2 composed of a sentence asking what the facility is. Note that the content of the text object 102D2 is not limited to the question sentence.

  If the existing technology is used, for example, if the augmented reality technology known in the example in which the name of the building is displayed on the screen when the smartphone camera is held over the building, the captured image 101 is displayed. It is possible to identify the facilities that are shown. If the facility can be identified, the location information of the facility can be identified, and as a result, the facility can be pointed on the map image.

  According to the third embodiment, the notification information can be used in various situations.

<Embodiment 4>
In the first to third embodiments, as illustrated in FIG. 3, the example in which one information output unit 23 and one information input unit 26 are provided in the information terminal 11 has been described. In contrast, in the example of FIG. 16, two information output units 23a and 23b and two information input units 26a and 26b are provided. However, three or more information output units 23 and three or more information input units 26 may be provided.

  According to the example of FIG. 16, since there are two display units 24a and 24b, there are two display screens. The two display screens can be provided by physically separate display devices. Such a configuration may be referred to as dual display, dual monitor, multi-display, multi-monitor, etc.

  It is also possible to provide two display screens by a single display device. In this case, the two display units 24a and 24b are realized by a single display device.

  For example, the left screen and the right screen are used as the display screens of the display units 24a and 24b by dividing the display screen of the single display device into left and right parts. Such a configuration may be called a screen division method or the like.

  For example, the single display device may have a configuration conforming to a so-called split view. That is, the split view has different visible images depending on the viewing direction of the display screen, and the split view display screen has a portion that provides an image that is visible from the left direction and a portion that provides an image that is visible from the right direction. For this reason, the display screen of display part 24a, 24b is provided by these two parts.

  The display screens of the split-screen and split-view display devices are arranged, for example, in the center of the dashboard, that is, between the front of the driver seat and the front of the passenger seat. In this case, a driver's seat (that is, driver's) screen and a passenger seat screen are provided. Further, for example, the screen division method or the split view method may be employed for the mobile terminal 11.

  Similarly, the two audio output units 25a and 25b may be configured by separate audio output devices or may be configured by a single audio output device.

  Further, the two information input units 26a and 26b may be configured with separate touch panels or may be configured with one touch panel.

  Here, when the information terminal 11 has the two information output units 23a and 23b, it is assumed that the notification information displayed on one display unit 24a or 24b is transmitted to the other display unit 24b or 24a. . In this case, between the information terminal 11a (see FIG. 16) including the information output unit 23a and the information input unit 26a, and the information terminal 11b (see FIG. 16) including the information output unit 23b and the information input unit 26b, It can be understood that notification information is transmitted and received. In the example of FIG. 16, the information processing unit 21, the storage unit 22, and the external communication unit 27 are shared by the two information terminals 11a and 11b.

<Embodiment 5>
In the first to fourth embodiments, the example in which the notification information transmission function unit 200 (see FIG. 8) is provided in the information terminal 11 has been described. On the other hand, as shown in FIG. 17, the notification information transmission function unit 200 may be provided in the information processing unit of the overall system control device 12.

  In addition, in Embodiment 1-4, the notification information transmission function part 200 is provided in the transmission side terminal 11, and replaces the own terminal 11 with the transmission side terminal 11 in description of Embodiment 1-4, and others. If the terminal 11 is replaced with the receiving terminal 11, the fifth embodiment is fully understood.

  The various effects described above can also be obtained by the fifth embodiment.

<Modification>
In the above description, the example in which the image displayed on the transmission-side terminal 11 and the partial designation operation is performed, that is, the operation-terminal-side image is the captured image 101 has been mainly described. Further, it is assumed that the captured image 101 is a live image. However, the operation terminal side image may not be a live image or a captured image.

  Here, the live image is an image that is displayed with the time length (that is, delay) from the imaging timing to the display timing for the same frame image being substantially negligible (in other words, practically).

  The live image is not limited to the captured image 101 itself. For example, an image obtained by combining a car or the like reflected in a captured image on a map image displayed by map data prepared in advance at a corresponding position on the map image is also a kind of live image. That is, according to the map data, a non-moving object such as a road can be displayed, but a car or the like moving in the real space is not recorded in the map data. On the other hand, according to the composite image using the captured image 101, the current time situation in the real space can be displayed. Therefore, it can be said that the composite image is a kind of live image.

  The present invention can be freely combined with each other within the scope of the invention, and each embodiment can be appropriately modified or omitted.

  Although the present invention has been described in detail, the above description is illustrative in all aspects, and the present invention is not limited thereto. It is understood that countless variations that are not illustrated can be envisaged without departing from the scope of the present invention.

  DESCRIPTION OF SYMBOLS 10 Information communication system, 11 Information terminal, 12 Whole system control apparatus (information communication relay apparatus), 15 Image supply source (imaging part), 21 Information processing part (information processing apparatus), 28 External input part, 100, 100B-100D Notification information, 101 captured image, 102, 102B, 102C, 102D1, 102D2 object, 200 notification information transmission function unit, 201 first display control unit, 202 second display control unit, ST100, ST200 operation flow, ST101 to ST106 , ST201 to ST202 processing steps.

Claims (25)

A first display control unit for controlling display of the first information terminal;
The first information terminal is present in the moving body together with the second information terminal, and the first information terminal or the second information terminal is a driver terminal for the moving body,
When a part designation operation for designating a part in an operation-side terminal image that is an image displayed on the first information terminal is detected by the first information terminal, the part designation operation is performed notification information e second display control unit further Bei for controlling the display of the second information terminal to be displayed on the second information terminal,
The information processing apparatus , wherein the operation-side terminal image includes an image captured by a camera mounted on the moving body . The information processing apparatus according to claim 1, wherein the second display control unit configures the notification information according to a type of gesture operation performed as the partial designation operation. When the part specifying operation is a touch operation on a part of the operation side terminal image, the second display control unit adds a position object having a design representing a position to the part of the operation side terminal image . the configuration, constituting the notification information, the information processing apparatus according to claim 2. When the partial designation operation is a directional gesture operation that draws a directional trajectory in the portion of the operation-side terminal image, the second display control unit has a design that represents the direction of the directional gesture operation. and the configuration of the direction object was added to the portion of the operating-side terminal image, constituting the notification information, the information processing apparatus according to claim 2 or 3. A first display control unit for controlling display of the first information terminal;
The first information terminal is present in the moving body together with the second information terminal, and the first information terminal or the second information terminal is a driver terminal for the moving body,
When a part designation operation for designating a part in an operation-side terminal image that is an image displayed on the first information terminal is detected by the first information terminal, the part designation operation is performed notification information e second display control unit further Bei for controlling the display of the second information terminal to be displayed on the second information terminal,
The second display control unit configures the notification information according to a type of gesture operation performed as the partial designation operation,
When the part designation operation is a surrounding gesture operation that draws a locus so as to surround a part of the operation side terminal image, the second display control unit displays a range object having a design representing a range as the operation side terminal image. The information processing apparatus which comprises the said notification information by the structure added to the part . The information according to any one of claims 1 to 5 , wherein the second display control unit configures the notification information according to a content displayed in a part where the part specifying operation is performed. Processing equipment. A first display control unit for controlling display of the first information terminal;
The first information terminal is present in the moving body together with the second information terminal, and the first information terminal or the second information terminal is a driver terminal for the moving body,
When a part designation operation for designating a part in an operation-side terminal image that is an image displayed on the first information terminal is detected by the first information terminal, the part designation operation is performed notification information e second display control unit further Bei for controlling the display of the second information terminal to be displayed on the second information terminal,
The second display control unit configures the notification information according to the content displayed in the part where the part designation operation is performed,
When the part on which the part designation operation is performed is a display of the moving speed of the moving body, the second display control unit configures the notification information by a configuration having a content to call attention regarding the moving speed. An information processing apparatus. A first display control unit for controlling display of the first information terminal;
The first information terminal is present in the moving body together with the second information terminal, and the first information terminal or the second information terminal is a driver terminal for the moving body,
When a part designation operation for designating a part in an operation-side terminal image that is an image displayed on the first information terminal is detected by the first information terminal, the part designation operation is performed notification information e second display control unit further Bei for controlling the display of the second information terminal to be displayed on the second information terminal,
The second display control unit configures the notification information according to the content displayed in the part where the part designation operation is performed,
The operation side terminal image includes an image captured by a camera mounted on the moving body,
When the part on which the part designation operation has been performed is outside the lane where the moving body is located, the second display control unit displays a movement caution object for calling attention regarding movement in the operation side terminal image. An information processing apparatus that configures the notification information by a configuration that is added to a portion for which a partial designation operation has been performed . A first display control unit for controlling display of the first information terminal;
The first information terminal is present in the moving body together with the second information terminal, and the first information terminal or the second information terminal is a driver terminal for the moving body,
When a part designation operation for designating a part in an operation-side terminal image that is an image displayed on the first information terminal is detected by the first information terminal, the part designation operation is performed notification information e second display control unit further Bei for controlling the display of the second information terminal to be displayed on the second information terminal,
The second display control unit configures the notification information according to the content displayed in the part where the part designation operation is performed,
The operation side terminal image includes an image captured by a camera mounted on the moving body,
When the part where the part designation operation is performed is a facility, the second display control unit configures the notification information by a configuration in which a location of the facility is indicated on a map image . Prior Ki構 formed including text objects consisting of text asking the facility is what information processing apparatus according to claim 9. A first display control unit for controlling display of the first information terminal;
The first information terminal is present in the moving body together with the second information terminal, and the first information terminal or the second information terminal is a driver terminal for the moving body,
When a part designation operation for designating a part in an operation-side terminal image that is an image displayed on the first information terminal is detected by the first information terminal, the part designation operation is performed notification information e second display control unit further Bei for controlling the display of the second information terminal to be displayed on the second information terminal,
The information processing apparatus , wherein the second display control unit configures the notification information according to a situation where the moving body is located . A first display control unit for controlling display of the first information terminal;
The first information terminal is present in the moving body together with the second information terminal, and the first information terminal or the second information terminal is a driver terminal for the moving body,
When a part designation operation for designating a part in an operation-side terminal image that is an image displayed on the first information terminal is detected by the first information terminal, the part designation operation is performed notification information e second display control unit further Bei for controlling the display of the second information terminal to be displayed on the second information terminal,
In a case where the position of the part where the part designation operation is performed in the operation side terminal image changes with the change of the operation side terminal image, the second display control unit, in the notification information, the part designation operation An information processing apparatus that causes a display position of an object associated with a portion where the operation is performed to follow a position change of the portion . A first display control unit for controlling display of the first information terminal;
The first information terminal is present in the moving body together with the second information terminal, and the first information terminal or the second information terminal is a driver terminal for the moving body,
When a part designation operation for designating a part in an operation-side terminal image that is an image displayed on the first information terminal is detected by the first information terminal, the part designation operation is performed notification information e second display control unit further Bei for controlling the display of the second information terminal to be displayed on the second information terminal,
The information processing apparatus , wherein the first display control unit adds an object associated with a part where the part designation operation is performed in the notification information to the operation side terminal image . If pre-assigned notification information termination operation is detected by said first information terminal, the second display control unit terminates the display of the notification information to the second information terminal, according to claim 1 to 13 The information processing apparatus according to any one of the above. Wherein the second display control unit generates and transmits the notification information to the second information terminal, the information processing apparatus according to any one of claims 1-14. The second display control unit transmits the information required to create the notification information to the second information terminal, the second information terminal generates the notification information, one of claims 1-14 The information processing apparatus according to any one of claims. The information processing apparatus according to any one of claims 1 to 16 , wherein the first information terminal is not the driver terminal, and the second information terminal is the driver terminal. The information processing apparatus according to claim 17 , wherein the display unit of the second information terminal is an instrument panel provided in a driver's seat of the moving body. A first display control unit for controlling display of the first information terminal;
The first information terminal is present in the moving body together with the second information terminal, and the first information terminal or the second information terminal is a driver terminal for the moving body,
When a part designation operation for designating a part in an operation-side terminal image that is an image displayed on the first information terminal is detected by the first information terminal, the part designation operation is performed notification information e second display control unit further Bei for controlling the display of the second information terminal to be displayed on the second information terminal,
The information processing apparatus , wherein the first information terminal and the second information terminal share a display screen and display each image constituting a split view on the display screen . An information terminal comprising the information processing apparatus according to any one of claims 1 to 19 and functioning as the first information terminal. An information communication system comprising the information terminal according to claim 20 as the first information terminal and the second information terminal. The information processing apparatus according to any one of claims 1 to 19 ,
The first information terminal;
An information communication system comprising the second information terminal. (A) comprising a step of controlling display of the first information terminal;
The first information terminal is present in the moving body together with the second information terminal, and the first information terminal or the second information terminal is a driver terminal for the moving body,
(B) When the first information terminal detects a partial designation operation for designating a part in the operation-side terminal image that is an image displayed on the first information terminal, the partial designation operation is performed. notification information further example Bei the step of controlling the display of said second information terminal to be displayed on the second information terminal related parts,
The information providing method , wherein the operation-side terminal image includes an image captured by a camera mounted on the moving body . (A) comprising a step of controlling display of the first information terminal;
The first information terminal is present in the moving body together with the second information terminal, and the first information terminal or the second information terminal is a driver terminal for the moving body,
(B) When the first information terminal detects a partial designation operation for designating a part in the operation-side terminal image that is an image displayed on the first information terminal, the partial designation operation is performed. notification information further example Bei the step of controlling the display of said second information terminal to be displayed on the second information terminal related parts,
The step (b)
(B1) configuring the notification information in accordance with the type of gesture operation performed as the partial designation operation
The step (b1) includes:
When the part specifying operation is a surrounding gesture operation that draws a trajectory so as to surround a part of the operation-side terminal image, the configuration includes adding a range object having a design representing a range to the part of the operation-side terminal image. Steps for configuring notification information
A method of providing information. (A) comprising a step of controlling display of the first information terminal;
The first information terminal is present in the moving body together with the second information terminal, and the first information terminal or the second information terminal is a driver terminal for the moving body,
(B) When the first information terminal detects a partial designation operation for designating a part in the operation-side terminal image that is an image displayed on the first information terminal, the partial designation operation is performed. notification information further example Bei the step of controlling the display of said second information terminal to be displayed on the second information terminal related parts,
The step (a)
A step of adding an object associated with the part for which the part designation operation has been performed in the notification information to the operation side terminal image.
An information providing method.