JP7784236B2 - Video transmission device, server device, video transmission method, video transmission program, and storage medium - Google Patents

Description

The present invention relates to a video transmission device, a server device, a video transmission method, a video transmission program, a storage medium, and a data structure, for example, a video transmission device, a server device, a video transmission method, a video transmission program, a storage medium, and a data structure for providing users with video from a mobile object.

There are communication systems that communicate between an onboard device installed in a vehicle and an external terminal located outside the vehicle. For example, Patent Document 1 discloses a system in which a voice call is made between the driver of the vehicle and the operator of the external terminal, and while the voice call is being made, video data showing an image of the area in front of the vehicle is transmitted from the onboard device to the external terminal, and the video is displayed on the external terminal.

JP 2016-213791 A

In systems such as those described in Patent Document 1, it is desirable to reduce the amount of data communication when continuously transmitting video, for example. On the other hand, when a user of an external terminal is viewing video of the scene ahead of the vehicle and talking to the driver of the vehicle or giving directions, it is also important to allow the user of the external terminal to experience a sensation similar to that experienced by the driver when looking outside the vehicle.

The present invention was made in consideration of the above points, and one of its objectives is to provide a video transmission device, server device, video transmission method, video transmission program, storage medium, and data structure that can ensure high reproducibility of the scenery seen by the driver of a mobile vehicle when the operator of an external terminal views video transmitted from the mobile vehicle, while suppressing data communication volume.

The invention described in claim 1 is a video transmission device that travels with a moving object, comprising: a video acquisition unit that sequentially acquires video from an imaging device that captures video around the moving object; a location information acquisition unit that acquires location information of the moving object; a map information acquisition unit that acquires map information including points or sections on a road to which a frame rate is associated; and a video transmission unit that transmits the video at a frame rate based on the frame rate associated with the point or section on the road indicated by the location information.

The invention described in claim 8 is a video transmission device that travels with a moving object, comprising: a video acquisition unit that sequentially acquires video from an imaging device that captures video around the moving object; a location information transmission unit that acquires and transmits location information about the moving object; a frame rate receiving unit that receives information indicating a frame rate associated with a point or section on a road indicated by the location information; and a video transmission unit that transmits the video at a frame rate based on the frame rate indicated by the information received by the receiving unit.

The invention described in claim 9 is a server device characterized by having a location information receiving unit that receives location information of a mobile body from a video transmission device that travels with the mobile body, and a frame rate transmitting unit that identifies a frame rate associated with a point or section on a road indicated by the location information received by the receiving unit based on map information including points or sections on a road associated with frame rates, and transmits information indicating the identified frame rate to the video transmission device.

The invention described in claim 10 is a video transmission method executed by a video transmission device that travels with a moving object, comprising: a video acquisition step of sequentially acquiring video from an imaging device that captures video around the moving object; a location information acquisition step of acquiring location information of the moving object; a map information acquisition step of acquiring map information including points or sections on a road associated with frame rates; and a video transmission step of transmitting the video at a frame rate based on the frame rate associated with the point or section on the road indicated by the location information.

The invention described in claim 11 is a video transmission program that includes a computer and is executed by a video transmission device that moves with a mobile object, and that causes the computer to execute the following steps: a video acquisition step that sequentially acquires video from an imaging device that captures video around the mobile object; a location information acquisition step that acquires location information of the mobile object; a map information acquisition step that acquires map information including points or sections on a road to which a frame rate is associated; and a video transmission step that transmits the video at a frame rate based on the frame rate associated with the point or section on the road indicated by the location information.

The invention described in claim 12 is a computer-readable storage medium that stores a video transmission program that includes a computer and causes a video transmission device that travels with a moving object to execute the following steps: a video acquisition step that sequentially acquires video from an imaging device that captures video of the area around the moving object; a location information acquisition step that acquires location information of the moving object; a map information acquisition step that acquires map information including points or sections on a road to which a frame rate is associated; and a video transmission step that transmits the video at a frame rate based on the frame rate associated with the point or section on the road indicated by the location information.

The invention described in claim 13 is a data structure of map information used by a video transmission device equipped with a computer and traveling with a mobile body, the data structure including position data indicating a point or section on a road and frame rate data indicating a frame rate associated with the point or section on the road indicated by the position data, wherein the video transmission device acquires the frame rate associated with the point or section on the road corresponding to the position of the mobile body on the road, and transmits video at a frame rate based on the acquired frame rate.

1 is a diagram illustrating a video distribution system according to a first embodiment of the present invention. 1 is a perspective view of a front seat portion of a vehicle equipped with an in-vehicle device. FIG. 2 is a block diagram showing an example of the configuration of an in-vehicle device. FIG. 3 is a diagram illustrating an example of information included in map information according to the present embodiment. FIG. 3 is a diagram illustrating an example of information included in map information according to the present embodiment. FIG. 3 is a diagram illustrating an example of information included in map information according to the present embodiment. FIG. 2 is a block diagram illustrating an example of a configuration of a server. FIG. FIG. 2 is a block diagram showing an example of the configuration of an external device. FIG. 4 is a flow diagram of an operation routine of the in-vehicle device. FIG. 4 is a flow diagram of an operation routine of the in-vehicle device. FIG. 4 is a flow diagram of an operation routine of the in-vehicle device. FIG. 4 is a flow diagram of the operation routine of the server.

1. System Configuration
A video distribution system 100 according to a first embodiment of the present invention will be described below with reference to the accompanying drawings.

Figure 1 shows a video distribution system 100 according to a first embodiment of the present invention. As shown in Figure 1, the video distribution system 100 includes an in-vehicle device 10, a server 40, and an external device 70. Note that Figure 1 shows a case in which the in-vehicle device 10 is mounted on an automobile M, which is an example of a mobile body. Also, Figure 1 shows a smartphone as an example of the external device 70.

The in-vehicle device 10, server 40, and external device 70 can send and receive data to and from each other via a network NW using communication protocols such as TCP/IP or UDP/IP. The network NW can be constructed using internet communication, including wireless communication such as a mobile communication network or Wi-Fi (registered trademark), and wired communication.

In the video distribution system 100 of this embodiment, a voice call is established between the in-vehicle device 10 and the external device 70, and then the video captured in the automobile M is distributed from the in-vehicle device 10 to the external device 70. In the following description, the communication mode in which a voice call is established between the in-vehicle device 10 and the external device 70 while the video captured in the automobile M is distributed from the in-vehicle device 10 to the external device 70 as described above is referred to as video communication. In this embodiment, the in-vehicle device 10 and the external device 70 perform video communication via the server 40.

By performing this type of video communication, the user of the external device 70 viewing the video transmitted from the in-vehicle device 10 can experience the sensation of being a passenger in the vehicle M with the driver of the vehicle M. In other words, video communication can be used to realize a virtual passenger ride with the user of the external device 70 in the vehicle M. A system that realizes this type of video communication is referred to as a virtual passenger ride system. In the following, Example 1, an example is described in which the in-vehicle device 10 is a car navigation device. In addition, Example 1 also describes an example in which the in-vehicle device 10 is a terminal device of a so-called cloud-based car navigation device that receives from the user a destination to which the user wishes to be guided, transmits the destination to the server 40, and the server 40 generates a route to the destination.

Figure 2 is a perspective view showing the area near the front seat of an automobile M equipped with an in-vehicle device 10 as a video transmission device. Figure 1 shows an example of an installation in which the in-vehicle device 10 is installed in the dashboard DB of the front seat of the automobile M.

The GPS receiver 11 is a device that receives signals (GPS signals) from GPS (Global Positioning System) satellites. The GPS receiver 11 is located, for example, on a dashboard database. The GPS receiver 11 may be located anywhere as long as it can receive GPS signals. The GPS receiver 11 is capable of transmitting the received GPS signals to the in-vehicle device 10.

The exterior camera 12 serving as the imaging unit is an imaging device that captures images in front of the automobile M. In this embodiment, the exterior camera 12 is mounted on the dashboard DB so that the imaging direction is toward the front. For example, the exterior camera 12 is a wide-angle camera that can capture images of a wide area in front of the automobile M through the windshield. The exterior camera 12 may also be mounted on the rearview mirror RM or on the inside of the windshield FG.

The in-vehicle camera 13 is an imaging device that captures images of the interior of the vehicle M. In this embodiment, the in-vehicle camera 13 is installed on the top edge of the windshield FG or on the ceiling near the top edge, and is capable of capturing images of the driver of the vehicle M.

During video communication, video captured by the exterior camera 12 or the interior camera 13 is distributed to the external device 70. The following mainly describes the case where video from the exterior camera 12 is distributed to the external device 70.

The touch panel 14 is, for example, a touch panel monitor that combines a display, such as a liquid crystal display capable of displaying images, with a touchpad. The touch panel 14 is disposed, for example, on the center console CC of the dashboard DB. The touch panel 14 may be disposed in a location that is visible to the driver and within the driver's reach. For example, the touch panel 14 may be attached to the dashboard DB.

The touch panel 14 is capable of displaying a screen based on the control of the in-vehicle device 10. The touch panel 14 is also capable of transmitting signals representing input operations received from the user onto the touch panel 14 to the in-vehicle device 10. For example, the touch panel 14 may display car navigation guidance. Furthermore, operations related to the car navigation function, such as setting a destination, may be possible via the touch panel 14.

In addition, the touch panel 14 may display information related to video communication, or may display an operation reception screen for connecting for video communication (hereinafter also referred to as video connection). The occupants of the automobile M may perform the video communication connection operation by inputting information into the touch panel 14.

The speaker 15 is provided, for example, on the interior side of the A-pillar AP. The speaker 15 is capable of emitting sounds such as music and voices under the control of the in-vehicle device 10. During video communication, the speaker 15 emits sound from the external device 70 during a voice call.

Microphone 17 is a microphone device that picks up sounds inside the vehicle and is located, for example, on the dashboard DB. Microphone 17 may be located anywhere, such as on the rearview mirror RM or steering wheel, as long as it can pick up sounds inside the vehicle. During video communication, the audio picked up by microphone 17 is transmitted to external device 70 as audio for a voice call.

Figure 3 is a block diagram showing the configuration of the in-vehicle device 10. For example, the in-vehicle device 10 is a device in which a large-capacity storage device 23, a control unit 25, an input unit 27, an output unit 29, an encoder unit 30, and a data communication unit 31 cooperate with each other via a system bus 21.

The large-capacity storage device 23 is composed of, for example, a hard disk drive, a solid state drive (SSD), flash memory, etc., and stores various programs such as the operating system and terminal software.

The various programs may be obtained, for example, from another server device via a network, or may be recorded on a recording medium and read via various drive devices. In other words, the various programs stored in the mass storage device 23 (including programs for executing processes in the in-vehicle device 10, described below) can be transmitted via a network, or recorded on a computer-readable recording medium and transferred.

The large-capacity storage device 23 also has a map information database (shown as map information DB in the figure) 23A. The map information database 23A stores map information, including road maps. The map information includes location data, which is information indicating points or sections on roads.

In this map information, for example, points and sections on roads are represented by, for example, nodes on the roads and links on the roads. Nodes indicate intersections and dead-end points in the road network. Links indicate sections between nodes on the roads. The map information database 23A can be used, for example, to generate routes for route guidance.

In this embodiment, the in-vehicle device 10 transmits video at a frame rate based on the frame rate associated with the link or node on which the vehicle M is located. In the following description, the frame rate associated with the position data indicating a link or node on a road in the map information is also referred to as the "video transmission frame rate."

For example, the frame rate associated with a link is determined in advance so that the impression given to the user of the external device 70 when the user of the external device 70 views the image acquired from the automobile M located at that link and displayed on the external device 70 is closer to the impression given to the driver of the automobile M viewing the same scene.

Figure 4 shows an information table included in the map information stored in the map information database 23A. In the example shown in Figure 4, a frame rate is associated with each link on a road. As shown in Figure 4, in the map information of the map information database 23A, frame rate data is associated with position data indicating the link.

The frame rate data indicates a frame rate that is predetermined and associated with a link on a road as a frame rate at which video can be properly viewed when the video is captured from a vehicle located on that link.

The frame rate associated with location data in map information is, for example, a recommended frame rate for each link. Furthermore, in the map information, for example, a lower limit of the allowable frame rate may be associated with each link, or a range of allowable frame rates may be associated with each link.

Figure 5 shows an example of map information in which a frame rate is associated with each node. As shown in Figure 5, node X and node Y are associated with a recommended frame rate when traveling in the uphill direction and a recommended frame rate when traveling in the downhill direction. In other words, when a frame rate is associated with each node, the node becomes a point where the frame rate should be switched.

Note that Figure 6 shows the positional relationship between links A to C and nodes X and Y shown as examples in Figures 4 and 5. As shown in Figure 6, node X is a node between links A and B, and node Y is a node between links B and C. As shown in Figure 6, if the direction from link A to link B is considered the upstream direction, it is recommended that, for example, at upstream node X, the frame rate associated with link A is switched from 15 fps to 20 fps, which is associated with link B.

As described above, the example shown in Figure 4 describes a case where a frame rate is associated with each link, and the example shown in Figure 5 describes a case where a frame rate is associated with each node. Note that this embodiment is not limited to associating a frame rate with a link or node; for example, a frame rate may be associated with each region (area) on a map that indicates an area including roads.

The control unit 25 is composed of a CPU (Central Processing Unit) 25A, ROM (Read Only Memory) 25B, RAM (Random Access Memory) 25C, etc., and functions as a computer. The CPU 25A reads and executes various programs stored in the ROM 25B and the large-capacity storage device 23, thereby realizing various functions. In this embodiment, the control unit 25 performs functions such as video distribution and car navigation.

The input unit 27 is an interface unit that communicatively connects the in-vehicle device 10 with the exterior camera 12, interior camera 13, touch panel 14, and microphone 17. The in-vehicle device 10 can sequentially acquire images captured by the exterior camera 12 and interior camera 13 via the input unit 27. In other words, the control unit 25 functions as an image acquisition unit that sequentially acquires image data showing images of the surroundings of the vehicle captured by the exterior camera 12 via the input unit 27.

The in-vehicle device 10 can receive signals indicating input operations to the touch panel 14 via the input unit 27. For example, the in-vehicle device 10 can accept connection requests made by the user via the touch panel 14 and microphone 17, or destination setting inputs for car navigation, via the input unit 27.

The input unit 27 is an interface unit that communicatively connects the in-vehicle device 10 and the GPS receiver 11. The in-vehicle device 10 receives GPS signals from the GPS receiver 11 via the input unit 27 and can acquire information about the current position of the in-vehicle device 10, i.e., the current position of the automobile M in this embodiment, from the GPS signals. In other words, the control unit 25 functions as a location information acquisition unit that acquires location information of the automobile M from the GPS receiver 11.

The output unit 29 is communicatively connected to the touch panel 14 and speaker 15, and can transmit video or image signals to the touch panel 14 to display the image, or transmit audio signals to the speaker 15 to output sound.

The encoder unit 30 encodes (hereinafter also referred to as encoding processing) the video captured by the exterior camera 12 or the interior camera 13 (also referred to as captured video) based on instructions from the control unit 25. The encoder unit has a CPU for video encoding, known as a GPU, and encoding may be performed by the GPU.

The encoder unit 30 encodes the captured video using, for example, an MPEG-4 standard encoding method to generate encoded data as video encoded data. For example, the encoder unit 30 generates encoded data from the captured video using a codec such as H.264, Xvid, DivX, VP8, or VP9.

Furthermore, based on instructions from the control unit 25, the encoder unit 30 encodes the video captured by the exterior camera 12 at the frame rate specified by the control unit 25. The control unit 25 reads map information from the map information database 23A in the mass storage device 23. In other words, the control unit 25 functions as a map information acquisition unit that acquires map information including points or sections on roads to which frame rates are linked.

As described above, in the map information of the map information database 23A, a video transmission frame rate is associated with position data indicating a point or section on a road. Based on the read map information, the control unit 25 obtains a video transmission frame rate corresponding to the position of the automobile M, i.e., a frame rate associated with the point or section on a road indicated by the position information of the automobile M, and instructs the encoder unit 30 to encode the video at a frame rate based on the obtained frame rate.

The data communication unit 31 is connected to the above-mentioned network NW and transmits and receives various data to and from the server 40. The data communication unit 31 also transmits and receives various data to and from the external device 70 via the server 40.

For example, the control unit 25 of the in-vehicle device 10 can transmit location identification information as location data that can identify the current location of the in-vehicle device 10, i.e., information on the current location of the automobile M in this embodiment, to the server 40 via the data communication unit 31. Also, for example, the control unit 25 can transmit information including a destination input by the user to the server 40 via the data communication unit 31, and receive route information or navigation information to the destination from the server 40.

The control unit 25 can transmit audio data of the audio picked up by the microphone 17 to the external device 70 via the data communication unit 31 for audio calls in video communication. The control unit 25 can also receive audio data of the audio input to the external device 70 via the data communication unit 31 for audio calls in video communication.

The control unit 25 also transmits the encoded data encoded by the encoder unit 30 to the external device 70 via the data communication unit 31. For example, the control unit 25 buffers the encoded data and transmits it to the external device 70 via the data communication unit 31.

The control unit 25, encoder unit 30, and data communication unit 31 cooperate in the transmission process of encoded data during video connection in video communication, functioning as a video transmission unit 32 that transmits video at a frame rate based on the frame rate associated with the point or section on the road indicated by the position information of the vehicle M.

In this way, the in-vehicle device 10 transmits video at a frame rate associated with the road location where the vehicle M is located, based on the map information. In other words, it generates video data for transmission at a frame rate associated with the road location, i.e., it encodes and transmits the video at the frame rate associated with the road location.

This enhances the sense of sharing the video between the occupants of the vehicle M and the user of the external device 70, allowing for smooth information sharing.

When distributing video captured by the exterior camera 12 to the external device 70 via the server 40, the in-vehicle device 10 transmits the video at a low frame rate to reduce communication volume. For example, if the exterior camera 12 captures video at 27.5 fps, the communication volume can be reduced by lowering the frame rate to, for example, around 15 fps.

If the frame rate of the transmitted video is set low in this way, the user of the external device 70 (hereinafter simply referred to as the "external user") may feel uncomfortable when viewing the video sent from the in-vehicle device 10.

Specifically, if the frame rate is low, the external user may feel that the sense of speed visually obtained from the video, or the so-called perceived speed, is faster than the actual speed of the automobile M. Due to the difference between the actual perceived speed of the occupants of the automobile M (hereinafter simply referred to as the occupants) and the perceived speed of the external user, there is a risk that the sense of sharing the video between the occupants and the external user will be weakened. Furthermore, in such a case, there is a possibility that the external user will feel frightened, for example.

Such differences in perceived speed, etc. may also be caused by differences in the scenery surrounding the point or section of the road where vehicle M is located.

For example, if an external user views video captured at a low frame rate in a location where there are many objects such as fences and buildings near the road on which automobile M is traveling, for example to the right or left of the road in front of automobile M, the external user may perceive the speed to be faster than the perceived speed of the occupants of automobile M, or may perceive greater shaking. On the other hand, if an external user views video captured at a low frame rate in a location where there are few objects such as buildings near the road on which automobile M is traveling and where the scenery does not change much, the perceived speed of the external user may be the same as the perceived speed of the occupants.

In this embodiment, as described in Figures 4 to 6, the in-vehicle device 10 pre-associates appropriate frame rates with points or sections on a road (hereinafter also simply referred to as positions on a road), and uses map information in which the frame rates associated with the positions on the road are associated with position data indicating the positions on the road. For example, in the map information of this embodiment, points or sections that have a large impact on perceived speed are associated with frame rates that are larger (higher) than points or sections that have a small impact on perceived speed. Also, for example, points or sections with beautiful scenery may be associated with frame rates that are larger (higher) than other points or sections.

By using such map information to transmit video at a frame rate corresponding to the link on which automobile M is located, data communication volume can be reduced while ensuring high reproducibility of the scenery seen by the driver of the mobile vehicle when the operator of the external terminal views the video transmitted from the mobile vehicle.

Figure 7 is a block diagram showing the configuration of the server 40. For example, the server 40 is a device in which a large-capacity storage device 43, a control unit 45, and a data communication unit 47 cooperate via a system bus 41. The server 40 has functions similar to a SIP server, establishing a voice call between the in-vehicle device 10 and the external device 70 during video communication and transferring data from the voice call.

The server 40 also has the function of receiving from the in-vehicle device 10 the location information of the vehicle M and information on a destination set by a user who is an occupant of the vehicle M, and generating a route to the destination based on the location information and the destination information.

The server 40 also has the function of transferring encoded data sent from the in-vehicle device 10 to the external device 70.

The large-capacity storage device 43 is composed of, for example, a hard disk drive and an SSD (solid state drive), and stores various programs such as the operating system and software for the server 40.

The large-capacity storage device 43 also includes a map information database (shown as map information DB in the figure) 43A, which stores map information including road maps. The map information in the map information database 43A is a database containing information equivalent to the map information used in a navigation device, for example. Furthermore, the map information in the map information database 43A may contain information indicating the video transmission frame rate in addition to information equivalent to the map information used in a navigation device. That is, in the map information database 43A, position data indicating points or sections on a road may be associated with a video transmission frame rate such as that described in FIG. 4.

The control unit 45 is composed of a CPU (Central Processing Unit) 45A, ROM (Read Only Memory) 45B, RAM (Random Access Memory) 45C, etc., and functions as a computer. The CPU 45A reads and executes various programs stored in the ROM 45B and the large-capacity storage device 43, thereby realizing various functions.

The data communication unit 47 is connected to the network NW and transmits and receives various data between the in-vehicle device 10 and the external device 70.

The control unit 45 acquires location information indicating the current location of the automobile M from the in-vehicle device 10 via the data communication unit 47. The control unit 45 also acquires destination information entered into the in-vehicle device 10 by an occupant of the automobile M from the in-vehicle device 10 via the data communication unit 47. The control unit 45 generates a route to the destination based on the location information and destination information, and transmits information indicating the route to the in-vehicle device 10.

The control unit 45 also transfers the encoded data and audio received from the in-vehicle device 10 to the external device 70 via the data communication unit 47. The control unit 45 also transfers the audio received from the external device 70 to the in-vehicle device 10 via the data communication unit 47.

The control unit 45 may also acquire location information indicating the current location of the vehicle M from the in-vehicle device 10, and transmit the video transmission frame rate associated with the acquired location information to the in-vehicle device 10.

Figure 8 is a front view showing the appearance of the external device 70. As described above, in Example 1, the external device 70 is a smartphone.

The touch panel 71 is, for example, a touch panel monitor that combines a display, such as a liquid crystal display, capable of displaying images, with a touchpad. The touch panel 71 is capable of generating signals that represent input operations received from the user onto the touch panel 71. In this embodiment, the touch panel 71 displays images distributed from the in-vehicle device 10.

In addition, the touch panel 71 may display information about video communication or a screen for accepting operations to establish a video connection. The user of the external device 70 may perform video communication connection operations by inputting information to the touch panel 71.

The speaker 73 is capable of emitting sounds such as music and voice. During video communication, the speaker 73 emits sound from the in-car device 10 during a voice call.

The microphone 75 is a microphone device that receives sounds emitted toward the external device 70. During video communication, the sound picked up by the microphone 75 is transmitted to the external device 70 as the sound of the voice call.

Figure 9 is a block diagram showing the configuration of external device 70. For example, external device 70 is a device in which a large-capacity storage device 83, a control unit 84, an input unit 85, an output unit 86, a data communication unit 87, and a decoder unit 88 cooperate via a system bus 81.

The large-capacity storage device 83 is composed of, for example, a hard disk drive, SSD (solid state drive), flash memory, etc., and stores various programs such as the operating system and terminal software.

The various programs may be obtained, for example, from another server device via a network, or may be recorded on a recording medium and read via various drive devices. In other words, the various programs stored in the mass storage device 83 can be transmitted via a network, or recorded on a computer-readable recording medium and transferred.

The control unit 84 is composed of a CPU (Central Processing Unit) 84A, ROM (Read Only Memory) 84B, RAM (Random Access Memory) 84C, etc., and functions as a computer. The CPU 84A reads and executes various programs stored in the ROM 84B and the large-capacity storage device 83, thereby realizing various functions.

The input unit 85 is an input interface for the touch panel 71 and microphone 75. The control unit 84 can receive signals indicating input operations on the touch panel 71 and audio input signals from the microphone 75 via the input unit 85. For example, the control unit 84 can accept connection requests made by the user via the touch panel 71 and microphone 75 via the input unit 85.

The output unit 86 is an output interface to the touch panel 71 and speaker 73. The control unit 84 can send video or image signals via the output unit 86 to the touch panel 14 to display the image, or send audio signals to the speaker 15 to output sound.

The data communication unit 87 is connected to the above-mentioned network NW and transmits and receives various data to and from the server 40. The data communication unit 87 also transmits and receives various data to and from the in-vehicle device 10 via the server 40, including encoded data of video transmitted from the in-vehicle device 10. For example, the control unit 84 functions as an encoded data receiving unit that receives encoded data via the data communication unit 87.

For example, the control unit 84 of the external device 70 can receive location information capable of identifying the current location of the automobile M transmitted from the in-vehicle device 10 from the server 40 via the data communication unit 31. Furthermore, for example, the control unit 84 can receive route information or navigation information for the automobile M from the server 40 via the data communication unit 87.

The control unit 84 can also transmit audio data of the audio picked up by the microphone 75 to the in-vehicle device 10 via the data communication unit 87 for audio calls in video communication. The control unit 84 can also receive audio data transmitted from the in-vehicle device 10 via the data communication unit 87 for audio calls in video communication.

The decoder unit 88, which serves as a video output unit, decodes, plays, and outputs encoded data received from the in-vehicle device 10 based on commands from the control unit 84. The decoder unit 88 decodes the encoded data using the codec used when encoding the encoded data, for example, an encoding method conforming to the MPEG-4 standard, and plays and outputs the video. The played-back video is then displayed on the touch panel 71 by the control unit 84.

In this way, the external device 70 receives and displays the video transmitted by the in-vehicle device 10.

2. Operation of the video distribution system
The operation of the video distribution system 100 including the in-vehicle device 10, the server 40, and the external device 70 will be described below.

The control routines of the in-vehicle device 10 and the external device 70 for realizing the operation of the video distribution system 100 of Example 1 will be described. Hereinafter, it will be explained that when video distribution begins via video communication, the video transmission unit 32 begins encoding and transmitting the video at a low frame rate set to reduce communication traffic.

Figure 10 is a flowchart showing the video distribution routine RT1 executed by the control unit 25 of the in-vehicle device 10. For example, when the in-vehicle device 10 is powered on, the control unit 25 starts the video distribution routine RT1 and executes it repeatedly.

First, the control unit 25 determines whether the in-car device 10 has started video communication with the external device 70 (step S101). This determination is made, for example, based on whether a video communication has been initiated in the in-car device 10 and communication has been established with the external device 70. If the control unit 25 determines that video communication has not started (step S101: NO), it ends routine RT1.

When the control unit 25 determines that the in-vehicle device 10 has started video communication (step S101: YES), it acquires location information indicating the location of the automobile M based on the signal from the GPS receiver 11, acquires video from the exterior camera 12, encodes the acquired video, and distributes it together with the location information to the external device 70 via the server 40, starting a video distribution operation (step S102).

In other words, in step S102, the control unit 25 functions as an image acquisition unit that sequentially acquires images from the exterior camera 12. Also in step S102, the control unit 25 functions as a location information acquisition unit that acquires location information indicating the location of the vehicle M. Also in step S102, the control unit 25 functions as a data transmission unit that encodes the image data at a default frame rate and transmits the encoded data. Note that in step S102, the location information does not necessarily have to be transmitted to the external device 70.

When step S102 is executed and the video distribution operation is started, the control unit 25 determines whether the video communication between the in-car device 10 and the external device 70 has ended (step S103). If the control unit 25 determines that the video communication has not ended, i.e., that the video communication is continuing (step S103: NO), it repeatedly executes step S103. In other words, the video distribution operation continues as long as the video communication does not end.

When the control unit 25 determines in step S103 that video communication with the external device 70 has ended, it ends the video distribution operation (step S104), and routine R1 ends. In other words, when the control unit 25 determines that video communication with the external device 70 has ended, it ends the acquisition of location information, the acquisition of video, and the encoding and transmission of that video.

Figure 11 is a flowchart showing the frame rate control routine RT2 executed by the control unit 25 of the in-vehicle device 10. The control unit 25 starts the frame rate control routine RT2 when the in-vehicle device 10 is powered on, for example, and executes it repeatedly. Note that in this embodiment, it is assumed that when video distribution operation starts, encoding is initially performed at a default frame rate and the video is transmitted.

First, the control unit 25 determines whether the in-car device 10 is performing video communication with the external device 70 to distribute video (step S201).

When the control unit 25 determines that the in-vehicle device 10 is performing video communication video distribution operations (step S201: YES), it acquires location information indicating the current location of the vehicle M based on a signal from the GPS receiver 11 (step S202). Note that if the control unit 25 has acquired location information in step S102 of the video distribution routine RT1, it may use the acquired location information in step S202.

After executing step S202, the control unit 25 references the map information in the map information database 23A and obtains the frame rate associated with the point or section on the road indicated by the position information of the automobile M (step S203). In step S203, the control unit 25 obtains the frame rate associated as the video transmission frame rate with the position data in the map information corresponding to the position information of the automobile M.

After executing step S203, the control unit 25 sets the frame rate of the video being distributed to the frame rate acquired in step S203 (step S204). In step S204, the control unit 25 instructs the encoder unit 30 to encode the video at the acquired frame rate.

In step S102 of the video distribution routine RT1 and steps S203 and S204 of the frame rate control routine RT2, the control unit 25, encoder unit 30, and data communication unit 31 cooperate in the video transmission process to function as a video transmission unit 32 that transmits video at a frame rate based on the frame rate associated with the point or section on the road indicated by the position information of the vehicle M.

Step S102 of the video distribution routine RT1 corresponds to the location information acquisition step and video acquisition step in the video transmission method, program, and storage medium of the present invention. Furthermore, step S203 of the frame rate control routine RT2 corresponds to the map information acquisition step in the video transmission method, program, and storage medium of the present invention. Step S102 of the video distribution routine RT1 and step S204 of the frame rate control routine RT2 correspond to the video transmission step in the video transmission method, program, and storage medium of the present invention.

If the control unit 25 determines in step S201 that video communication video distribution operation is not being performed (step S201: NO), or after executing step S204, it terminates the frame rate control routine RT2.

In step S204 of the frame rate control routine RT2, the control unit 25 may set the frame rate of the video being distributed to a frame rate based on the frame rate obtained in step S203. For example, in step S204, the frame rate of the video being distributed may be set to a frame rate equal to or higher than the frame rate obtained in step S203. In step S204, for example, the frame rate may be set to a frame rate obtained by adding a predetermined value to the frame rate obtained in step S203 (e.g., adding 5 fps).

For example, in step S204, the control unit 25 may not change the frame rate setting if the frame rate of the video being distributed is equal to or higher than the frame rate acquired in step S203. Furthermore, if the frame rate acquired in step S203 is lower than the initial frame rate, the control unit 25 may set the frame rate to the initial value in step S204.

According to the video distribution routine RT1 and the frame rate control routine RT2, the in-vehicle device 10 sequentially acquires video from the exterior camera 12 that captures video around the automobile M, acquires location information for the automobile M, acquires map information including points or sections on roads to which frame rates are linked, and transmits the video captured by the exterior camera 12 at a frame rate based on the frame rate linked to the points or sections on roads indicated by the location information. The transmitted video is displayed on the external device 70 via the server 40.

In the map information, a frame rate is associated with position data indicating a point or section on a road. The frame rate is predetermined and associated with each point or section on a road so that the video is displayed appropriately. By setting the frame rate of the video to be transmitted to a frame rate based on the frame rate corresponding to the point or section on the road where the automobile M is located, the in-vehicle device 10 can always transmit video at an appropriate frame rate. Therefore, the in-vehicle device 10 can transmit video appropriately without, for example, a frame rate that is too low, which would impair the sense of sharing the video between the occupants of the automobile M and the user of the external device 70.

Therefore, it is possible to provide a video transmission device, video transmission method, video transmission program, and recording medium that can ensure high reproducibility of the scenery seen by the driver of a mobile vehicle when the operator of an external terminal views the video transmitted from the mobile vehicle, while suppressing data communication volume.

With reference to Figures 12 and 13, a modified version of the frame rate control routine RT2 will be described, in which the server 40 identifies a frame rate suitable for video transmission based on the position of the vehicle M, and the in-vehicle device 10 controls the frame rate based on the frame rate received from the server. Figures 12 and 13 will be described assuming that in the map information database 43A, the frame rate for video transmission is associated with position data indicating links on roads, as shown in Figure 4, for example.

Figure 12 is a flowchart showing the frame rate control routine RT3 executed by the control unit 25 of the in-vehicle device 10. The control unit 25 starts the frame rate control routine RT3 when the in-vehicle device 10 is powered on, for example, and executes it repeatedly. Note that, as with the frame rate control routine RT2, when video distribution operation starts, the encoding process is initially performed at the default frame rate and the video is transmitted.

First, the control unit 25 determines whether the in-car device 10 is performing video communication with the external device 70 to distribute video (step S301).

When the control unit 25 determines that the in-vehicle device 10 is performing video communication video distribution operations (step S301: YES), it acquires location information indicating the current location of the automobile M based on a signal from the GPS receiver 11 and transmits the information to the server 40 (step S302). In step S302, the control unit 25 functions as a location information transmission unit that acquires and transmits location information of the mobile object.

In step S302, for example, the control unit 25 transmits location information intermittently and continuously, for example, approximately once per second. Note that if the control unit 25 has acquired location information in step S102 of the video distribution routine RT1, the control unit 25 may use the acquired location information in step S302.

After executing step S302, the control unit 25 waits to receive a frame rate from the server 40 (step S303). In step S303, the control unit 25 waits to receive a frame rate associated with a point or section on a road indicated by the position information of the automobile M.

After executing step S303, the control unit 25 determines whether a frame rate has been received from the server 40 (step S304). In step S304, the control unit 25 functions as a frame rate receiving unit that receives information indicating a frame rate associated with a point or section on a road indicated by the location information.

If the control unit 25 determines in step S304 that the frame rate has not been received (step S304: NO), it repeats step S304 and determines again whether the frame rate has been received.

If the control unit 25 determines in step S304 that it has received a frame rate (step S304: YES), it sets the frame rate of the video being distributed to the frame rate received in step S304 (step S305). In step S305, the control unit 25 instructs the encoder unit 30 to encode the video at the received frame rate.

In step S305, the control unit 25 may set the frame rate of the video being distributed to a frame rate based on the frame rate received in step S304. For example, in step S305, the frame rate of the video being distributed may be set to a frame rate equal to or higher than the frame rate received in step S304.

If the control unit 25 determines in step S301 that video communication video distribution operation is not being performed (step S301: NO), or after executing step S305, it terminates the frame rate control routine RT3.

Figure 13 is a flowchart showing the frame rate transmission routine RT4 executed by the control unit 45 of the server 40. For example, when the server 40 is powered on, the control unit 45 starts the frame rate transmission routine RT4 and executes it repeatedly.

First, the control unit 45 determines whether the in-vehicle device 10 is performing video communication with the external device 70 to distribute moving images (step S401). In step S401, the control unit 45 determines that the in-vehicle device 10 is performing video communication with the external device 70 to distribute moving images (video communication) if the control unit 45 is performing a process of transferring encoded data received from the in-vehicle device 10 to the external device 70 via the data communication unit 47.

When the control unit 45 determines that the in-vehicle device 10 is performing video communication video distribution operations (step S401: YES), it waits for location information indicating the current location of the automobile M from the in-vehicle device 10 (step S402).

After executing step S402, the control unit 45 determines whether or not location information of the automobile M has been received from the in-vehicle device 10 (step S403). In step S403, for example, the control unit 45 determines that location information has been received if new location information of the automobile M has been received via the data communication unit 47. In step S403, the control unit 45 functions as a location information receiving unit that receives location information of the automobile M from the in-vehicle device 10 that is moving together with the automobile M.

In step S403, if the control unit 45 determines that location information has not been received (step S403: NO), it repeats step S403 and determines again whether location information has been received.

In step S403, the control unit 45 determines that location information has been received (step S403: YES), and then references the map information in the map information database 43A to identify the frame rate associated with the point or section on the road indicated by the location information of the automobile M (step S404). In step S404, the control unit 45 identifies the frame rate associated as the video transmission frame rate with the location data in the map information corresponding to the location information of the automobile M received in step S403.

After executing step S404, the control unit 45 transmits the frame rate determined in step S404 to the in-vehicle device 10 (step S405). In step S405, the control unit 45 transmits the frame rate to the in-vehicle device 10 via the data communication unit. In step S405, the control unit 45 functions as a frame rate transmission unit.

If the control unit 45 determines in step S401 that the in-vehicle device 10 is performing video communication video distribution operations (step S401: NO), or after executing step S405, it terminates the frame rate transmission routine RT4.

In step S102 of the video distribution routine RT1, steps S404 and S405 of the frame rate transmission routine RT4, and step S305 of the frame rate control routine RT3, the control unit 25, encoder unit 30, and data communication unit 31 of the in-vehicle device 10 cooperate in the video transmission process to function as a video transmission unit 32 that transmits video at a frame rate based on the frame rate associated with the point or section on the road indicated by the position information of the vehicle M.

According to the frame rate control routine RT3 and the frame rate transmission routine RT4,
Based on the map information held by the server 40, the in-vehicle device 10 can obtain a frame rate associated with a point or section on the road corresponding to the position of the automobile M on the road, and transmit the video captured by the exterior camera 12 at a frame rate based on the obtained frame rate.

Therefore, even if the server 40 has map information in which frame rate data is associated with position data, video can be transmitted at an appropriate frame rate.

Therefore, it is possible to provide a video transmission device, video transmission method, video transmission program, and recording medium that can ensure high reproducibility of the scenery seen by the driver of a mobile vehicle when the operator of an external terminal views the video transmitted from the mobile vehicle, while suppressing data communication volume.

In the above embodiment, the in-vehicle device 10 is an in-vehicle navigation device, but the in-vehicle device 10 can be various terminals such as a smartphone or tablet.

The configurations, routines, etc. of the in-vehicle device 10, server 40, and external device 70 in the above-described embodiment are merely examples and can be selected or modified as appropriate depending on the application, etc.

In addition, in the above embodiment, video communication is performed between the in-vehicle device 10 and the external device 70 via the server 40, but video communication may also be performed directly between the in-vehicle device 10 and the external device 70 via P2P (Peer to Peer) communication, etc.

In the above embodiment, the in-vehicle device 10 is a car navigation device connected to the touch panel 14, but this is not limited to this. For example, the in-vehicle device 10 may communicate with a smartphone carried by the driver of the automobile M and display a screen related to video communication on the smartphone's display instead of the touch panel 14. Furthermore, the in-vehicle device 10 may be configured not to display a screen to be presented to the driver of the automobile M.

For example, the in-vehicle device 10 may have a configuration similar to that of a drive recorder. Therefore, the in-vehicle device 10 may be integrated with, for example, an exterior camera 12. Specifically, the in-vehicle device 10 may be a device in which hardware that performs the functions of the in-vehicle device 10 is built into the housing of the exterior camera 12. In this case, the in-vehicle device 10 may not provide the various display outputs described above.

Furthermore, the video transmission device of the present application may be configured by integrating a terminal device (e.g., FIG. 3) having a configuration similar to the in-vehicle device 10 in the above-described embodiment, an outside camera 12, and a display or touch panel.

In the above embodiment, an example was described in which the image from the exterior camera 12 was transmitted to the external device 70. However, the image transmitted to the external device 70 may be switched to the image captured by the in-vehicle camera 13. When the image captured by the in-vehicle camera 13 is being transmitted, the user of the external device 70 can, for example, communicate with the driver of the vehicle M while viewing the interior of the vehicle M.

The switching operation to switch the video to be transmitted to the external device 70 between the video from the exterior camera 12 and the video from the interior camera 13 may be performed by the in-vehicle device 10. Alternatively, the switching operation may be performed remotely by operating the external device 70.

In the above embodiment, the control unit 84 of the external device 70 receives the video of the outside of the automobile M transmitted from the in-vehicle device 10 via the server 40. However, in addition to the video, it may also receive a map image showing the current location and planned travel route of the automobile M, the name of the driver of the automobile M, the traveling speed of the automobile M, etc.

For example, the control unit 84 of the external device 70 may display on the touch panel 71 a map image received from the in-vehicle device 10, the name of the driver of the vehicle M, and the traveling speed of the vehicle M superimposed on the image of the exterior of the vehicle M, or may display them in a display area separate from the image of the exterior of the vehicle M. In this case, the operator of the external device 70 may be able to freely switch between displaying and hiding the map image, the name of the driver of the vehicle M, and the traveling speed of the vehicle M.

For example, the control unit 25 of the in-vehicle device 10 may obtain the traveling speed of the automobile M based on the acceleration measured by an acceleration sensor provided in the automobile M and transmit this together with the video during video communication. The control unit 25 may also obtain the traveling speed calculated from the distance and time traveled by the automobile M, or may obtain the traveling speed of the automobile M by receiving a vehicle speed signal (vehicle speed pulse).

In the above embodiment, an example was described in which the in-vehicle device 10 was mounted on an automobile M, but the in-vehicle device 10 may also be mounted on other moving objects such as a bicycle or motorcycle. Furthermore, the in-vehicle device may be held by a person, who may, for example, use video communication to distribute video while walking.

In the above embodiment, an example was described in which the in-vehicle device 10 starts video distribution operations after communication between the in-vehicle device 10 and the external device 70 is established. However, video distribution may be performed in a manner similar to the video distribution of YouTube (registered trademark) and Niconico Live Broadcast (registered trademark) described above. In other words, the video distribution operation may be started even if communication with a viewer's terminal such as the external device 70 is not established. Specifically, the in-vehicle device 10 may start uploading video data to the server 40 even if communication between the in-vehicle device 10 and the viewer's terminal such as the external device 70 is not established.

For example, the video distribution operation by the in-vehicle device 10 may be initiated after communication between the in-vehicle device 10 and the server 40 has been established, without establishing a communication connection with the external device 70. In this case, an unspecified or authorized specific external device 70 can connect to the server 40 to receive the video distributed from the in-vehicle device 10, allowing the user of the external device 70 to watch the video.

In the video distribution system 100 of this embodiment, a voice call is established between the in-vehicle device 10 and the external device 70, and then the video captured in the automobile M is distributed from the in-vehicle device 10 to the external device 70. However, it is also possible to transmit only the video without establishing a voice call.

100 Video distribution system 10 In-vehicle device 23A, 43A Map information database 25, 45, 84 Control unit 30 Encoder unit 31, 87 Data communication unit 32 Video transmission unit 40 Server 70 External device 88 Decoder unit

Claims (14)

A video transmission device that moves with a moving object,
an image acquisition unit that sequentially acquires images of the surroundings of the moving object from an imaging device that captures the images;
a location information acquisition unit that acquires location information of the moving object;
a map information acquisition unit that acquires map information in which a frame rate is associated with each point or section on a road;
a video transmission unit that transmits the video at a frame rate based on the location information and the map information;
Equipped with
A video transmission device characterized in that each of the points is associated with a recommended frame rate for each direction of travel of the moving object . The video transmission device according to claim 1 , wherein the video transmission unit transmits the video at a frame rate equal to or higher than a frame rate associated with the point or section on the road determined based on the location information. The video transmission device according to claim 1 , wherein the video transmission unit transmits the video at a frame rate associated with the point or section on the road determined based on the position information. A video transmission device according to any one of claims 1 to 3, characterized in that the map information associates a frame rate with an area on the map. 5. The video transmission device according to claim 1, wherein the map information is associated with a lower limit value of an allowable frame rate at each point or section on the road. A video transmission device as described in any one of claims 1 to 5, characterized in that in the map information, points or sections where a decrease in frame rate causes a large fluctuation in the perceived speed of a user viewing the video transmitted and displayed by the video transmission unit are associated with a frame rate that is higher than points or sections where the fluctuation in the perceived speed is small. 7. The video transmission device according to claim 1, wherein in the map information, points or sections with good scenery are associated with a frame rate higher than that of other points or sections . A video transmission device that moves with a moving object,
an image acquisition unit that sequentially acquires images of the surroundings of the moving object from an imaging device that captures the images;
a location information transmitting unit that acquires and transmits location information of the mobile object;
a frame rate receiving unit that receives information indicating a frame rate associated with a point or section on a road determined based on the location information;
a video transmitting unit that transmits the video at a frame rate based on the frame rate indicated by the information received by the frame rate receiving unit;
and
A video transmission device characterized in that the location is associated with a frame rate recommended for each direction of travel of a moving object . a location information receiving unit that receives location information of the moving object from a video transmitting device that moves together with the moving object;
a frame rate transmitting unit that identifies a frame rate associated with a point or section on a road determined based on the position information received by the position information receiving unit, based on map information including the point or section on the road associated with a frame rate, and transmits information indicating the identified frame rate to the video transmitting device;
and
The server device is characterized in that , in the map information, each of the points is associated with a recommended frame rate for each direction of travel of the moving object . A video transmission method executed by a video transmission device moving with a moving object, comprising:
an image acquisition step of sequentially acquiring images of the surroundings of the moving object from an imaging device that captures the images;
a position information acquisition step of acquiring position information of the moving body;
a map information acquisition step of acquiring map information in which a frame rate is associated with each point or section on a road;
a video transmission step of transmitting the video at a frame rate based on the location information and the map information;
Including,
A video transmission method characterized in that each of the points is associated with a recommended frame rate for each direction of travel of the moving object . A video transmission program that includes a computer and is executed by a video transmission device that moves together with a moving object,
The computer,
an image acquisition step of sequentially acquiring images of the surroundings of the moving object from an imaging device that captures the images;
a position information acquisition step of acquiring position information of the moving body;
a map information acquisition step of acquiring map information in which a frame rate is associated with each point or section on a road;
a video transmission step of transmitting the video at a frame rate based on the location information and the map information;
Execute
A video transmission program characterized in that each of the points is associated with a recommended frame rate for each direction of travel of the moving object . A video transmission device equipped with a computer and moving with the moving object,
an image acquisition step of sequentially acquiring images of the surroundings of the moving object from an imaging device that captures the images;
a position information acquisition step of acquiring position information of the moving body;
a map information acquisition step of acquiring map information in which a frame rate is associated with each point or section on a road;
a video transmission step of transmitting the video at a frame rate based on the location information and the map information;
Execute
A computer-readable storage medium that stores a video transmission program in which each of the locations is associated with a recommended frame rate for each direction of travel of the moving object . A video transmission device that moves with a moving object,
an image acquisition unit that sequentially acquires images of the surroundings of the moving object from an imaging device that captures the images;
a location information acquisition unit that acquires location information of the moving object;
a map information acquisition unit that acquires map information including points or sections on roads to which frame rates are associated;
a video transmitting unit that transmits the video at a frame rate based on a frame rate associated with the point or section on the road indicated by the location information;
Equipped with
A video transmission device characterized in that, in the map information, points or sections where the perceived speed of a user viewing the video transmitted and displayed by the video transmission unit fluctuates greatly due to a decrease in frame rate are associated with a frame rate that is higher than points or sections where the perceived speed fluctuates little. A video transmission device that moves with a moving object,
an image acquisition unit that sequentially acquires images of the surroundings of the moving object from an imaging device that captures the images;
a location information acquisition unit that acquires location information of the moving object;
a map information acquisition unit that acquires map information including points or sections on roads to which frame rates are associated;
a video transmitting unit that transmits the video at a frame rate based on a frame rate associated with the point or section on the road indicated by the location information;
Equipped with
A video transmission device characterized in that in the map information, points or sections with good scenery are associated with a frame rate that is higher than that of other points or sections.