JP7568022B2 - Image processing device and image processing method - Google Patents

Description

The present invention relates to an image processing device and an image processing method.

Patent Document 1 discloses a vehicle information processing device that performs vehicle-to-vehicle communication with other vehicles. A user in a vehicle specifies the direction of travel and angle width of other vehicles to reduce the amount of communication data. The other vehicles cut out and transmit images within a range corresponding to the specified direction of travel and angle width.

JP 2006-115360 A

The vehicle information processing device in Patent Document 1 had a problem in that it was not possible to select and transmit an appropriate image according to the vehicle's operation, such as whether the vehicle was moving or stopped.

In view of the above problems, the present disclosure aims to provide an image processing device and an image processing method capable of transmitting images according to the operation of a vehicle.

The image processing device according to this embodiment includes an image acquisition unit that acquires an image including an image of the inside of the vehicle including the driver of the vehicle captured by an imaging unit arranged in the vehicle, a vehicle information acquisition unit that acquires vehicle information including information indicating whether the vehicle is stopped, a transmission image generation unit that generates a transmission image by performing at least one of the following when the vehicle is stopped: cutting out a part of the image of the inside of the vehicle in such a way that the driver is visible, selecting an image to be transmitted from the image of the inside of the vehicle in such a way that the driver is visible, and combining multiple images including the image of the inside of the vehicle in such a way that the driver is visible, and a communication unit that transmits the transmission image by wireless communication.

The image processing method according to this embodiment includes an image acquisition step of acquiring an image including an image of the inside of the vehicle including the driver of the vehicle captured by an imaging unit arranged in the vehicle, a vehicle information acquisition step of acquiring vehicle information including information indicating whether the vehicle is stopped, a transmission image generation step of generating a transmission image by performing at least one of the following when the vehicle is stopped: cutting out a part of the image of the inside of the vehicle in such a way that the driver is visible, selecting an image to be transmitted from the image of the inside of the vehicle in such a way that the driver is visible, and combining a plurality of images including the image of the inside of the vehicle in such a way that the driver is visible, and a communication step of transmitting the transmission image by wireless communication.

The present disclosure provides an image processing device and an image processing method that can transmit images according to the operation of a vehicle.

1 is a diagram showing an image processing system according to an embodiment of the present invention; 3A and 3B are diagrams for explaining examples of arrangement of cameras in an imaging unit; FIG. 2 is a diagram showing a schematic view of a captured image of the inside of a vehicle cabin; FIG. 2 is a diagram showing a schematic view of a captured image of the area ahead of the vehicle; FIG. 2 is a control block diagram of the image processing system according to the present embodiment. 11A and 11B are diagrams for explaining a transmission image generated based on posture information. 4 is a flowchart showing an image processing method according to the present embodiment. 3A and 3B are diagrams for explaining examples of arrangement of cameras in an imaging unit; 11A and 11B are diagrams for explaining a transmission image when the image size is changed in accordance with vehicle information. 13 is a diagram for explaining a transmission image when a cut-out position is changed according to vehicle information; FIG. 1A and 1B are top views for explaining examples of camera arrangement and their imaging ranges.

An example of an embodiment of the present invention will be described below with reference to the drawings. The following description shows a preferred embodiment of the present invention, and the technical scope of the present invention is not limited to the following embodiment. In each drawing, the same or corresponding elements are given the same reference numerals, and duplicated explanations will be omitted as necessary for clarity of explanation.

First embodiment
1 is a diagram illustrating an image processing system 1 according to a first embodiment. The image processing system 1 includes an imaging unit 2, a pair of virtual reality (VR) goggles 3, an image processing device 100 provided in a vehicle 4, and a server device 5.

The imaging unit 2 includes one or more cameras, and captures an image of the outside or inside of the vehicle 4. The imaging unit 2 is disposed in the vehicle 4. The imaging unit 2 is, for example, an in-vehicle camera such as a front camera, a rear camera, or a side camera installed on the outside of the vehicle 4, and captures an image of the surroundings of the vehicle 4. Alternatively, the imaging unit 2 may be a camera of a drive recorder installed inside the vehicle 4. Furthermore, the imaging unit 2 may be an electronic device having a camera. For example, the imaging unit 2 may be a camera of a smartphone, a tablet computer, or the like.

The imaging unit 2 may be a wide-angle camera using a fisheye lens or the like. The imaging unit 2 may also be a 360° camera provided on the top of or inside the vehicle 4 in order to capture an image of the entire surroundings of the vehicle 4. The imaging unit 2 may also be a camera that is provided so as to be rotatable 360°. Alternatively, multiple cameras may be arranged facing different directions. By using such a camera as the imaging unit 2, it is possible to capture an image of a wider range outside the vehicle 4.

The vehicle 4 includes an image processing device 100, which will be described later, and processes the captured image captured by the imaging unit 2. Specifically, the image processing device 100 processes the captured image to generate a transmission image. The processing in the image processing device 100 will be described later.

The image processing device 100 transmits the image to the server device 5. The server device 5 is, for example, a distribution server that distributes the transmitted image to the VR goggles 3. The VR goggles 3 is a head-mounted display or the like, and displays the received transmitted image. The VR goggles 3 functions as an image display device that displays the transmitted image. In other words, a user wearing the VR goggles 3
It is possible to view images of the outside or inside of the vehicle 4. In the following description, a person wearing the VR goggles 3 is identified as a user, and a person riding in the vehicle 4 is identified as a driver or passenger.

The VR goggles 3 are equipped with a gyro sensor, a direction sensor, an acceleration sensor, etc., and detect posture. The VR goggles 3 acquires the posture detected by the sensors as posture information. Specifically, the acceleration sensor detects the direction of gravity and detects the up-down direction of the user's head. The direction sensor and gyro sensor detect the direction of the user's face or the left-right direction. Detecting the head direction is used as posture information of the user. Also, the user's posture can be detected not only by the VR goggles 3, but also by electronic devices having similar sensors, such as a smartphone. The VR goggles 3 are connected to the server device 5.
The posture information is transmitted to the vehicle 4 via the VR goggles 3. The display device that receives and displays the transmitted image is not limited to the VR goggles 3, but may be a smartphone, a tablet computer, or the like. It may also be a display device such as a monitor or projector that does not have a VR function. The communication between the vehicle 4 and the VR goggles 3 is carried out via P2P (Peer to P
eer) communication.

Next, the imaging unit 2 will be described. Fig. 2 is a diagram for explaining an example of the arrangement of cameras provided in the imaging unit 2. Fig. 2 is a diagram of a driver's cab inside the vehicle 4 as viewed from the driver's seat toward the front of the vehicle 4. The vehicle 4 has a steering wheel 10, a dashboard 12, a windshield 14, a center console 16, a cluster panel 18 that displays the vehicle's traveling speed, engine RPM, and the like. The center console 16 may also be provided with a center display unit 20 that displays a navigation screen, and the like.

A smartphone 210 is disposed on the dashboard 12.
10 functions as an imaging unit 2. The smartphone 210 includes an in-camera 211, an out-camera 212, and a display screen 213. The in-camera 211 and the display screen 213 are provided on the same surface of the smartphone 210. The out-camera 212 is
The second electrode 11 is provided on the surface opposite to the surface on which the second electrode 11 is provided.

The in-camera 211 faces rearward and captures an image of the inside of the driver's cab. As shown in FIG. 3, the in-camera 211 captures an image of the inside of the driver's cab including the driver sitting in the driver's seat.
The outer camera 212 faces forward and captures an image of the scenery ahead of the vehicle 4.
4 is a schematic diagram illustrating a captured image 50 by the outer camera 212. As illustrated in Fig. 4, the outer camera 212 captures an image of a scene including a road, buildings, vehicles, etc. ahead of the vehicle 4 through the windshield 14. The captured image may be a moving image or a plurality of still images that are continuous in time.

An image processing device 100 is also provided inside the dashboard 12. The image processing device 100 performs predetermined image processing on images captured by the in-camera 211 and the out-camera 212. The installation location of the image processing device 100 is not limited to inside the dashboard. The image processing device 100 is not limited to an in-vehicle device, and may be a portable device. In other words, the image processing device 100 may be a device fixed to the vehicle 4, or may be a device that can be taken out of the vehicle 4.

Furthermore, the image processing device 100 is not limited to being a single physical device, and may be composed of multiple devices. In other words, the processing of the image processing device 100 may be performed by distributed processing devices. For example, an in-vehicle device installed in the vehicle 4 may perform some of the processing, and a portable device may perform the remaining processing. The connections between the multiple devices may be wired connections, or may be connected to a Blu-ray player or other suitable devices.
The connection may be wireless, such as via ETooth (registered trademark) or WiFi (registered trademark).
The smartphone 210 may function as the image processing device 100.
10 may perform some or all of the image processing.

5 is a control block diagram of the image processing system according to the present embodiment. The image processing system 1 includes an imaging unit 2, an image processing device 100, and a VR goggle 3. The image processing device 100 and the VR goggle 3 are connected via a network 6.

First, a description will be given of the image processing device 100. The image processing device 100 includes a captured image acquisition unit 101, a vehicle information acquisition unit 102, a transmission image generation unit 104, and a communication unit 105. An orientation information acquisition unit 103 may also be included.

The captured image acquisition unit 101 acquires captured images from the imaging unit 2. For example, as shown in Fig. 2, when the imaging unit 2 has two cameras, the captured image acquisition unit 101 acquires captured images from each camera. That is, the captured image acquisition unit 101 acquires an image captured by the in-camera 211 and an image captured by the out-camera 212. The image processing device 100 and the imaging unit 2 may be connected by wire or wirelessly.

The vehicle information acquisition unit 102 acquires vehicle information related to the traveling of the vehicle 4. The vehicle information may include, for example, the speed and acceleration of the vehicle 4. The vehicle information may also include operation information of the vehicle 4. The operation information includes information related to the operation of the gear, steering wheel, brake, and accelerator. Here, it is assumed that the operation information includes gear information related to the gear operation of an automatic transmission vehicle. For example, the gear information may include, for example, a forward motion (
D range), reverse operation (R range), or parking (P range).

The vehicle information acquisition unit 102 may acquire vehicle information via a control area network (CAN). That is, the image processing device 100 may be connected to the CAN. Alternatively, the image processing device 100 may acquire vehicle information via Bluetooth from a car navigation system or the like connected to the CAN.
The vehicle information acquisition unit 102 may acquire the vehicle information using, for example, the Vehicle Information Acquisition System (VISA) 2010-1022-1069 (registered trademark).

Furthermore, if the image processing device 100 is equipped with an acceleration sensor, a gyro sensor, or the like,
The vehicle information acquisition unit 102 can acquire vehicle information based on these sensor information. For example, by using an acceleration sensor or the like provided in the smartphone 210, the vehicle information acquisition unit 102 can acquire vehicle information such as speed, acceleration, and traveling direction (forward, backward, left turn, right turn).

The posture information acquisition unit 103 acquires posture information detected by the posture detection unit 32 of the VR goggles 3. The transmission image generation unit 104 processes the captured image to generate a transmission image. Specifically, the transmission image generation unit 104 processes the image by performing at least one of cutting out a part of the captured image, selecting a transmission image from a plurality of captured images, and synthesizing a plurality of captured images based on the vehicle information. The transmission image generation unit 104 may also process the captured image based on the posture information. For example, the transmission image generation unit 104 generates a transmission image by performing at least one of cutting out a part of the captured image, selecting a transmission image from a plurality of captured images, and synthesizing a plurality of captured images based on at least one of the vehicle information and the posture information.

The communication unit 105 has a long-distance wireless communication function. Specifically, the communication unit 105 is a modulation circuit that modulates and outputs the transmission image generated by the transmission image generation unit 104 to an antenna for outputting a wireless signal. The communication unit 105 may be configured to include an antenna. Also, the communication unit 105 may be configured to include a demodulation circuit for a signal received by the antenna. The antenna receives a reception signal and radiates a transmission signal into space. The demodulation circuit demodulates the reception signal received by the antenna. The modulation circuit modulates a transmission signal and outputs it to the antenna. For example, the communication unit 105 performs communication processing according to a mobile phone communication standard such as 3G, LTE (registered trademark), 4G, or 5G. The communication standard of the communication unit 105 is not particularly limited.

Wireless communication by the communication unit 105 enables data communication with a network 6 outside the vehicle. The network 6 is, for example, an external network such as the Internet.
The communication unit 105 transmits a transmission image to the VR goggles 3 via the network 6. Furthermore, the communication unit 105 receives posture information transmitted from the VR goggles 3 via the network 6. For example, the network 6 includes the server device 5 shown in Fig. 1. The communication unit 105 may perform wireless communication for voice calls.

Next, a description will be given of the configuration of the VR goggles 3. The VR goggles 3 include a display unit 31, a posture detection unit 32, a control unit 33, and a communication unit .

The display unit 31 displays an image to a user wearing the VR goggles 3.
The display unit 31 includes a flat panel display such as a liquid crystal monitor or an organic EL (Electro-Luminescence) monitor. The display unit 31 includes an optical system for guiding an image displayed on the display to the user's eye. The display unit 31 may include an optical system for the left eye and an optical system for the right eye.

The posture detection unit 32 detects the posture of the user wearing the VR goggles 3. Specifically,
The posture detection unit 32 has a head tracking function. The posture detection unit 32 has a sensor that detects the posture state, such as a three-axis sensor or a six-axis sensor. Furthermore, the posture detection unit 32 has a circuit that estimates the amount of movement from the sensor output. The posture detection unit 32 generates a movement amount signal according to the detection result of each axis. By detecting the position and direction of the user's head using the posture detection unit 32, the VR goggles 3 can change the displayed image.

The communication unit 35 has a long-distance wireless communication function similar to the communication unit 105.
The communication unit 105 performs communication processing according to a mobile phone communication standard such as 3G, LTE (registered trademark), 4G, or 5G. The communication standard of the communication unit 105 is not particularly limited.
5 may be a short-range wireless communication function such as Bluetooth communication or WiFi communication.
The communication unit 35 may also be a wired connection interface such as HDMI (registered trademark). In such a case, the communication unit 35 is connected to a Wi-Fi router or the like that is capable of communicating with the network 6.

Communication by the communication unit 35 enables the VR goggles 3 to communicate data with the network 6. The communication unit 35 transmits posture information to the image processing device 100 via the network 6. Furthermore, the communication unit 35 receives a transmission image transmitted from the image processing device 100 via the network 6. Furthermore, the communication unit 35 may perform wireless communication for voice calls.

The control unit 33 performs overall control of the VR goggles 3. For example, a transmission image received by the communication unit 35 is input to the control unit 33. The control unit 33 generates a video signal and a control signal based on the transmission image. The display of the display unit 31 generates a display image based on the video signal and the control signal. For example, when the image size of the transmission image differs from the display size of the display, the control unit 33 converts the transmission image to the display size of the display.
1 can display a display image based on the transmitted image. Note that the transmitted image and the display image are not limited to being the same image. For example, the control unit 33 may add appropriate information to the transmitted image to generate the display image.

It should be noted that a part or all of the processing in the image processing device 100 can be realized by a computer program. Similarly, a part or all of the processing in the VR goggles 3 can be realized by a computer program. The image processing device 100 and the VR goggles 3 each include a memory and a processor. The memory stores a processing program, various parameters, data, etc. The processor executes the processing program stored in the memory.
The processor executes a processing program to execute each process. The processor may be, for example, a CPU (Central Processing Unit), an FPGA (Field-Programmable Gate Array
), DSP (Digital Signal Processor), ASIC (Application Specific Integrate)
d Circuit), or GPU (Graphics Processing Unit), etc.

The process in the transmission image generating unit 104 will be described in detail.
An example will be described in which the transmission image generating unit 104 generates a transmission image based on vehicle information. Specifically, the transmission image generating unit 104 selects a transmission image based on vehicle information on whether the vehicle 4 is stopped or not, for example, gear information of the vehicle 4. The captured image acquiring unit 101 acquires, for example, an image captured by the in-camera 211 and an image captured by the out-camera 212, as shown in Fig. 2. The captured image by the in-camera 211 is an image of the inside of the driver's cab as shown in Fig. 3, and the captured image by the out-camera 212 is a view in front of the vehicle 4 as shown in Fig. 4.
4 may determine a cut-out range of the transmission image based on the vehicle information.

When the gear is in the D range, that is, when the vehicle is moving forward, the transmission image generating unit 104
The transmission image generating unit 104 selects an image captured by the outer camera 212 capturing an image of the view ahead. When the gear is in the P range, that is, when the vehicle is parked, the transmission image generating unit 104 selects an image captured by the inner camera 211 capturing an image of the interior of the driver's cab. In this way, the transmission image generating unit 104 selects one image to be transmitted from among the multiple captured images based on the vehicle information. The communication unit 105 then
The selected transmission image is transmitted to the VR goggles 3. This enables the VR goggles 3 to display a display image based on the selected image.

The user can enjoy the scenery while driving because the user can view the display image displayed on the VR goggles 3. In other words, the user can view the same scenery ahead as the driver sees. In addition, while parking, the user can view the driver's face in the same way as with a video phone call. This allows the user to have a simulated experience of traveling with the driver.

The vehicle information is not limited to gear information, and may be information indicating whether the vehicle 4 is stopped or not. For example, the transmission image generating unit 104 or the vehicle information acquiring unit 102 may determine whether the vehicle is stopped or not depending on the acceleration or speed detected by a sensor. For example, the transmission image generating unit 104 or the vehicle information acquiring unit 102 may determine whether the vehicle is stopped depending on whether the engine is stopped or the position information has not changed. If the vehicle information indicates that the vehicle is stopped, the transmission image generating unit 104 may select an image captured by the in-camera 211. If the vehicle information indicates that the vehicle is traveling forward, the transmission image generating unit 104 may select the out-camera 212.

In the above description, the image captured by the outer camera 212 is the image to be transmitted while the vehicle is moving, and the image captured by the inner camera 211 is the image to be transmitted while the vehicle is stopped. However, the selection of the image to be transmitted is not limited to this example. When the imaging unit 2 has three or more cameras, the transmission image generating unit 104 can appropriately select the image to be transmitted according to the vehicle information.
In the case of a 0° camera, the transmission image generation unit 104 may appropriately cut out a part of the captured image in accordance with the vehicle information. Also, the transmission image generation unit 104 selects one or more images from the three or more captured images to be transmitted.

For example, when the gear is in the R range, that is, when the vehicle is moving backward, an image captured by a rear camera capturing an image of the rear of the vehicle 4 may be used as the transmitted image. The user can visually confirm the rear of the vehicle 4. This allows the user to check safety during the backward movement from a remote location. For example, when there is an obstacle behind the vehicle 4, the user can notify the driver of the presence of the obstacle by voice communication. This can improve safety.

In this way, the transmission image generating unit 104 generates a transmission image by processing the captured image based on the vehicle information. For example, the transmission image generating unit 104 generates a transmission image by selecting a transmission image from a plurality of captured images.
The VR goggles 3 can transmit images according to the operation of the vehicle 4. In addition, the amount of data of the transmitted images can be reduced. This can suppress communication delays, allowing the VR goggles 3 to perform highly responsive display.

Next, an example will be described in which the transmission image generation unit 104 generates a transmission image based on the posture information. Here, a transmission image will be described in which the user changes the direction of the head while the vehicle is moving forward. As shown in FIG. 4, the transmission image generation unit 104 acquires an image captured by the outer camera 212. The posture detection unit 32 detects the direction or position of the user's head. Then, the VR goggles 3 transmit the direction or position of the head to the image processing device 100 as posture information.

When the image processing device 100 receives the posture information transmitted by the VR goggles 3, the transmission image generation unit 104 generates a transmission image in accordance with the posture information. Here, the transmission image generation unit 104 cuts out a part of the captured image in accordance with the posture information.

For example, as shown in Fig. 6, the captured image 50 includes the view ahead of the vehicle 4, similar to Fig. 4. The transmission image generation unit 104 determines the cut-out position of the transmission image based on the posture information. For example, when the user faces forward, the transmission image generation unit 104 cuts out a part of the center of the captured image 50 on each of the top, bottom, left and right sides to generate a transmission image 60C. When the user faces left, the transmission image generation unit 104 cuts out a part of the left side of the captured image 50,
When the user faces upward, the transmission image generating unit 104 cuts out a part of the upper side of the captured image 50 and sets it as a transmission image 60U.

In this way, the transmission image generating unit 104 processes the captured image based on the posture information to generate a transmission image. The transmission image generating unit 104 cuts out a part of the captured image to generate a transmission image. This makes it possible to reduce the image size of the transmission image smaller than the image size of the captured image, thereby reducing the amount of transmission data. This makes it possible to suppress communication delays, allowing the VR goggles 3 to perform highly responsive display.

In particular, when the imaging unit 2 can capture an image of 360° around the vehicle 4, the image size can be reduced by cutting out a part of the captured image as the transmission image. Therefore, the transmission image can be transmitted without reducing the resolution. In addition, the processing time for expanding the compressed image on the VR goggles 3 side can be shortened. This enables the delivery of high image quality with low latency, and improves real-time performance.

Furthermore, the transmission image generating unit 104 determines the cut-out position based on the posture information.
The transmission image generating unit 104 calculates a pixel address from which to cut out the transmission image in the captured image based on the posture information. Specifically, when the user's head direction detected by the VR goggles 3 is facing forward, the transmission image generating unit 104 sets the center of the captured image by the out-camera 212 as the cut-out position. When the user's head direction detected by the VR goggles 3 is facing backward, that is, when the user faces backward, the transmission image generating unit 104 sets the center of the captured image by the in-camera 211 as the cut-out position.
The center of the captured image by the VR goggles 3 is set as the cutout position. When cutting out an image from a 360° camera, the center of the cutout position can be freely changed up, down, left, and right depending on the direction of the user's head. Therefore, the transmission image generation unit 104 can generate a view in the direction that the user wearing the VR goggles 3 wants to see as a transmission image. Therefore, the user can experience traveling with the driver while sitting in the passenger seat. Furthermore, the user of the VR goggles 3 may have a conversation with the driver or passenger of the vehicle 4 by voice call. This allows the driver or passenger to receive advice such as route guidance and tourist information from the user. Furthermore, the user can point out driving hazards that the driver may not notice easily.

Furthermore, when the user wearing the VR goggles 3 faces backward, the transmission image generation unit 104 may switch to an image captured by the in-camera 211. In other words, when the user faces backward, the transmission image generation unit 104 may set an image captured inside the cab as a transmission image.

In the above description, examples of generating a transmission image based on posture information or vehicle information have been described separately, but the transmission image generating unit 104 may generate a transmission image based on both posture information and vehicle information. In this case, for example, the transmission image generating unit 104 may generate a transmission image based on posture information while the vehicle is moving forward, and may generate a transmission image based on vehicle information while the vehicle is stopped. This can be achieved by adjusting the cut-out position of the transmission image as shown in FIG. 6.

The image processing method according to the first embodiment will be described with reference to Fig. 7. Fig. 7 is a flowchart of the image processing method. First, the captured image acquisition unit 101 acquires an image captured by the imaging unit 2 (S11). The vehicle information acquisition unit 102 acquires vehicle information (S12). The posture information acquisition unit 103 acquires posture information of the user of the VR goggles 3 (S13).

The transmission image generating unit 104 generates a transmission image from the captured image (S14). That is, the transmission image generating unit 104 processes the captured image based on the vehicle information or the attitude information. For example, the transmission image generating unit 104 selects one transmission image from a plurality of captured images. Alternatively, the transmission image generating unit 104 cuts out a part of the captured image to generate the transmission image. The specific processing process is the same as above, and therefore will not be described. As a result, the transmission image is generated. The communication unit 105 transmits the transmission image (S15).

The image processing device 100 repeats the above process, so that the VR goggles 3 can display moving images. Thus, the user can have a simulated experience of riding in the vehicle 4. The image processing device 100 can transmit high-quality transmission images with low latency, and can achieve high real-time performance. In addition, when the frame interval of the captured images does not match the update interval of the vehicle information and the attitude information, the image processing device 100 may generate a transmission image using the latest vehicle information and attitude information. In addition, the transmission image generation unit 104 may generate a transmission image based on only one of the attitude information and the vehicle information.

Embodiment 2.
In the second embodiment, the vehicle information acquisition unit 102 acquires speed information indicating the speed of the vehicle as the vehicle information. For example, the vehicle information acquisition unit 102 can acquire the speed information by using a vehicle speed pulse output from the CAN. Alternatively, the vehicle information acquisition unit 102 may acquire the speed information by communicating with an acceleration sensor provided in the vehicle 4 or the smartphone 210.

Fig. 8 is a diagram showing an example of the arrangement of the imaging unit 2 according to the present embodiment. In this embodiment, a drive recorder 220 is provided instead of the smartphone 210. The drive recorder 220 is installed on the windshield 14, which is the front glass. The imaging unit 2 includes a camera 221 of the drive recorder 220. The camera 221 captures the scenery ahead in the same manner as the outer camera 212 in the first embodiment. Fig. 9 is a diagram showing an image captured by the camera 221.

The transmission image generating unit 104 changes the range of the image to be cut out, for example, the size of the image to be cut out (also called the cut-out size), according to the speed information. Fig. 9 shows a transmission image 61 in the case of a speed V1 and a transmission image 62 in the case of a speed V2. The speed V1 is faster than the speed V2, and the transmission image 61 has a smaller image size than the transmission image 62.
In the case of the image size extractor 04, the faster the vehicle speed indicated by the speed information is, the smaller the size of the image to be extracted becomes. In this way, a highly realistic display image can be obtained. Note that the aspect ratio of the transmitted image 61 and the transmitted image 62 is the same, but the number of pixels in the vertical and horizontal directions is different. In this case, it is preferable that the center points of the extracted transmitted images 61 and 62 are the same.
It is more preferable that the central point is the point at which the user is gazing or the central portion of the area at which the user is gazing, according to the posture information.

In addition, the faster the vehicle travels, the more unstable the communication becomes, and the lower the communication speed becomes. In addition, the faster the vehicle travels, the more likely it is that the transmitted image will be in MPEG (Moving Picture Experts Group) format.
When the captured image is compressed by a Real-Time Video (RTS) encoder or the like and transmitted, the change in the captured image is large, and the data size of the compressed image data may become large. Therefore, the faster the vehicle speed, the smaller the cutout size is set, so that communication delays can be suppressed. The VR goggles 3 can display images with high real-time quality.

Furthermore, the transmission image generating unit 104 may change the range of the image to be cut out, for example, the cut-out position, in accordance with the speed information. For example, the transmission image generating unit 104 can change the cut-out position of the transmission image in the captured image in the vertical direction in accordance with the speed information.
A transmitted image 61 in the case of a speed V1 and a transmitted image 62 in the case of a speed V2 are shown. The speed V1 is
This is a speed faster than speed V2, and transmission image 61 is located above transmission image 62. The faster the vehicle speed indicated by the speed information, the higher the cut-out position is set by transmission image generation unit 104. In this way, the faster the vehicle speed is, the more a transmission image can be generated that captures a position farther away from vehicle 4. This makes it possible to obtain a display image with a more realistic feel.

The transmission image generating unit 104 shifts the cutout position upward as the vehicle speed increases.
In this way, communication delays can be suppressed. For example, as the vehicle speed increases, roads are no longer included in the transmitted image. Assume that the transmitted image is compressed by an MPEG encoder or the like and transmitted. Since the transmitted image not including roads has a small amount of image change, the compression rate is higher than that of a transmitted image including roads, which has a large amount of image change. Therefore, by the image processing device 100 compressing the transmitted image not including roads, the data volume of the transmitted image can be further reduced. In addition, the faster the vehicle travels, the higher the possibility of the user experiencing problems such as dizziness due to gazing at roads, which have a large amount of image change.
It is preferable to use the cut-out position as information. The transmitted image may be an image that does not include the road at all, or an image that includes a part of the road. By reducing the area of the road in the transmitted image, the transmitted image can be generated with a higher compression rate. This makes it possible to suppress communication delays, and the VR goggles 3 can display images with high real-time quality.

In the above description, examples in which the transmission image generating unit 104 changes the image size and the cut-out position according to the speed information have been described separately, but the transmission image generating unit 104 may change both the image size and the cut-out position according to the speed information. For example, the image size may be made smaller and the cut-out position may be moved upward as the vehicle speed indicated by the speed information becomes faster.

The second embodiment may be combined with the first embodiment. For example, the transmission image generating unit 104
can determine the extraction position based on the attitude information and the velocity information.

Embodiment 3.
In the third embodiment, the vehicle information acquisition unit 102 acquires acceleration information indicating the acceleration of the vehicle as the vehicle information. For example, the vehicle information acquisition unit 102 can acquire the acceleration information by using a vehicle speed pulse output from the CAN. Alternatively, the vehicle 4 or the smartphone 2
The vehicle information acquisition unit 102 may acquire acceleration information using an acceleration sensor provided in the vehicle information acquisition unit 10 .

The transmission image generating unit 104 changes the size of the image to be cut out according to the acceleration information. The transmission image generating unit 104 reduces the image size of the transmission image as the acceleration increases. In Fig. 9, a transmission image 61 is a transmission image in the case of acceleration A1, and a transmission image 62 is a transmission image in the case of acceleration A2. Note that acceleration A1 is greater than acceleration A2.

In this way, by changing the image size (cutout size) according to the acceleration information,
The R goggles 3 can display a display image with a higher sense of realism. For example, when the vehicle 4 starts moving, the acceleration increases, resulting in a smaller image size. A user wearing the VR goggles 3 can view the display image with a higher sense of realism.

Alternatively, the transmission image generating unit 104 may change the cut-out position according to the acceleration information. The higher the acceleration, the higher the cut-out position of the transmission image generated by the transmission image generating unit 104 is. In Fig. 10, a transmission image 61 is a transmission image in the case of acceleration A1, and a transmission image 62 is a transmission image in the case of acceleration A2. Note that acceleration A1 is greater than acceleration A2.

In this way, by changing the cut-out position according to the acceleration information, the VR goggles 3:
For example, when the vehicle 4 starts moving, the acceleration increases, and the cut-out position becomes higher. A user wearing the VR goggles 3 can view the display image with a high sense of realism.

In the above description, the examples in which the transmission image generation unit 104 changes the image size and the cropping position in accordance with the speed information and the acceleration information have been described separately.
Both the image size and the cut-out position may be changed according to the acceleration information or the speed information.
For example, the image size may be made smaller and the cut-out position may be shifted upward as the vehicle speed indicated by the speed information increases or the acceleration indicated by the acceleration information increases. Furthermore, the transmission image generating unit 104 may process the captured image in accordance with both the acceleration information and the speed information.

The third embodiment may be combined with the first embodiment. For example, the transmission image generating unit 10
4 can determine the cut-out position based on the posture information and the acceleration information.

Embodiment 4.
In the image processing system according to the fourth embodiment, the imaging unit 2 has a plurality of cameras.
An example of the arrangement of the cameras in the imaging unit 2 will be described with reference to FIG. 11. FIG. 11 is a schematic diagram of the vehicle 4 as viewed from above. In order to cover the entire 360° surroundings of the vehicle 4, the imaging unit 2 includes a front camera 231, a rear camera 232, a left side camera 233, and a right side camera 234.
11, the imaging ranges of the front camera 231, the rear camera 232, the left side camera 233, and the right side camera 234 are indicated by dashed lines.

The front camera 231, the rear camera 232, the left side camera 233, and the right side camera 234 are wide-angle cameras in order to cover a wide imaging range.

The front camera 231 captures an image in front of the vehicle 4, and the rear camera 232 captures an image in the rear of the vehicle 4. The left side camera 233 captures an image on the left side of the vehicle 4, and the right side camera 234 captures an image on the right side of the vehicle 4.
The front camera 231 captures an image of the right side of the vehicle 4. A part of the imaging range of the front camera 231 is captured by the left side camera 2
The imaging range of the front camera 231 partially overlaps with the imaging range of the right side camera 234. Similarly, the imaging range of the rear camera 232 partially overlaps with the imaging range of the left side camera 233 and the imaging range of the right side camera 234.

The posture information acquisition unit 103 acquires posture information indicating the posture of the user. In this example, it is assumed that the user is facing diagonally forward and to the left. In this case, the range to be displayed in the transmitted image is set as a range 65 as shown in FIG. 11. The left and right ends of the range 65 are set as a left end 66L and a right end 66R.
includes a part of the imaging range captured by the front camera 231 and a part of the imaging range captured by the left side camera 233. A left end 66L of the range 65 is included only in the imaging range of the left side camera 233, and a right end 66R is included only in the imaging range of the front camera 231.

The transmission image generating unit 104 cuts out a part of the image captured by the front camera 231. Specifically, the transmission image generating unit 104 cuts out the captured image in a cut-out range 67 on the left side of the imaging range of the front camera 231. The transmission image generating unit 104 also cuts out a part of the image captured by the left side camera 233. Specifically, the transmission image generating unit 104 cuts out the captured image in a cut-out range 68 on the front side of the image captured by the left side camera 233. The cut-out range 67 and the cut-out range 68 partially overlap. The transmission image generating unit 104 synthesizes the two cut-out images to generate a transmission image.

The transmission image generating unit 104 synthesizes two captured images. That is, the transmission image generating unit 104 synthesizes the captured images taken by the two cameras to generate a transmission image. The transmission image generating unit 104 may also cut out a part of the captured image from one camera and synthesize it with the entire captured image from the other camera. Of course, the two captured images to be synthesized are synchronized. That is, the transmission image generating unit 104 synthesizes two captured images taken by different cameras at the same timing.
Furthermore, the transmission image generating unit 104 may generate a transmission image by combining images captured by three or more cameras.

The above-mentioned first to fourth embodiments can be appropriately combined.
can generate a transmission image by performing at least one of cutting out a part of a captured image, selecting an image to be transmitted from a plurality of captured images, and synthesizing a plurality of captured images based on the vehicle information. In this manner, the image processing device 100 can transmit an image according to the operation of the vehicle 4. In addition, since the data amount of the transmission image can be reduced, it becomes possible to display the image in high quality and in real time on the VR goggles 3.

In the above embodiment, the transmission image generating unit 104 determines the cut-out position in the captured image based on the posture information, but the cut-out position may be determined based on information other than the posture information. For example, the user may change the direction in which the user wants to view by voice input through a microphone or operation input through a button or the like. In other words, when the user specifies the direction in which the user wants to view by voice input or operation input, the transmission image generating unit 104 changes the cut-out position so that the direction is included in the transmission image. Furthermore, the camera may be switched or the image size may be changed by the user's voice input or operation input.

The image processing device 100 and the VR goggles 3 are configured by a processor such as a CPU (Central Processing Unit), a storage device such as a memory, a user interface, and various peripheral circuits. In other words, the image processing device 100 and the VR goggles 3 have functions as a computer. In addition, the image processing device 100 may realize components such as a captured image acquisition unit 101, a vehicle information acquisition unit 102, a posture information acquisition unit 103, a transmission image generation unit 104, and a communication unit 105 by the processor executing a program stored in the storage device. The VR goggles 3 may realize components such as a display unit 31, a posture detection unit 32, a control unit 33, and a communication unit 35 by the processor executing a program stored in the storage device.

In addition, each component of the image processing device 100 and the VR goggles 3 is not limited to being realized by software programs, but may be realized by any combination of hardware, firmware, and software.
Each component of the present invention may be realized by using an integrated circuit programmable by a user, such as a field-programmable gate array (FPGA) or a microcomputer. In this case, the integrated circuit may be used to realize a program composed of each of the above-mentioned components. This also applies to other embodiments described later.

Furthermore, the program for executing the above-described display control method can be stored in various types of non-transitory computer readable media and supplied to a computer. The non-transitory computer readable media includes various types of tangible storage media. Examples of the non-transitory computer readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory Cards), and the like.
y Memory), CD-R, CD-R/W, semiconductor memory (e.g., mask ROM, PROM
(Programmable ROM), EPROM (Erasable PROM), Flash ROM, RAM (random access memory)
The program may be provided to the computer by various types of transitory computer readable media. Examples of the transitory computer readable media include an electric signal, an optical signal, and an electromagnetic wave. The transitory computer readable media may provide the program to the computer via a wired communication path such as an electric wire and an optical fiber, or via a wireless communication path.

REFERENCE SIGNS LIST 1 System 2 Imaging unit 3 VR goggles 4 Vehicle 5 Server device 6 Network 31 Display unit 32 Attitude detection unit 33 Control unit 35 Communication unit 50 Captured image 61 Transmitted image 62 Transmitted image 100 Image processing device 101 Captured image acquisition unit 102 Vehicle information acquisition unit 103 Attitude information acquisition unit 104 Transmitted image generation unit 105 Communication unit 210 Smartphone 211 In-camera 212 Out-camera 220 Drive recorder 221 Camera 231 Front camera 232 Rear camera 233 Left side camera 234 Right side camera

Claims (2)

an image acquisition unit that acquires an image including an image of the inside of the vehicle including a driver of the vehicle, the image being captured by an imaging unit disposed in the vehicle;
a vehicle information acquisition unit that acquires vehicle information including information indicating whether the vehicle is stopped;
a transmission image generating unit that generates the transmission image by performing at least one of the following when the vehicle is stopped: cutting out a part of a captured image inside the vehicle in such a way that the driver can be visibly recognized; selecting a transmission image from the captured image inside the vehicle in such a way that the driver can be visibly recognized; and combining a plurality of the captured images including the captured image inside the vehicle in such a way that the driver can be visibly recognized;
a communication unit that transmits the transmission image by wireless communication;
An image processing device comprising: acquiring an image including an image of an interior of the vehicle including a driver of the vehicle, the image being captured by an imaging unit disposed in the vehicle;
acquiring vehicle information including information indicating whether the vehicle is stopped;
generating the transmission image by performing at least one of the following when the vehicle is stopped: cutting out a part of a captured image inside the vehicle in such a manner that the driver can be visibly recognized; selecting a transmission image from the captured images inside the vehicle in such a manner that the driver can be visibly recognized; and combining a plurality of the captured images including the captured image inside the vehicle in such a manner that the driver can be visibly recognized;
transmitting the transmission image by wireless communication;
An image processing method comprising:

Images