JP7740329B2 - Image processing device and image processing method - Google Patents

Description

This technology relates to an image processing device and an image processing method, and more specifically to an image processing device and an image processing method that generate a 360-degree image and obtain an image for display in a specified direction based on that image.

For example, Patent Document 1 discloses a technology for synthesizing and displaying images captured by cameras on the front, rear, left, and right sides of a vehicle. The display image (synthetic image) in this case is intended to be displayed on an in-vehicle display, and high resolution is not required. Synthesizing captured images at high resolution results in a high processing load.

Japanese Patent Application Laid-Open No. 2020-127171

The purpose of this technology is to enable the generation of display images that appear less unnatural without increasing the amount of processing.

The concept of this technology is:
a 360-degree image generating unit that generates a rectangular parallelepiped 360-degree image based on a plurality of captured images;
The image processing device includes a display image generation unit that generates a display image in a predetermined direction based on the 360-degree image.

In this technology, a 360-degree image generating unit generates a rectangular parallelepiped 360-degree image based on multiple captured images. For example, the multiple captured images may be four captured images obtained by four imaging units attached to the exterior of the vehicle, each capturing an image of the front, right, left, and rear. In this case, it is possible to generate a rectangular parallelepiped 360-degree image with the vehicle at its center.

In this case, for example, the 360-degree image generation unit may be configured to attach images obtained by the imaging unit capturing images of the front side to the front region of the rectangular parallelepiped, images obtained by the imaging unit capturing images of the rear side to the rear region of the rectangular parallelepiped, images obtained by the imaging unit capturing images of the right side to the right region of the rectangular parallelepiped and the right half regions of the upper and lower surfaces adjacent to the right region, and images obtained by the imaging unit capturing images of the left side to the left region of the rectangular parallelepiped and the left half regions of the upper and lower surfaces adjacent to the left region, thereby generating a rectangular parallelepiped-shaped 360-degree image. This makes it easy to create a 360-degree image centered on the vehicle based on four captured images from the front, rear, left, and right.

In this case, for example, the imaging units capturing images of the front and rear may be wide-angle imaging units, and the imaging units capturing images of the right and left may be fisheye imaging units. This allows the images captured by the four imaging units (front, rear, left, and right) to capture a wider range, making it possible to include a wider range of images as a 360-degree image centered on the vehicle. Here, the 360-degree image generation unit may be configured to crop and use the center portion of each of the four captured images. This makes it possible to generate a 360-degree image with less distortion.

In this case, for example, a receiving unit may be further provided that receives four captured images from the vehicle. This makes it possible to receive and process the four captured images from the vehicle at a location away from the vehicle.

The display image generation unit generates a display image in a specified direction based on the 360-degree image. For example, the display image generation unit may generate a display image in a specified direction by capturing a rectangular 360-degree image with a virtual camera with a specified angle of view, installed at the center of the rectangular prism, and set to capture the image in a specified direction. This makes it easy to generate a display image in a specified direction based on the 360-degree image.

Furthermore, for example, a selection unit may be provided that selects a predetermined direction based on a user operation. In this case, since the predetermined direction can be selected based on a user operation, it is possible to obtain a display image in any direction.

Also, for example, the device may further include a distribution unit that distributes display images. This makes it possible to distribute display images. Also, the device may further include a display unit that displays display images. This makes it possible to display and check the display images.

In this way, this technology generates a rectangular 360-degree image based on multiple captured images, and then generates a display image in a specified direction based on this 360-degree image. Compared to generating a display image in a specified direction based on a spherical or cylindrical 360-degree image obtained by performing distortion correction, stitching, etc., this technology makes it possible to obtain a display image that looks less strange without increasing the amount of processing. Furthermore, because the processing load is low, processing of each frame of video can easily be performed in real time.

In this technology, for example, the 360-degree image generation unit may provide a boundary at the seams between the faces of a rectangular 360-degree image. This makes it possible to reduce the sense of incongruity at the seams between captured images, even when a rectangular 360-degree image is generated by simply pasting captured images from different image capture units onto each face. In this case, for example, the boundary may be generated by blurring the images of the two faces corresponding to the seams. This makes it possible to more naturally reduce the sense of incongruity at the seams between captured images. Also, in this technology, for example, the multiple captured images may be captured after specific object regions have been subjected to identification processing to prevent the identification of the object. This makes it possible to prevent the identification of specific objects, such as people, and thereby protect privacy.

Another concept of the present technology is
A procedure for generating a rectangular parallelepiped 360-degree image based on a plurality of captured images;
The image processing method includes a procedure for generating an image for display in a predetermined direction based on the 360-degree image.

1 is a diagram showing an example of the appearance of a vehicle according to the present technology; FIG. 10 is a diagram showing how images captured by four cameras on the front, rear, left and right sides are transmitted from a transmitter on a vehicle to a remote monitoring device. FIG. 2 is a block diagram showing an example of the configuration of a remote monitoring device. 1A and 1B are diagrams showing an example of an image captured by four cameras on the front, rear, left and right sides, and an example of each cutout range. FIG. 10 is a diagram for explaining generation of a 360-degree image by pasting images. FIG. 10 is a diagram illustrating an example of a boundary portion. 10 is a diagram schematically illustrating an example of a display image (two-dimensional image) when the predetermined direction is selected as the front upper direction. FIG. FIG. 10 is a diagram showing an example of a display image of the upper front part of a 360-degree image. FIG. 10 is a diagram showing an example of a display image for the lower rear of a 360-degree image. 10 is a flowchart illustrating an example of a processing procedure for generating a display image in a remote monitoring device and displaying and distributing the image. 10A and 10B are diagrams illustrating an example of performing identification processing on a region of a specific object so that the specific object is not identified when the specific object is included in a captured image after clipping. 10 is a flowchart illustrating another example of a processing procedure for generating a display image in a remote monitoring device and displaying and distributing the image. FIG. 2 is a block diagram illustrating an example of the hardware configuration of a computer.

The following describes modes for carrying out the invention (hereinafter referred to as "embodiments") in the following order:
1. Embodiment 2. Modification

1. Embodiment
1 shows an example of the appearance of a vehicle 10 according to the present technology. The vehicle 10 is an autonomous vehicle or a remotely controlled vehicle and has no windows. A display 11 is provided on the exterior of the vehicle 10. This display 11 is provided to display advertisements and the like.

In addition, cameras 12a, 12b, 12c, and 12d are attached to the front, right side, left side, and rear of the exterior of vehicle 10 as imaging units for capturing images of the front, right, left, and rear, respectively.

In this embodiment, camera 12a, which captures images of the front, and camera 12d, which captures images of the rear, are each wide-angle image capture units equipped with wide-angle lenses, while camera 12b, which captures images of the right, and camera 12c, which captures images of the left, are each fisheye image capture units equipped with fisheye lenses. As a result, the images captured by the four front, rear, left, and right cameras 12a, 12b, 12c, and 12d capture a wider range, making it possible to include a wider range of images as a 360-degree image centered on vehicle 10, as described below.

In addition, a transmitting device (not shown) is located inside the vehicle 10 for transmitting the four images captured by cameras 12a, 12b, 12c, and 12d to a remote monitoring device that constitutes the image processing device of this technology.

Figure 2 shows that four captured images obtained by cameras 12a, 12b, 12c, and 12d are transmitted from the transmitter 13 of the vehicle 10 to the remote monitoring device 100. The transmitter 13 transmits, for example, the four captured images input from each of the cameras 12a, 12b, 12c, and 12d, i.e., a forward image Ia, a right image Ib, a left image Ic, and a rear image Id, to the remote monitoring device 100, for example, via a telephone line. In this case, the resolution of the four captured images is limited depending on the communication capacity of the line.

Figure 3 shows an example configuration of the remote monitoring device 100. This remote monitoring device 100 has a control unit 101, a user operation unit 102, a receiving unit 103, an image cropping unit 104, an image pasting unit 105, a boundary processing unit 106, a display image generation unit 107, a display unit 108, and a distribution unit 109.

The control unit 101 controls the operation of each unit of the remote monitoring device 100. The user operation unit 102 is the part where the user issues various operation instructions. This operation includes the selection of a specific direction required when generating a display image from a 360-degree image.

The receiving unit 103 receives captured images Ia, Ib, Ic, and Id captured by the cameras 12a, 12b, 12c, and 12d, which are transmitted from the transmitting device 13 (see FIG. 2 ) of the vehicle 10. The image cropping unit 104 crops out the central portion, i.e., the portion with least distortion, from each of the captured images Ia, Ib, Ic, and Id received by the receiving unit 103, to obtain cropped captured images Ia ' , Ib ' , Ic ' , and Id ' . By cropping in this manner, it is possible to generate a 360-degree image with little distortion centered on the vehicle 10, which will be described later.

Figure 4 shows examples of captured images Ia, Ib, Ic, and Id captured by cameras 12a, 12b, 12c, and 12d, and examples of their respective cropping ranges. In the illustrated example, the cropping ranges are shown enclosed in rectangular frames. Note that the cropping ranges are fixed, for example, but may be changeable in size and position in response to user operation.

Returning to Fig. 3, the image pasting unit 105 pastes the captured images Ia ' , Ib ' , Ic ' , and Id ' obtained by the image cropping unit 104 onto each face of a rectangular parallelepiped to generate a rectangular parallelepiped-shaped 360-degree image. Fig. 5(a) shows the captured images Ia ' , Ib ' , Ic ' , and Id ' after cropping for the front, right, left, and rear. Fig. 5(b) is a diagram of the rectangular parallelepiped unfolded, showing the areas where each of the captured images Ia ' , Ib ' , Ic ' , and Id ' is pasted.

In this case, the front captured image Ia ' is pasted on the front surface of the rectangular parallelepiped. The rear captured image Id ' is pasted on the rear surface of the rectangular parallelepiped. The right captured image Ib' is pasted on the right surface of the rectangular parallelepiped and on the right half of each of the upper and lower surfaces adjacent to this right surface position. The left captured image Ic' is pasted on the left surface of the rectangular parallelepiped and on the left half of each of the upper and lower surfaces adjacent to this left surface position. This makes it easy to create a 360-degree image centered on the vehicle 10 based on the front, rear, left, and right captured images Ia ' , Ib ' , Ic ' , and Id ' .

To give an idea of this 360-degree image, as shown in Figure 5(c), when a user places their head inside a rectangular prism, they can observe an image Ia ' of the front of the rectangular prism, an image Ib' of the right side of the vehicle 10 can be observed on the right half of the top surface, right surface, and right half of the bottom surface of the rectangular prism, an image Ic ' of the left side of the vehicle 10 can be observed on the left half of the top surface, left surface, and left half of the bottom surface of the rectangular prism, and an image Id' of the rear of the vehicle 10 can be observed on the rear surface of the rectangular prism.

Returning to Figure 3, the boundary portion processing unit 106 provides a boundary portion at the joints between the faces of the rectangular 360-degree image generated by the image pasting unit 105. This boundary portion satisfies the following requirements, for example: the scenery can be clearly seen, the scenery is not obstructed, and the unnaturalness of the joints can be eliminated.

Figure 6 shows examples of boundary areas. Figure 6(a) is an example in which a window frame 110a is provided as the boundary area. Figure 6(b) is an example in which a frame 110b is provided as the boundary area. Figure 6(c) is an example in which a line (solid line) 110c is provided as the boundary area. Figure 6(d) is an example in which a dotted line 110d is provided as the boundary area. Figure 6(e) is an example in which a color pick line 110e is provided as the boundary area. Figure 6(f) is an example in which a blur processing unit 110f is provided that performs blur processing using images of two surfaces corresponding to the joint portion as the boundary area.

By creating a boundary in this way, it is possible to reduce the sense of incongruity at the seams between captured images. In particular, by creating a boundary using blur processing, it is possible to more naturally reduce the sense of incongruity at the seams between captured images.

Here, in the rectangular parallelepiped 360-degree image, a captured image Ib ' of the right side of the vehicle 10 is pasted on the right half of the top surface and the right surface and the right half of the bottom surface, and the boundary portions between the top surface and the right surface and the bottom surface are not seams between the captured images. Similarly, in the rectangular parallelepiped 360-degree image, a captured image Ic ' of the left side of the vehicle 10 is pasted on the left half of the top surface and the left surface and the left half of the bottom surface, and the boundary portions between the top surface and the left surface and the bottom surface are not seams between the captured images.

In this way, by providing a boundary even if it is not at the seams between captured images, the sense of incongruity in a 360-degree image can be reduced. In this case, the captured images on the right and left are fisheye images that also include images above and below, and these images above and below are pasted onto the top and bottom surfaces of the rectangular prism. The human eye has a narrow vertical field of vision, so when looking at the images on the right or left, the images above and below are not visible. In other words, it is unnatural for the images above and below, which are not visible when looking at the images on the right or left normally, to be connected, and providing a boundary can reduce this sense of incongruity.

In addition to providing a boundary at the seams between the faces of the rectangular 360-degree image generated by the image pasting unit 105 as described above, the boundary processing unit 106 also provides similar boundary at the seams between the right and left captured images of the top and bottom faces. This reduces the sense of incongruity at these seams.

Returning to Figure 3, the display image generation unit 107 generates a display image (two-dimensional image) in a predetermined direction based on the 360-degree image after the boundary portion processing unit 106 has provided the boundary portion. This predetermined direction can be selected, for example, by a selection operation using the user operation unit 102. The display image generation unit 107 generates a display image in a predetermined direction, for example, by performing a process of capturing a rectangular 360-degree image with a virtual camera with a predetermined angle of view that is installed at the center position of the rectangular prism (see the head position in Figure 5 (c)) and has the capturing direction set in a predetermined direction. Figure 7 schematically shows an example of a display image when, for example, the predetermined direction is selected to be upward in front.

Furthermore, Figures 8 and 9 show examples of display images when the images Ia, Ib, Ic, and Id captured by cameras 12a, 12b, 12c, and 12d (see Figure 1) attached to the front, right side, left side, and rear exterior of vehicle 10 are as shown in Figure 4. Figure 8 shows a display image when the specified direction is selected as the upper front, and Figure 9 shows a display image when the specified direction is selected as the lower rear. Note that the boundaries in these display images have been formed by blurring.

Returning to Figure 3, the display unit 108 is composed of, for example, an LCD panel or an organic EL panel, and displays the display image generated by the display image generation unit 107. This allows the user of the remote monitoring device 100 to check the display image, i.e., the image of the surroundings of the vehicle 10. In this case, if the specified direction for generating the display image is changed, the direction of the display image generated by the display image generation unit 107 can be changed, and therefore the user can check the image of any direction around the vehicle 10.

The distribution unit 109 distributes the display image generated by the display image generation unit 107. This allows the end user (the user to whom the image is distributed) to view the display image, i.e., the image of the area around the vehicle 10. In this case, by allowing the end user to change the specified direction in which the display image is generated based on instructions from the end user, the end user can view an image of the area around the vehicle 10 in any direction.

The flowchart in Figure 10 shows an example of the processing steps for generating a display image in the remote monitoring device 100 shown in Figure 3 and displaying or distributing it.

First, in step ST1, the remote monitoring device 100 receives, via the receiving unit 103, captured images Ia, Ib, Ic, and Id of the front, right, left, and rear of the vehicle 10 captured by the cameras 12a, 12b, 12c, and 12d, which are sent from the transmitting device 13 (see Figure 2) of the vehicle 10.

Next, in step ST2, the remote monitoring device 100 uses the image cropping unit 104 to crop the central portion, i.e., the portion with least distortion, from each of the captured images Ia, Ib, Ic, and Id.

Next, in step ST3, the remote monitoring device 100 causes the image pasting unit 105 to paste the cut-out captured images Ia ' , Ib ' , Ic ' , and Id ' of the front, right, left, and rear onto each face of the rectangular parallelepiped to generate a 360-degree rectangular parallelepiped image.

Next, in step ST4, the remote monitoring device 100 uses the boundary processing unit 106 to create boundaries, such as by performing blur processing using two corresponding images at the joints between the faces of the rectangular 360-degree image.

Next, in step ST5, the remote monitoring device 100 generates a display image (two-dimensional image) in a specified direction based on the rectangular 360-degree image with a boundary in the display image generation unit 107.

Next, in step ST6, the remote monitoring device 100 displays the generated display image on the display unit 108 and distributes it via the distribution unit 109.

In the above example, specific objects contained in the captured images Ia, Ib, Ic, and Id, such as people, address signs, and advertising billboards, are distributed as is. However, from the perspective of privacy protection, etc., it is also possible to distribute the specific objects after performing identification processing that makes them unidentifiable.

In this case, it is possible to identify the area of a specific object using a segmentation process that uses pattern matching or a learning model, and then perform specific processing on this area to prevent the specific object from being identified, such as blurring, blacking out, or replacing it with a character.

Figure 11 shows examples of captured images Ia, Ib, Ic, and Id. For example, captured image Ia includes a specific object, such as person M. In this case, in the illustrated example, the area of person M in captured image Ia is identified, as shown by an enlarged view of the area of person M, and a blurring process is applied as a specific process to prevent person M from being identified. Although a detailed explanation will be omitted, similar processing is also applied to specific objects included in the other captured images Ib, Ic, and Id.

As described above, the captured images Ia, Ib, Ic, and Id are not distributed as they are, but rather portions with less distortion are cut out from the captured images Ia, Ib, Ic, and Id, and the cut-out captured images Ia ' , Ib ' , Ic ' , and Id ' are distributed (see step ST2 in FIGS. 4 and 10). Therefore, in practice, it is sufficient to determine whether the cut-out captured images Ia ' , Ib ' , Ic ' , and Id ' contain a specific object, and if so, to perform identification processing on the area of the specific object so that the specific object cannot be identified.

The flowchart of Figure 12 shows another example of the processing procedure for generating a display image in the remote monitoring device 100 shown in Figure 3 and displaying or distributing it. In this flowchart, processing of step ST2a has been added between steps ST2 and ST3 in the flowchart of Figure 10, and although detailed explanation will be omitted, the rest is the same as the flowchart of Figure 10.

In step ST2a, if the captured images Ia ' , Ib ' , Ic ' , and Id ' cut out by the image cutout unit 104 contain a specific object such as a person, the remote monitoring device 100 identifies the area of the specific object and performs a identification process on that area so that the specific object is not identified.

In this case, the captured images Ia ' , Ib ' , Ic ' , and Id ' of the front, right, left, and rear directions cut out in step ST3 are captured images after identification processing has been performed so that the specific object is not identified. Although detailed description will be omitted, in correspondence with the flowchart of Fig. 12, in the remote monitoring device 100 shown in Fig. 3, between the image cutout unit 104 and the image pasting unit 105, there is a processing unit that identifies the area of the specific object and performs identification processing on that area so that the specific object is not identified. Note that the function of this processing unit may be configured to be possessed by the image cutout unit 104 or the image pasting unit 105.

The user may also be able to freely select what to select as a specific object, for example, whether to select only people. In the case of people, it may also be possible to apply specific processing to only the facial area so that it is not identified. The user may also be able to select whether or not to apply specific processing to prevent the identification of specific objects.

"Software processing"
The process shown in the flowchart of Fig. 10 can be executed by hardware, but can also be executed by software. When a series of processes is executed by software, the program constituting the software is installed from a recording medium into a computer incorporated in dedicated hardware, or into, for example, a general-purpose computer that can execute various functions by installing various programs.

13 is a block diagram showing an example of the hardware configuration of a computer 400. The computer 400 has a CPU 401, a ROM 402, a RAM 403, a bus 404, an input/output interface 405, an input unit 406, an output unit 407, a storage unit 408, a drive 409, a connection port 410, and a communication unit 411. Note that the hardware configuration shown here is an example, and some of the components may be omitted. Furthermore, the computer 400 may further include components other than those shown here.

The CPU 401 functions, for example, as an arithmetic processing device or control device, and controls the overall operation or part of the operation of each component based on various programs recorded in the ROM 402, RAM 403, memory unit 408, or removable recording medium 501.

ROM 402 is a means for storing programs loaded into CPU 401 and data used for calculations. RAM 403 temporarily or permanently stores, for example, programs loaded into CPU 401 and various parameters that change as appropriate when the programs are executed.

The CPU 401, ROM 402, and RAM 403 are connected to each other via a bus 404. Meanwhile, various components are connected to the bus 404 via an interface 405.

The input unit 406 may include, for example, a mouse, keyboard, touch panel, button, switch, lever, etc. Furthermore, the input unit 406 may also include a remote controller (hereinafter referred to as a remote control) capable of transmitting control signals using infrared rays or other radio waves.

The output unit 407 is a device capable of visually or audibly notifying the user of the acquired information, such as a display device such as a CRT (Cathode Ray Tube), LCD, or organic EL, an audio output device such as a speaker or headphones, a printer, a mobile phone, or a facsimile.

The memory unit 408 is a device for storing various types of data. Examples of the memory unit 408 include a magnetic memory device such as a hard disk drive (HDD), a semiconductor memory device, an optical memory device, or a magneto-optical memory device.

The drive 409 is a device that reads information recorded on a removable recording medium 501, such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, or writes information to the removable recording medium 501.

The removable recording medium 501 may be, for example, DVD media, Blu-ray (registered trademark) media, HD DVD media, various semiconductor storage media, etc. Of course, the removable recording medium 501 may also be, for example, an IC card equipped with a contactless IC chip, or an electronic device.

The connection port 410 is a port for connecting an external device 502, such as a USB (Universal Serial Bus) port, an IEEE 1394 port, a SCSI (Small Computer System Interface), an RS-232C port, or an optical audio terminal. The external device 502 is, for example, a printer, a portable music player, a digital camera, a digital video camera, or an IC recorder.

The communication unit 411 is a communication device for connecting to the network 503, such as a communication card for wired or wireless LAN, Bluetooth (registered trademark), or WUSB (Wireless USB), a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various types of communication.

In addition, the program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or it may be a program that processes in parallel or at the required timing, such as when called.

As explained above, the remote monitoring device 100 shown in Figure 3 generates a rectangular 360-degree image based on four captured images of the front, rear, left, and right sides of the vehicle 10, and then generates a display image for a specified direction based on this 360-degree image. Compared to generating a display image for a specified direction based on a spherical or cylindrical 360-degree image obtained by performing distortion correction processing, stitching processing, etc., it is possible to obtain a display image (2D image) that looks less strange without increasing the amount of processing. Furthermore, because the processing load is low, processing of each frame of video can easily be performed in real time.

In addition, in the remote monitoring device 100 shown in Figure 3, when generating a rectangular 360-degree image, boundaries are provided at the joints between surfaces, etc., so that even if a rectangular 360-degree image is generated by simply attaching captured images from different directions to each surface, it is possible to reduce the sense of incongruity at the joints between the captured images.

2. Modified Examples
In the above-described embodiment, an example has been shown in which a rectangular parallelepiped 360-degree image is generated based on captured images obtained by four cameras 12a to 12d attached to the front, rear, left, and right sides of the vehicle 10. However, in the present technology, the number of captured images is not limited to four, and a rectangular parallelepiped 360-degree image may be generated based on other numbers of captured images. For example, it is also conceivable to generate a rectangular parallelepiped 360-degree image based on captured images (e.g., fisheye images) captured by two cameras, one facing the front right and one facing the rear left.

In addition, while the above-described embodiment shows an example of processing images captured by a camera attached to a vehicle 10, similar examples of processing images captured by a camera attached to other mobile devices such as drones or robots are also possible. In such cases, the user will be able to remotely monitor images of the area around the mobile device.

In the above-described embodiment, an example was shown in which the distribution unit 109 distributed to the end user a display image (2D image) of a predetermined direction generated based on a rectangular parallelepiped 360-degree image. However, a configuration in which a rectangular parallelepiped 360-degree image is distributed to the end user is also conceivable. In this case, the end user would use a predetermined image processing application to generate and display a display image of a predetermined direction from the rectangular parallelepiped 360-degree image. Furthermore, a configuration in which a display image of a predetermined direction (2D image) generated by the remote monitoring device 100 is distributed to the vehicle 10 and displayed on a display unit provided in the passenger compartment of the vehicle 10 is also conceivable. In this case, the occupants of the vehicle 10 can view the display image of the 360-degree direction while facing the direction of travel.

Furthermore, while preferred embodiments of the present disclosure have been described in detail with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person with ordinary skill in the technical field of the present disclosure could conceive of various modified or revised examples within the scope of the technical ideas set forth in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

Furthermore, the effects described in this specification are merely descriptive or exemplary and are not limiting. In other words, the technology disclosed herein may achieve other effects in addition to or in place of the above-described effects that would be apparent to one skilled in the art from the description herein.

The present technology can also be configured as follows.
(1) a 360-degree image generating unit that generates a rectangular parallelepiped 360-degree image based on a plurality of captured images;
The image processing device includes a display image generation unit that generates a display image in a predetermined direction based on the 360-degree image.
(2) The image processing device according to (1), wherein the 360-degree image generating unit provides a boundary portion at a joint between faces of the rectangular parallelepiped 360-degree image.
(3) The image processing device according to (2), wherein the boundary portion is generated by blurring processing using images of two surfaces corresponding to the joint portion.
(4) The image processing device described in any one of (1) to (3), wherein the display image generation unit generates the display image in the specified direction by performing a process of capturing the rectangular 360-degree image with a virtual camera with a specified angle of view, the imaging direction of which is set in the specified direction, and the rectangular 360-degree image is captured in the specified direction.
(5) The image processing device according to any one of (1) to (4), further comprising: a selection unit that selects the predetermined direction based on a user operation.
(6) The image processing device according to any one of (1) to (5), further comprising: a distribution unit that distributes the display image.
(7) The image processing device according to any one of (1) to (6), further comprising a display unit that displays the display image.
(8) The image processing device described in any one of (1) to (7), wherein the plurality of captured images are four captured images obtained by four imaging units attached to the exterior of the vehicle, each capturing an image of the front, right, left, and rear.
(9) The image processing device described in (8) above, wherein the 360-degree image generation unit pastes the image obtained by the imaging unit that images the front onto a front region of the rectangular parallelepiped, pastes the image obtained by the imaging unit that images the rear onto a rear region of the rectangular parallelepiped, pastes the image obtained by the imaging unit that images the right onto a right surface region of the rectangular parallelepiped and onto the right half regions of each of the upper and lower surfaces adjacent to the right surface region, and pastes the image obtained by the imaging unit that images the left onto a left surface region of the rectangular parallelepiped and onto the left half regions of each of the upper and lower surfaces adjacent to the left surface region.
(10) The imaging units for capturing images of the front and rear are imaging units for capturing wide-angle images,
The image processing device according to (8) or (9), wherein the imaging units for capturing images to the right and left are imaging units for capturing fisheye images.
(11) The image processing device according to (10), wherein the 360-degree image generating unit cuts out and uses a central portion of each of the four captured images.
(12) The image processing device according to any one of (8) to (11), further comprising a receiving unit that receives the four captured images from the vehicle.
(13) The image processing device according to any one of (1) to (12), wherein the plurality of captured images are captured images after a specific object area has been subjected to a specific processing so that the specific object is not identified.
(14) generating a rectangular parallelepiped 360-degree image based on a plurality of captured images;
An image processing method comprising the step of generating an image for display in a predetermined direction based on the 360-degree image.

10: Vehicle 12a to 12d: Cameras 13: Transmitting device 100: Remote monitoring device 101: Control unit 102: User operation unit 103: Receiving unit 104: Image cropping unit 105: Image pasting unit 106: Boundary part processing unit 107: Display image generating unit 108: Display unit 109: Distribution unit

Claims (14)

a 360-degree image generating unit that generates a rectangular parallelepiped 360-degree image by attaching a plurality of captured images to each surface of a rectangular parallelepiped ;
The image processing device includes a display image generation unit that generates a display image in a predetermined direction based on the 360-degree image. The image processing device according to claim 1 , wherein the 360-degree image generating unit provides a boundary portion at a joint between faces of the rectangular parallelepiped 360-degree image. The image processing device according to claim 2 , wherein the boundary portion is generated by blurring processing using images of two surfaces corresponding to the joint portion. The image processing device according to claim 1, wherein the display image generation unit generates the display image in the predetermined direction by performing a process of capturing the rectangular parallelepiped 360-degree image with a virtual camera with a predetermined angle of view, the imaging direction of which is set in the predetermined direction, and the rectangular parallelepiped is installed at a center position of the rectangular parallelepiped. The image processing device according to claim 1 , further comprising a selection unit that selects the predetermined direction based on a user operation. The image processing device according to claim 1 , further comprising a distribution unit that distributes the display image. The image processing device according to claim 1 , further comprising a display unit that displays the display image. The image processing device according to claim 1 , wherein the plurality of captured images are four captured images obtained by four imaging units attached to the exterior of a vehicle, each capturing an image of a front, right, left, and rear direction, respectively. 9. The image processing device according to claim 8, wherein the 360-degree image generation unit pastes the captured image obtained by the imaging unit that captures the front side onto a front region of the rectangular parallelepiped, pastes the captured image obtained by the imaging unit that captures the rear side onto a rear region of the rectangular parallelepiped, pastes the captured image obtained by the imaging unit that captures the right side onto a right region of the rectangular parallelepiped and onto right half regions of each of the upper and lower surfaces adjacent to the right region, and pastes the captured image obtained by the imaging unit that captures the left side onto a left region of the rectangular parallelepiped and onto left half regions of each of the upper and lower surfaces adjacent to the left region. the imaging units for capturing images of the front and rear are imaging units for capturing wide-angle images,
The image processing device according to claim 8 , wherein the imaging units for capturing images to the right and left are fisheye image imaging units. The image processing device according to claim 10 , wherein the 360-degree image generating unit cuts out and uses a central portion of each of the four captured images. The image processing device according to claim 8 , further comprising a receiving unit that receives the four captured images from the vehicle. The image processing device according to claim 1 , wherein the plurality of captured images are captured images after a specific object area has been subjected to a specific processing so that the specific object is not identified. A procedure for generating a 360-degree rectangular parallelepiped image by attaching a plurality of captured images to each surface of a rectangular parallelepiped ;
An image processing method comprising the step of generating an image for display in a predetermined direction based on the 360-degree image.