JP7807737B2 - Image Generation System - Google Patents

Description

This disclosure relates to an image generation system that generates images to be displayed on a remote control display device for remotely operating a work machine.

JP 2018-207244 A (Patent Document 1) discloses a remote control system that is mounted on a work machine having a rotating body that rotates around a rotating axis and a work implement supported on the rotating body, and that includes an imaging device that captures images, a display device that is external to the work machine, and a control device that is external to the work machine and is capable of communicating with the work machine, the control device having an image data acquisition unit that acquires images from the imaging device and a display control unit that displays the images acquired by the image data acquisition unit on the display device.

Japanese Patent Application Laid-Open No. 2018-207244

When an imaging device is mounted on the body of a work machine, structures that make up the body may be within the field of view of the imaging device. These structures may appear in the captured image, reducing the visibility of the subject being imaged.

This disclosure proposes an image generation system that can improve the visibility of the subject being imaged.

In accordance with the present disclosure, an image generation system is proposed that generates images to be displayed on a remote control display device for remotely operating a work machine. The work machine has an imaging device that captures images of an object present at a work site. The image generation system includes a display device, an image recording unit that records images captured by the imaging device, and an image processing computer that processes the captured images. The image processing computer acquires a latest image, which is the most recent captured image. The image processing computer generates a converted image by converting a previous captured image recorded in the image recording unit to match the orientation of the imaging device when the imaging device captured the latest image. The image processing computer displays on the display device a complemented image in which blind spots in the latest image, where the object is obscured by an obstruction, are complemented with the converted image.

The image generation system according to the present disclosure can improve the visibility of the imaged subject.

FIG. 1 is an external view of a hydraulic excavator. FIG. 1 is a diagram schematically illustrating an example of a remote control system for a hydraulic excavator. FIG. 1 is a diagram schematically illustrating an example of a remote control device. FIG. 1 is a functional block diagram showing an example of the configuration of an image generation system. 1 is a flowchart illustrating an example of an image generation method. FIG. 10 is a diagram illustrating an example of a latest image. FIG. 7 is an enlarged view of region VII in FIG. 6. FIG. 10 is a diagram showing an example of a previous image. FIG. 10 is a diagram showing an image in which the viewpoint is transformed. FIG. 10 is a diagram illustrating a process of combining images. FIG. 10 is a diagram illustrating an example of a composite image. FIG. 10 is a diagram in which a plurality of past images and the latest image are combined. FIG. 10 is a functional block diagram showing an example of the configuration of an image generation system according to a second embodiment. FIG. 10 is a functional block diagram showing an example of the configuration of an image generation system according to a third embodiment. 11 is a flowchart illustrating an example of an image generating method according to the third embodiment. FIG. 10 is a schematic diagram showing an image capturing situation by a camera according to a fourth embodiment. FIG. 13 is a diagram illustrating an example of a captured image according to the fourth embodiment. FIG. 13 is a diagram illustrating an example of a composite image according to the fourth embodiment.

The following describes the embodiments with reference to the accompanying drawings. In the following description, identical components are designated by the same reference numerals. Their names and functions are also the same. Therefore, detailed descriptions of them will not be repeated.

[First embodiment]
Fig. 1 is an external view of a hydraulic excavator 100 as an example of a work machine remotely controlled by a remote control system based on an embodiment. As shown in Fig. 1, in this example, the hydraulic excavator 100 will be mainly used as an example of the work machine.

The hydraulic excavator 100 is located at a work site and performs work there. The hydraulic excavator 100 performs excavation work of objects such as earth and sand or ore, and performs loading work of loading the excavated objects onto transport machinery such as a dump truck. The hydraulic excavator 100 of this embodiment is a work machine that can be remotely controlled. The hydraulic excavator 100 is controlled by radio signals from a remote location using a remote control system. The hydraulic excavator 100 is not equipped with a control function for an onboard operator.

The hydraulic excavator 100 has a main body 1. The main body 1 has a rotating body 3 and a running body 5.

The running body 5 has a pair of tracks 5Cr and a travel motor 5M. The hydraulic excavator 100 is capable of traveling by rotation of the tracks 5Cr. The travel motor 5M is provided as a drive source for the running body 5. The travel motor 5M is a hydraulic motor that is hydraulically operated. Note that the running body 5 may also have wheels (tires).

The rotating unit 3 is placed on and supported by the running unit 5. The rotating unit 3 is mounted on the running unit 5 so as to be able to rotate relative to the running unit 5 around a rotation axis RX. The rotating unit 3 has a vehicle compartment 4. The front of the vehicle compartment 4 is covered with a transparent window panel 4W. The window panel 4W is made of tempered glass or the like. A space 4S is formed within the vehicle compartment 4.

The passenger compartment 4 has multiple pillars that support the roof portion. The multiple pillars include a left front pillar 4FP1 and a right front pillar 4FP2. The left front pillar 4FP1 is located at the front left corner of the passenger compartment 4. The right front pillar 4FP2 is located at the front right corner of the passenger compartment 4. A window pane 4W is located between the left front pillar 4FP1 and the right front pillar 4FP2. The left edge of the window pane 4W is attached to the left front pillar 4FP1, and the right edge of the window pane 4W is attached to the right front pillar 4FP2.

The rotating body 3 has an engine room 9 that houses the engine, and a counterweight provided at the rear of the rotating body 3. The engine room 9 contains an engine 31 and a hydraulic pump 32, which will be described later.

A handrail 19 is provided in front of the engine compartment 9 on the rotating unit 3. An antenna 21 is attached to the handrail 19. The antenna 21 is, for example, an antenna for GNSS (Global Navigation Satellite Systems). The antenna 21 includes a first antenna 21A and a second antenna 21B that are mounted on the rotating unit 3 and spaced apart in the vehicle width direction.

The hydraulic excavator 100 is equipped with a hydraulically operated work implement 2. The work implement 2 is supported on a revolving unit 3. The work implement 2 has a boom 6, an arm 7, and a bucket 8. The base end of the boom 6 is rotatably connected to the revolving unit 3 via a boom foot pin 13. The base end of the arm 7 is rotatably connected to the tip of the boom 6 via a boom tip pin 14. The bucket 8 is rotatably connected to the tip of the arm 7 via an arm tip pin 15. The arm 7 and bucket 8 are each movable members that can move at the tip of the boom 6.

The bucket 8 has multiple blades. The bucket 8 does not have to have blades. The tip of the bucket 8 may be formed from a straight steel plate. The bucket 8 is an example of an attachment that can be attached to the tip of the work implement 2. Depending on the type of work, the attachment can be replaced with a breaker, grapple, lifting magnet, or other attachment.

In this embodiment, the positional relationships of each part of the hydraulic excavator 100 will be explained using the work machine 2 as the reference.

The boom 6 of the work implement 2 rotates around the boom foot pin 13 relative to the revolving unit 3. The trajectory of a specific portion of the boom 6 that rotates relative to the revolving unit 3, such as the tip of the boom 6, is an arc, and a plane containing this arc is identified. When the hydraulic excavator 100 is viewed from above, this plane is represented as a straight line. The direction in which this straight line extends is the fore-and-aft direction of the main body 1 of the hydraulic excavator 100 or the fore-and-aft direction of the revolving unit 3, and will hereinafter be referred to simply as the fore-and-aft direction. The left-and-right direction (vehicle width direction) of the main body 1 of the hydraulic excavator 100 or the left-and-right direction of the revolving unit 3 is a direction perpendicular to the fore-and-aft direction in a plan view, and will hereinafter be referred to simply as the left-and-right direction. The up-and-down direction of the vehicle main body or the up-and-down direction of the revolving unit 3 is a direction perpendicular to the plane defined by the fore-and-aft direction and the left-and-right directions, and will hereinafter be referred to simply as the up-and-down direction.

In the front-to-back direction, the side where the work implement 2 protrudes from the main body 1 of the hydraulic excavator 100 is the front direction, and the direction opposite the front direction is the rear direction. Looking forward, the right and left sides in the left-to-right direction are the right and left directions, respectively. In the up-down direction, the side with the ground facing the ground is the bottom side, and the side with the sky facing the top side.

The work implement 2 has a boom cylinder 10, an arm cylinder 11, and a bucket cylinder 12. The boom cylinder 10 drives the boom 6. The arm cylinder 11 drives the arm 7. The bucket cylinder 12 drives the bucket 8. The boom cylinder 10, arm cylinder 11, and bucket cylinder 12 are each hydraulic cylinders driven by hydraulic oil.

The hydraulic excavator 100 is equipped with a camera 20. The camera 20 is an imaging device that captures images of an object present at a work site around the hydraulic excavator 100 and acquires an image of the object. The camera 20 is mounted on the revolving body 3.

Images of objects captured by the camera 20 include construction objects being constructed at the work site. Construction objects include excavation objects being excavated by the work implement 2 of the hydraulic excavator 100. Excavation objects include the excavation object before excavation (i.e., the current terrain), the excavation object being excavated, and the excavation object after excavation. Images of objects captured by the camera 20 include obstacles around the hydraulic excavator 100.

The subject imaged by the camera 20 includes at least a portion of the hydraulic excavator 100. The subject imaged by the camera 20 includes at least a portion of the work implement 2. The subject imaged by the camera 20 includes other work machines arranged around the hydraulic excavator 100. The other work machines include transport machines that transport the excavation target of the hydraulic excavator 100. The other work machines include dump trucks.

The camera 20 has an optical system and an image sensor that receives light that passes through the optical system. The image sensor includes a CCD (Couple Charged Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

The cameras 20 include a right front camera 20A, a right side camera 20B, a rear camera 20C, a left side camera 20D, and a front camera 20E.

The right front camera 20A and the right side camera 20B are positioned on the right edge of the top surface of the rotating unit 3. The right front camera 20A is positioned further forward than the right side camera 20B. The right front camera 20A and the right side camera 20B are positioned side by side, front to back, near the center of the rotating unit 3 in the fore-and-aft direction. The right front camera 20A captures images of the area to the right front of the rotating unit 3. The optical axis of the right front camera 20A extends diagonally forward and to the right from the right front camera 20A. The right side camera 20B captures images of the area to the right rear of the rotating unit 3. The optical axis of the right side camera 20B extends diagonally rear and to the right from the right side camera 20B.

The rear camera 20C is located at the rear end of the rotating unit 3 in the longitudinal direction and at the center of the rotating unit 3 in the left-right direction. A counterweight is installed at the rear end of the rotating unit 3 to balance the vehicle body during mining, etc. The rear camera 20C is located on the top surface of the counterweight. The rear camera 20C captures images behind the rotating unit 3. The optical axis of the rear camera 20C extends rearward from the rear camera 20C. The left side camera 20D is located on the left edge of the top surface of the rotating unit 3. The left side camera 20D is located near the center of the rotating unit 3 in the longitudinal direction. The left side camera 20D captures images to the left of the rotating unit 3. The optical axis of the left side camera 20D extends leftward from the left side camera 20D.

The front camera 20E is located at the front of the rotating unit 3. The front camera 20E is located on the left side of the work implement 2. The front camera 20E is located inside the vehicle cabin 4. The front camera 20E is housed in the vehicle cabin 4. The front camera 20E is located in the space 4S within the vehicle cabin 4. The front camera 20E is located at a position that would be the operator's eye level if the operator were to board the vehicle cabin 4. The front camera 20E captures images of the area in front of the rotating unit 3 through the window panel 4W. The optical axis of the front camera 20E extends forward from the front camera 20E. The image captured by the front camera 20E may include the terrain in front of the rotating unit 3 and at least a portion of the work implement 2.

The hydraulic excavator 100 is equipped with a communication device 22 and a vehicle controller 26. The communication device 22 includes a communication antenna. The communication antenna is located, for example, above the engine compartment 9. The communication device 22 receives control signals transmitted from a remote location. The vehicle controller 26 controls the engine 31, work implement 2, and rotating bed 3 based on the received control signals. The communication device 22 also transmits signals containing information about the hydraulic excavator 100 to the remote location. The communication device 22 transmits images of the object captured by the camera 20, position information and attitude information of the hydraulic excavator 100, and the like to the remote location.

Figure 2 is a diagram schematically illustrating an example of a remote control system 200 for a hydraulic excavator 100. The remote control system 200 exists outside the hydraulic excavator 100. The hydraulic excavator 100 is remotely controlled by the remote control system 200. The remote control system 200 is provided, for example, in a remote control facility. The remote control facility may be installed at the work site where the hydraulic excavator 100 is located, or may be installed in a remote location away from the work site.

The remote operation system 200 mainly comprises a remote operation device 40, a display device 50, a remote controller 60, and a communication device 72. The remote operation device 40, the display device 50, the remote controller 60, and the communication device 72 are each provided separately from the hydraulic excavator 100.

The hydraulic excavator 100 is connected to the remote operation system 200 via a network 400. The network 400 includes at least one of the Internet, a local area network (LAN), a mobile phone network, and a satellite network. The network 400 may also include a relay station that relays the data being communicated.

The communication device 22 of the hydraulic excavator 100 transmits a signal containing information about the hydraulic excavator 100 to the remote operation system 200 via the network 400. The remote operation system 200 processes the image of the object captured by the camera 20 received by the communication device 72 in the remote controller 60 and displays it on the display device 50.

Figure 3 is a schematic diagram showing an example of a remote control device 40. The remote control device 40 and display device 50 shown in Figure 3 are located in a remote control room provided in a remote control facility. The remote control device 40 is operated by an operator seated in an operator seat 45. The operator sits in the operator seat 45 so as to directly face the display screen of the display device 50. The operator visually confirms the situation at the work site via the display device 50. The operator operates the remote control device 40 while looking at the display screen of the display device 50.

The remote control device 40 includes a left operating lever 41 and a right operating lever 42 that are operated to operate the work implement 2 and the rotating body 3, and a left traveling lever 43 and a right traveling lever 44 that are operated to operate the traveling body 5.

The left operating lever 41 is located to the left of the operator's seat 45. An operator seated on the operator's seat 45 grasps the left operating lever 41 with their left hand and operates it. The left operating lever 41 operates the arm 7 and the rotating unit 3. The left operating lever 41 accepts operator input regarding the rotation direction of the rotating unit 3 and the up and down movement of the arm 7. Operating the left operating lever 41 in the forward and backward directions corresponds to the rotation of the rotating unit 3, and the right and left rotation of the rotating unit 3 is performed in response to the forward and backward operation. Operating the left operating lever 41 in the left and right directions corresponds to the operation of the arm 7, and the arm 7 is moved upward and downward in response to the left and right operation.

The right working lever 42 is located to the right of the operator's seat 45. An operator seated on the operator's seat 45 grasps the right working lever 42 with their right hand and operates the right working lever 42. The boom 6 and bucket 8 are operated by the right working lever 42. The right working lever 42 accepts operator input regarding the up and down movement of the boom 6 and the up and down movement of the bucket 8. Operation of the right working lever 42 in the forward and backward directions corresponds to operation of the boom 6, and the boom 6 is raised and lowered in response to this operation. Operation of the right working lever 42 in the left and right directions corresponds to operation of the bucket 8, and the bucket 8 is moved downward and upward in response to this operation in the left and right directions.

A left console 48 is also located to the left of the pilot's seat 45. A right console 49 is also located to the right of the pilot's seat 45.

The left travel lever 43 and right travel lever 44 are located in front of the operator seat 45. The left travel lever 43 and right travel lever 44 are located side by side, with the left travel lever 43 on the left side and the right travel lever 44 on the right side. The left travel lever 43 and right travel lever 44 receive input from the operator regarding the travel of the running unit 5. The left track 5Cr of the running unit 5 moves forward or backward in response to forward or backward operation of the left travel lever 43. The right track 5Cr of the running unit 5 moves forward or backward in response to forward or backward operation of the right travel lever 44.

The display device 50 is located in front of the operator's seat 45. The display device 50 is located farther from the operator's seat 45 than the left travel lever 43 and right travel lever 44. The display device 50 includes a flat panel display such as a liquid crystal display or organic EL display. The display device 50 displays images. The display device 50 is capable of displaying captured images taken by the camera 20 and acquired via the network 400. The display device 50 is also capable of displaying images after the captured images have been processed by the remote controller 60. Specifically, the display device 50 is capable of displaying images that compensate for blind spots by combining multiple captured images taken by the camera 20.

While Figure 3 shows an example in which one display screen constitutes the display device 50, the display device 50 may include multiple display screens. The display device 50 may include a total of five display screens, with one display screen arranged adjacent to each side of the central display screen, or may include a total of nine display screens arranged adjacent to each other in three columns and three rows, or may include any number of display screens arranged in any other manner. In addition to the display screen located in front of the pilot's seat 45, the display device 50 may also include display screens located to the left of the pilot's seat 45, to the right of the pilot's seat, and/or above the pilot's seat 45.

The display device 50 may be a monitor permanently installed in the remote control room, or a head-mounted display or head-up display. The display device 50 may be a monitor for a personal computer, or a mobile device such as a tablet computer or smartphone.

Figure 4 is a functional block diagram showing an example configuration of an image generation system that generates an image to be displayed on the remote control display device 50, based on an embodiment. The remote control system 200 is located in a remote control facility outside the hydraulic excavator 100. The remote control system 200 is connected to the hydraulic excavator 100 via a network 400.

The hydraulic excavator 100 is configured so that a hydraulic pump 32 is driven by an engine 31, and hydraulic oil discharged from the hydraulic pump 32 is supplied to various hydraulic actuators 34 via a directional control valve 33. By controlling the supply and discharge of hydraulic pressure to the hydraulic actuators 34, the operation of the work implement 2, the rotation of the rotating body 3, and the traveling operation of the traveling body 5 are controlled. The hydraulic actuators 34 include the boom cylinder 10, arm cylinder 11, bucket cylinder 12, and traveling motor 5M shown in FIG. 1, as well as a swing motor.

The engine 31 is, for example, a diesel engine. The output of the engine 31 is controlled by adjusting the amount of fuel injected into the engine 31 in accordance with the output of a control signal from the vehicle controller 26. The engine 31 has a drive shaft for connection to the hydraulic pump 32.

The hydraulic pump 32 is connected to the drive shaft of the engine 31. The rotational driving force of the engine 31 is transmitted to the hydraulic pump 32, thereby driving the hydraulic pump 32. The hydraulic pump 32 is a variable displacement hydraulic pump that has a swash plate and changes the discharge capacity by changing the tilt angle of the swash plate. The hydraulic pump 32 supplies hydraulic oil used to drive the work machine 2, travel the traveling body 5, and rotate the rotating body 3. The hydraulic oil discharged from the hydraulic pump 32 is reduced to a constant pressure by a pressure reducing valve and supplied to the directional control valve 33.

The directional control valve 33 is a spool-type valve that switches the direction of hydraulic oil flow by moving a rod-shaped spool. The directional control valve 33 has separate spools that adjust the amount of hydraulic oil supplied to the boom cylinder 10, arm cylinder 11, bucket cylinder 12, travel motor 5M, and swing motor. The amount of hydraulic oil supplied to the hydraulic actuator 34 is adjusted by moving each spool axially in accordance with the output of a control signal from the vehicle controller 26.

The remote controller 60 has a control signal generation unit 69. The control signal generation unit 69 generates a control signal for operating the hydraulic excavator 100 based on the operation of the remote control device 40 by the operator. The communication device 72 transmits the control signal generated by the control signal generation unit 69 to the hydraulic excavator 100 via the network 400.

The captured image captured by the camera 20 shown in FIG. 1 is input to the vehicle body controller 26. The vehicle body controller 26 has an image processing unit 260. The image processing unit 260 processes the captured image input to the vehicle body controller 26. The communication device 22 transmits the processed image to the remote control system 200 via the network 400.

The remote controller 60 has an image output unit 67. The image output unit 67 outputs the image processed by the image processing unit 260 to the display device 50, causing the image to be displayed on the display device 50.

Figure 5 is a flowchart showing an example of an image generation method based on an embodiment. A method for generating an image to be displayed on the remote control display device 50 for remotely operating the hydraulic excavator 100 of this embodiment will be described below with reference to Figures 4, 5, and subsequent figures. Below, an example of capturing an image of an object while the rotating body 3 is rotating will be described.

As shown in FIG. 5, first, in step S1, the latest image is acquired. The image processing unit 260 has an image acquisition unit 261. The image acquisition unit 261 acquires the latest image, which is the most recent image captured by the camera 20 of the hydraulic excavator 100.

Figure 6 is a diagram showing an example of the latest image. Figure 6 shows image IMG captured by forward camera 20E. Image IMG includes a portion of the work implement 2. Image IMG includes image IFP1 of the left front pillar 4FP1 of the passenger compartment 4 and image IFP2 of the right front pillar 4FP2. Image IMG includes a landscape LS. Shapes such as circles, crosses, triangles, and rectangles in Figure 6 indicate portions of a simplified landscape LS. The landscape LS is blocked by images IFP1 and IFP2. The left front pillar 4FP1 and right front pillar 4FP2 correspond to obstructions that block the image capture target in this embodiment.

The rotation direction RD indicated by the curved arrow in Figure 6 indicates the rotation direction of the rotating unit 3. In this embodiment, the rotating unit 3 is rotating to the right.

Figure 7 is an enlarged view of region VII in Figure 6. Region VII is set to include image IFP1 of the left front pillar 4FP1, which is an obstructing object, as well as scenery LS that is not obstructed by the obstructing object. In the example shown in Figure 7, unobstructed scenery LS exists on both the left and right sides of image IFP1 in image IMG1.

In step S2, a previous image, which is an image captured before the latest image, is read. The hydraulic excavator 100 has a recording unit 270. Previous image data 278 is recorded in the recording unit 270. The recording unit 270 records images captured by the camera 20 and acquired by the image acquisition unit 261 as previous images. The image processing unit 260 has a movement information acquisition unit 262. The movement information acquisition unit 262 reads previous images from the recording unit 270. Typically, the movement information acquisition unit 262 reads previous images, which are previous images captured immediately before the latest image.

Figure 8 is a diagram showing an example of a previous image. Image IMG2 shown in Figure 8 is an enlarged view of the area of the previous image corresponding to area VII shown in Figure 6. As with image IMG1 in Figure 7, in image IMG2, unobstructed scenery LS is present on both the left and right sides of image IFP1 of left front pillar 4FP1. Because the position of the left front pillar 4FP1, which is an obstruction, does not change relative to front camera 20E, the position of image IFP1 of left front pillar 4FP1 remains unchanged between images IMG1 and IMG2. However, because the images were captured while the rotating unit 3 was rotating, the scenery LS included in images IMG1 and IMG2 has changed. Portions of the scenery LS that were not visible in image IMG1 are visible in image IMG2.

Specifically, in image IMG1 in Figure 7, the left half of the circle is obscured by image IFP1 of the left front pillar 4FP1. In contrast, in image IMG2 in Figure 8, the circle is not obscured and the entire circle is shown. Image IMG1 includes only a portion of the x, while image IMG2 shows the entire x.

In step S3, the optical flow is calculated. The movement information acquisition unit 262 performs image processing on the latest image and the immediately preceding image. The movement information acquisition unit 262 calculates the movement vectors of pixels corresponding to the same feature points in two temporally consecutive images, specifically image IMG1 (Figure 7) and image IMG2 (Figure 8). In this embodiment, the revolving unit 3 is turning right, so in image IMG1, which is a part of the latest image, the scenery LS has moved leftward compared to image IMG2, which is a part of the immediately preceding image. The movement information acquisition unit 262 compares image IMG1 and image IMG2 and calculates the amount of movement of the scenery LS from the time the immediately preceding image was captured to the time the latest image was captured, based on the difference between image IMG1 and image IMG2.

In step S4, a converted image is generated. The image processing unit 260 has a viewpoint conversion unit 264. The viewpoint conversion unit 264 converts the previous image to match the orientation of the camera 20 when the camera 20 captured the latest image. The viewpoint conversion unit 264 shifts the previous image by the amount of movement of the scenery LS calculated in step S3, so that the scenery LS included in the previous image exactly overlaps with the scenery LS included in the latest image.

Figure 9 shows an image in which the viewpoint has been transformed based on optical flow. The portion of image IMG3 shown in Figure 9 surrounded by a dashed line corresponds to image IMG2 shown in Figure 8 shifted to the left in the figure. The circles and crosses included in image IMG3 and the circles and crosses included in image IMG1, which is part of the latest image shown in Figure 7, overlap each other.

As the rotating body 3 rotates, the front camera 20E undergoes translation and rotation. Because the front camera 20E is placed near the rotation axis RX, the change in the scenery LS between the latest image and the immediately preceding image is dominated by the rotation of the front camera 20E. This makes it possible to utilize the rotation of the front camera 20E. A transformed image can be generated by a simple transformation that is uniquely determined from the amount of rotation of the front camera 20E. Furthermore, the transformed image can be approximated by simply moving the image in one direction.

In step S5, blind spot areas are detected. The image processing unit 260 has an area acquisition unit 263. The area acquisition unit 263 performs image processing on the latest image and the immediately preceding image. The area acquisition unit 263 determines blind spot areas caused by the structure of the vehicle interior 4. The area acquisition unit 263 compares an image IMG1, which is a portion of the latest image, with an image IMG2, which is a portion of the immediately preceding image, and may recognize areas where there is little change and the amount of change is below a threshold as blind spot areas where an image IFP1 of the left front pillar 4FP1 is present and where the scenery LS is obstructed by the left front pillar 4FP1. The area acquisition unit 263 may detect the boundary of an obstruction obstructing the scenery LS. The area acquisition unit 263 may detect the blind spot area as a bitmap.

The left front pillar 4FP1 does not change position relative to the forward camera 20E, so even when the rotating unit 3 rotates, the left front pillar 4FP1 appears in the same position in the image. In other words, an object that appears in the same position even when the unit rotates is an obstruction. This makes it easy to determine the blind spot area.

In step S6, masking is performed. In step S7, the images are combined. The image processing unit 260 has an image combination unit 265. The image combination unit 265 masks blind spot areas in the latest image so that the blind spot areas are not used in combining images. Figure 10 is a diagram showing the image combination process. Image IMG41 shown in Figure 10 corresponds to the portion of image IMG1 shown in Figure 7 to the left of image IFP1 of the left front pillar 4FP1. Image IMG42 corresponds to the portion of image IMG1 shown in Figure 7 to the right of image IFP1 of the left front pillar 4FP1.

The image synthesis unit 265 generates a synthetic image by synthesizing the images so that the most recent images, images IMG41 and IMG42, are superimposed on image IMG3, which is a converted image obtained by viewpoint-converting the immediately preceding image. FIG. 11 is a diagram showing an example of a synthetic image. In image IMG5 shown in FIG. 11, image IMG51 is located to the left of image IFP1 of the left front pillar 4FP1, and images IMG52 and IMG52 are located to the right of IFP1. Image IMG51 is the same as image IMG41 shown in FIG. 10. Image IMG52 is the same as image IMG42 shown in FIG. 10. Image IMG53 is a portion of image IMG3 shown in FIGS. 9 and 10. Specifically, image IMG53 corresponds to the portion of image IMG3 immediately to the right of image IFP1 of the left front pillar 4FP1.

In image IMG5 shown in Figure 11, image IMG53 exists between images IMG51 and IMG52. In the latest image shown in Figure 7, image IMG53 is superimposed where image IFP1 of the left front pillar 4FP1 exists. The blind spot area in the latest image (image IMG1, Figure 7) where the scenery LS is obscured by the left front pillar 4FP1 is complemented by image IMG53, which is part of the converted image (image IMG3, Figures 9 and 10) obtained by converting the immediately preceding image (image IMG2, Figure 8).

Figure 12 is a diagram illustrating a composite of multiple past images and the latest image. In image IMG6 shown in Figure 12, images IMG61 and IMG62 are portions of the latest image, also shown in Figure 7. Image IMG63 is a portion of a converted image obtained by converting the immediately preceding image, also shown in Figure 9. Images IMG64 and IMG65 are portions of converted images obtained by similarly converting a past image captured before the immediately preceding image based on the amount of movement of the landscape LS between the time the past image was captured and the time the latest image was captured.

Images IMG61 and IMG62 are the most recent images. Image IMG63 is a converted image obtained by converting the image captured immediately before the most recent image. Image IMG64 is a converted image obtained by converting the image captured two images before the most recent image. Image IMG65 is a converted image obtained by converting the image captured three images before the most recent image. All blind spots in the most recent image are complemented with converted images obtained by converting previously captured images. By overlaying the converted images on the blind spots in the most recent image, image IMG6 is generated, which is a composite image in which the blind spots are complemented with previous images. The image composition unit 265 combines images IMG61 and IMG62, which are part of the most recent image, with images IMG63, IMG64, and IMG65, which are previously captured images, to generate composite image IMG6.

In step S8, the image is output. The communication device 22 transmits the image IMG6 generated by the image synthesis unit 265 to the remote controller 60 via the network 400. The image output unit 67 of the remote controller 60 outputs the image IMG6 received by the communication device 72 from the vehicle body controller 26 to the display device 50, causing the display device 50 to display the image IMG6.

After processing in step S8, the process returns and the process of generating a composite image using the newer captured image and displaying it on the display device 50 is repeated.

As shown in FIG. 4, the image generation system of the embodiment described above includes a remote-controlled display device 50, a recording unit 270 that records images captured by the camera 20, and an image processing unit 260 that processes the captured images. As shown in FIGS. 5-7, the image acquisition unit 261 acquires the latest image (image IMG1), which is the most recent captured image. As shown in FIGS. 5, 8-10, the viewpoint conversion unit 264 generates a converted image (image IMG3) by converting the previous captured image (image IMG2) recorded in the recording unit 270 to match the orientation of the camera 20 when it captured the latest image. As shown in FIGS. 5, 10-12, the image output unit 67 displays, on the display device 50, a complemented image (image IMG6) obtained by complementing image IFP1, which shows the blind spot where the subject is obscured by the left front pillar 4FP1 in the latest image, with the converted image.

The scenery LS that is obscured by an obstruction in the latest image IMG1 is not obscured by an obstruction in the previously captured image and is included in the captured image. The previously captured image IMG2 is converted to generate image IMG3 according to the orientation of the camera 20 when the latest image IMG1 was captured, and the latest image IMG1 and the converted image IMG3 are superimposed. At this time, image IFP1, which is a blind spot area in the latest image IMG1, is masked so that it is not used in the superimposition of image IMG1 and image IMG3. This results in a complemented image in which the blind spot area is complemented by the previously captured image.

By displaying the complementary image on the display device 50, the operator can be presented with an image that appears as if the left front pillar 4FP1 were transparent. This allows the operator to be provided with an image in which the object obstructing the image capture target appears to be removed, thereby improving the visibility of the image capture target.

The front camera 20E captures images from a viewpoint similar to that seen when an operator is operating the hydraulic excavator 100 from inside the vehicle compartment 4. By displaying an image for remote operation on the display device 50 based on the image captured by the front camera 20E, it is possible to present a clearer, more intuitive image to the operator remotely operating the hydraulic excavator 100.

As shown in Figures 5, 11-12, the image processing unit 260 generates image IMG6, which is a composite image obtained by overlaying image IMG3, which has been converted, onto image IFP1, which is a blind spot area in the latest image, and displays image IMG6 on the display device 50. By generating a composite image by combining multiple images using the image processing unit 260 and displaying this composite image on the display device 50, it is possible to reliably provide the operator with an image in which any obstructions blocking the image capture target have been removed, thereby improving the visibility of the image capture target.

As shown in Figure 12, the image processing unit 260 generates a converted image from multiple previously captured images. Converted images are generated until the blind spot areas in the most recent image are completely filled in, and by filling in the blind spot areas with these converted images, the operator can be reliably provided with an image in which any obstructions blocking the imaged subject appear to be eliminated, improving the visibility of the imaged subject.

As shown in Figure 1, the hydraulic excavator 100 has a rotating unit 3 that rotates around a rotation axis RX. As shown in Figures 1 and 4, a camera 20 is mounted on the rotating unit 3. By using a previous image captured while the rotating unit 3 was rotating to complement blind spots in the latest image, it is possible to reliably provide the operator with an image in which any obstructions blocking the imaged subject appear to be eliminated, thereby improving the visibility of the imaged subject.

As shown in Figure 1, the rotating unit 3 has a passenger compartment 4. The front camera 20E is housed in the passenger compartment 4. As shown in Figures 6 and 7, the image captured by the front camera 20E includes an image IFP1 of the left front pillar 4FP1, which obscures the landscape LS that is the subject of the image capture. By using the image generation system of the embodiment to complement the blind spot area in the latest image, it is possible to reliably provide the operator with an image in which any obstructions that may be obscuring the subject of the image capture are seemingly eliminated, thereby improving the visibility of the subject of the image capture.

As shown in Figures 5, 8 and 9, the image processing unit 260 generates a converted image by shifting the position of the previous captured image based on the rotation angle of the rotating unit 3 from when the previous captured image was taken to when the latest image was taken. In the hydraulic excavator 100, the direction in which the rotating unit 3 rotates is specified, so the process of shifting the position of the captured image based on the rotation angle to generate a converted image is easy, and a converted image with higher accuracy can be generated.

Second Embodiment
Fig. 13 is a functional block diagram showing an example of the configuration of an image generation system according to the second embodiment. Compared to Fig. 4, the image forming system according to the second embodiment shown in Fig. 13 differs in that the image processing unit 260 has a complementary image generation unit 266 instead of the image synthesis unit 265. In the first embodiment, a synthesized image is generated by superimposing a converted image obtained by converting a previously captured image onto a blind spot area in the latest image, and the synthesized image is displayed on the display device 50, but the present invention is not limited to this example.

In the image forming system of the second embodiment, the area acquisition unit 263 acquires the blind spot area in the latest image. The viewpoint conversion unit 264 converts past images, including the immediately preceding image, to match the orientation of the camera 20 when the camera 20 captured the latest image, to generate a converted image. The complement image generation unit 266 compares the latest image with the converted image, extracts a portion of the converted image that overlaps the blind spot area, and uses this extracted portion as a complement image. The complement image generation unit 266 converts past images, which are previously captured images, until it can completely complement the blind spot area, and extracts the portion that overlaps the blind spot area as a complement image. The complement image generation unit 266 can generate multiple complement images.

The communication device 22 transmits the latest image and the complementary image to the remote controller 60 via the network 400. The image output unit 67 of the remote controller 60 displays the latest image on the display device 50, and displays the complementary image in an area of the display device 50 that corresponds to the blind spot of the latest image.

In this way, by outputting an image of a non-blind spot area and an image that complements the blind spot area to the display device 50, an image that appears to have no blind spot areas can be displayed on the display device 50, similar to the case of synthesizing the composite image described in the first embodiment. Since there are no obstructions that block the imaged subject in the image displayed on the display device 50, the visibility of the imaged subject can be improved.

The converted image does not have to be the entire previous image converted. A converted image can be generated by converting only a portion of a previous image that corresponds to a blind spot area in the latest image where the subject is obscured by an obstruction. The portion of the previous image that corresponds to the blind spot area can be converted pixel by pixel to generate multiple converted images. These multiple converted images can be used as multiple complementary images pixel by pixel. By transmitting the multiple complementary images pixel by pixel to the remote controller 60 sequentially or simultaneously, and having the image output unit 67 display these multiple complementary images on the display device 50, it is possible to reliably provide the operator with an image in which the obstruction appears to have been removed.

[Third embodiment]
14 is a functional block diagram showing an example of the configuration of an image generation system according to the third embodiment. In the first and second embodiments, the vehicle controller 26 mounted on the hydraulic excavator 100 has the image processing unit 260. In contrast, in the image generation system according to the third embodiment, the remote controller 60 on the remote operation side has the image processing unit 600.

The hydraulic excavator 100 is equipped with a swing angle sensor 27. The swing angle sensor 27 detects the swing angle of the swing unit 3 relative to the running unit 5. The swing angle sensor 27 may be, for example, a resolver provided on the rotating shaft of the swing motor, or an IMU (Inertial Measurement Unit) mounted on the swing unit 3.

The vehicle controller 26 has an image acquisition unit 261 and a rotation angle acquisition unit 267, both of which are shown in Figure 4. The rotation angle acquisition unit 267 acquires the angle at which the rotating unit 3 has rotated within a predetermined time from the detection value of the rotation angle sensor 27.

The remote control system 200 is equipped with a recording unit 610. Past image data 618 is recorded in the recording unit 610. The recording unit 610 records captured images transmitted to the remote control system 200 via the network 400 as past images.

The recording unit 610 also records camera direction data 611. The camera direction data 611 indicates the orientation of the front camera 20E when it captures an image. For example, the camera direction data 611 may indicate the direction of the optical axis of the front camera 20E in the ITRF (International Terrestrial Reference Frame) coordinate system. The camera direction data 611 is input to the remote control system 200 and recorded in the recording unit 610.

Blind spot data 612 is also recorded in the recording unit 610. The blind spot data 612 is recorded in the recording unit 610 using the following procedure. Design values for the shape of the passenger compartment 4 and the mounting position of the front camera 20E relative to the passenger compartment 4 are input into the remote control system 200 in advance. From these design values, the areas in the captured image where structures that make up the passenger compartment 4 are located are determined in advance. For example, the positions of image IFP1 of the left front pillar 4FP1 and image IFP2 of the right front pillar 4FP2 in the image IMG shown in FIG. 6 are determined. The positions of the structures that make up the passenger compartment 4 in the captured image are recognized as blind spot areas and recorded in the recording unit 610. Because the positions of the left front pillar 4FP1 and right front pillar 4FP2 do not change relative to the front camera 20E, determining the blind spot areas is easy.

Figure 15 is a flowchart showing an example of an image generation method according to the third embodiment. The image generation method according to the third embodiment will be described with reference to Figures 14 and 15. As with the first embodiment, an example will be described in which an image of an object is captured while the revolving unit 3 is revolving.

As shown in FIG. 15, first, in step S11, the latest image is acquired. The image acquisition unit 261 acquires the latest image, which is the most recent image captured by the camera 20 of the hydraulic excavator 100.

In step S12, the rotation angle is acquired. The rotation angle acquisition unit 267 acquires the rotation angle of the rotating unit 3 from the detection value of the rotation angle sensor 27. The rotation angle acquisition unit 267 acquires the rotation angle of the rotating unit 3 from the time when the image immediately before the latest image was captured to the time when the latest image was captured.

The communication device 22 transmits the captured image captured by the camera 20 and acquired by the image acquisition unit 261, and the rotation angle of the rotating body 3 acquired by the rotation angle acquisition unit 267, to the remote control system 200 via the network 400.

In step S13, a previous image, which is an image captured before the latest image, is read. The image processing unit 600 has a viewpoint conversion unit 604. The viewpoint conversion unit 604 reads the previous image from the recording unit 610. Typically, the viewpoint conversion unit 604 reads the previous image, which is an image captured immediately before the latest image.

In step S14, the camera direction is acquired. The image processing unit 600 has a direction acquisition unit 601. The direction acquisition unit 601 reads camera direction data 611 indicating the orientation of the camera 20 when the camera 20 captured the previous image from the recording unit 610. The direction acquisition unit 601 reads data indicating the orientation of the front camera 20E from the recording unit 610.

In step S15, a converted image is generated. The viewpoint conversion unit 604 converts the previous image, including the immediately preceding image, based on the rotation angle of the rotating unit 3 and the orientation of the front camera 20E, to match the orientation of the front camera 20E when the latest image was captured by the front camera 20E. The viewpoint conversion unit 604 shifts the previous image by the rotation angle of the rotating unit 3 from the time the previous image was captured to the time the latest image was captured, so that the scenery included in the previous image exactly overlaps with the scenery included in the latest image.

In step S16, the blind spot area is acquired. The image processing unit 600 has an area acquisition unit 603. The area acquisition unit 603 reads blind spot area data 612 from the recording unit 610. The area acquisition unit 603 acquires the position of the blind spot area in the image captured by the front camera 20E, typically the image IFP1 of the left front pillar 4FP1 and the image IFP2 of the right front pillar 4FP2.

In step S17, masking is performed. In step S18, the images are combined. The image processing unit 600 has an image combination unit 265. The image combination unit 605 masks blind spots in the latest image so that the blind spots are not used in combining the images. The image combination unit 605 combines the images by superimposing a portion of the latest image and a portion of the converted image. If necessary, the image combination unit 605 combines the latest image with multiple converted images obtained by converting multiple past images to generate a combined image.

In step S19, the image is output. The image output unit 67 outputs the composite image generated in step S18 to the display device 50, causing the display device 50 to display the composite image.

After processing in step S19, the process returns and the process of generating a composite image using the newer captured image and displaying it on the display device 50 is repeated.

As with the first embodiment, the image generation system of the third embodiment can display an image on the display device 50 that appears to have no blind spots. Since the image displayed on the display device 50 does not contain any obstructions that block the imaged subject, the visibility of the imaged subject can be improved.

The image generation system that generates the images to be displayed on the display device 50 may be installed in the vehicle controller 26 on the work machine side, in the remote controller 60 on the remote control side, or in another computer such as a control computer.

[Fourth embodiment]
16 is a schematic diagram showing a situation in which images are captured by a camera according to the fourth embodiment. In the fourth embodiment, an example will be described in which scenes LSA, LSB, and LSC, which are arranged side by side on the same plane, are captured in order by a camera 20 rotating in a rotation direction RD.

Figure 17 is a diagram showing an example of a captured image according to the fourth embodiment. Image IMG101 includes the landscapes LSA and LSB. Image IMG101 is a captured image of the landscapes LSA and LSB captured from an oblique angle. Image IMG102 includes the landscape LSB and portions of the landscapes LSA and LSC. Image IMG102 is a captured image of the landscape LSB captured from the front. Image IMG103 includes the landscapes LSB and LSC. Image IMG103 is a captured image of the landscapes LSB and LSC captured from an oblique angle.

Figure 18 is a diagram showing an example of a composite image according to the fourth embodiment. As shown in Figure 16, the camera 20 is rotated in place to successively capture the multiple images IMG101, IMG102, and IMG103 shown in Figure 17, and the common portions of images IMG101, IMG102, and IMG103 are joined together so that they overlap, allowing them to be composited into a single image. Image IMG111 shown in Figure 18 is a modified version of image IMG101 shown in Figure 17. Image IMG112 is a modified version of image IMG102.

When generating a composite image based on the most recently captured image IMG103, a more precise composite image can be generated by shifting and deforming the positions of the previously captured images IMG101 and IMG102 based on the rotation angle of the camera 20. This further improves the visibility of the imaged subject.

In the above embodiment, an example has been described in which the image IFP1 of the left front pillar 4FP1 is seemingly removed from the image displayed on the display device 50. Structures that make up the cabin 4, such as pillars and side mirrors, are not limited to those that can be removed from the image. Part or all of the work implement 2 may also be removed from the image. The boom cylinder 10, arm cylinder 11, and bucket cylinder 12 may be removed, and only the boom 6, arm 7, and bucket 8 that make up the main body of the work implement 2 may be displayed on the display device 50. Only the bucket 8, which is a work tool for excavation and loading located at the tip of the work implement 2, may be displayed on the display device 50, or only the cutting edge of the bucket 8 may be displayed on the display device 50.

In the embodiment, an example has been described in which an image of an object is captured while the rotating unit 3 is rotating, but the present invention is not limited to image processing of images captured while the rotating unit 3 is rotating. When the rotating unit 3 is not rotating relative to the running unit 5 and the work implement 2 is operating relative to the rotating unit 3, it is also possible to prevent the work implement 2 from being displayed on the display device 50. In this case, based on multiple captured images, the time at which the multiple captured images were captured, and the movement speed of the work implement 2, it is possible to remove some or all of the work implement 2 from the image displayed on the display device 50. The movement speed of the work implement 2 can be obtained from the detection values of stroke sensors attached to each hydraulic cylinder that drives the work implement 2, the detection values of rotation angle sensors attached to each pin that serves as the rotation axis of each element of the work implement 2, image analysis of the captured image of the work implement 2, etc.

In the embodiment, a hydraulic excavator 100 is given as an example of a work machine, but the invention can also be applied to other types of work machines, such as loading shovels, mechanical rope shovels, and bucket cranes.

In the embodiment, an example has been described in which the hydraulic excavator 100 is equipped with a swing motor, which is a hydraulic actuator 34, and the swing body 3 swings when hydraulic oil is supplied to the swing motor. This example is not limiting, and the excavator may also be equipped with an electric motor. The excavator may be a hybrid excavator equipped with an engine that serves as a drive source for the operation and travel of the work implement 2, and an electric motor that drives the swing body 3 with electric energy. The excavator may also be an electric excavator in which the drive sources for the rotation of the swing body 3, the travel of the traveling body 5, and the operation of the work implement 2 are all electric motors, and the electric motors are driven by electric energy stored in a battery.

The embodiments have been described above, but configurations that can be combined with each other in each embodiment may be combined as appropriate. Furthermore, the embodiments disclosed herein are illustrative in all respects and should not be considered limiting. The scope of the present invention is indicated by the claims, not the above description, and is intended to include all modifications within the meaning and scope of the claims.

1 Main body, 2 Work machine, 3 Swivel body, 4 Vehicle compartment, 4FP1 Left front pillar, 4FP2 Right front pillar, 4S Space, 6 Boom, 7 Arm, 8 Bucket, 10 Boom cylinder, 11 Arm cylinder, 12 Bucket cylinder, 20 Camera, 20A Right front camera, 20B Right side camera, 20C Rear camera, 20D Left side camera, 20E Front camera, 22, 72 Communication device, 26 Vehicle body controller, 27 Swivel angle sensor, 40 Remote control device, 45 Operator seat, 50 Display device, 60 Remote controller, 67 Image output unit, 69 Control signal generation unit, 100 Hydraulic excavator, 200 Remote control system, 260, 600 Image processing unit, 261 Image acquisition unit, 262 Movement information acquisition unit, 263, 603 Area acquisition unit, 264, 604 Viewpoint conversion unit, 265, 605 image synthesis unit, 266 complementary image generation unit, 267 rotation angle acquisition unit, 270, 610 recording unit, 278, 618 past image data, 400 network, 601 direction acquisition unit, 611 camera direction data, 612 blind spot area data, RD rotation direction, RX rotation axis.

Claims (11)

An image generation system for generating an image to be displayed on a remote control display device for remotely operating a work machine,
The work machine includes a rotating body that rotates around a rotation axis, and an imaging device that is mounted on the rotating body and captures an image of an object that is present at a work site,
The image generation system includes:
the display device;
an image recording unit that records a captured image captured by the imaging device and information indicating the orientation of the imaging device when the captured image was captured ;
an image processing computer that processes the captured image,
the image processing computer acquires a latest image, which is the most recent captured image, and generates a converted image by converting the previous captured image recorded in the image recording unit to match the orientation of the imaging device when the imaging device captured the latest image, based on information indicating the orientation of the imaging device when the latest image was captured and information indicating the orientation of the imaging device when the previous captured image was captured, and displays on the display device a complemented image in which a blind spot area in the latest image, where the imaging target is blocked by an obstruction, is complemented with the converted image. The image generation system of claim 1, wherein the image processing computer generates a composite image by superimposing the converted image on the blind spot area, and displays the composite image on the display device. The image generating system according to claim 1 , wherein the image processing computer causes the display device to display the latest image and the converted image in an area of the display device that corresponds to the blind spot area. An image generation system according to any one of claims 1 to 3, wherein the image processing computer generates the converted image from a plurality of previous captured images. The image generation system according to claim 1 , wherein the rotating body has a cabin, and the imaging device is housed in the cabin. the imaging device captures an image of the imaging target while the rotating body is rotating,
6. The image generation system according to claim 5, wherein the image processing computer acquires a rotation angle of the rotating body from when the previous captured image was taken to when the latest image was taken, and generates the converted image by shifting the position of the previous captured image based on the rotation angle . The image generation system according to claim 6 , wherein the image processing computer generates the transformed image by transforming the previously captured image based on the rotation angle. An image generation system for generating an image to be displayed on a remote control display device for remotely operating a work machine,
the work machine has an imaging device that captures an image of an object to be captured that is present at a work site,
The image generation system includes:
the display device;
an image recording unit that records an image captured by the imaging device;
an image processing computer that processes the captured image,
the image processing computer acquires a latest image that is the latest captured image, generates a converted image by converting the previous captured image recorded in the image recording unit in accordance with the orientation of the imaging device when the imaging device captured the latest image, and displays on the display device a complemented image in which a blind spot area in the latest image where the imaging target is blocked by an obstruction is complemented with the converted image;
The work machine has a rotating body that rotates around a rotating shaft,
the imaging device is mounted on the rotating body and captures an image of the imaging target while the rotating body is rotating;
The image processing computer acquires the rotation angle of the rotating body from when the previous captured image was taken to when the latest image was taken, and generates the converted image by shifting and deforming the position of the previous captured image based on the rotation angle. An image generation system for generating an image to be displayed on a remote control display device for remotely operating a work machine,
The work machine has a rotating body that rotates around a rotation axis, and an imaging device that is mounted on the rotating body and captures an image of an object to be imaged that is present at a work site, the orientation of the imaging device changing as the rotating body rotates,
The image generation system includes:
the display device;
an image recording unit that records an image captured by the imaging device;
an image processing computer that processes the captured image,
The image processing computer acquires a latest image, which is the most recent captured image, generates a converted image by converting the previous captured image recorded in the image recording unit to match the orientation of the imaging device when the imaging device captured the latest image, compares the latest image with the previous captured image, determines that areas with little change are blind spot areas in the latest image where the imaging subject is blocked by an obstruction, and displays a complemented image in which the blind spot area is complemented with the converted image on the display device. An image generation system for generating an image to be displayed on a remote control display device for remotely operating a work machine,
The work machine has a rotating body that rotates around a rotation axis, and an imaging device that is mounted on the rotating body and captures an image of an object to be imaged that is present at a work site, the orientation of the imaging device changing as the rotating body rotates,
The image generation system includes:
the display device;
an image recording unit that records an image captured by the imaging device;
an image processing computer that processes the captured image,
the image processing computer acquires a latest image, which is the most recent captured image, generates a converted image by converting the previous captured image recorded in the image recording unit to match the orientation of the imaging device when the imaging device captured the latest image, compares the latest image with the converted image, and displays on the display device a complementary image obtained by extracting a portion of the converted image that overlaps a blind spot area in the latest image where the imaging target is blocked by an obstruction. The image generating system of claim 10 , wherein the image processing computer extracts the complementary images until the blind spot area is completely complemented.