JP6852026B2 - Excavator - Google Patents

Description

The present invention relates to excavators .

Conventionally, there is known a peripheral monitoring device that emits an alarm sound when an operator is detected within the monitoring range of an obstacle detector mounted on an excavator (see, for example, Patent Document 1). In addition, whether or not the worker who has entered the work area set around the excavator is a collaborative worker is judged from the light emission pattern of the LED attached to the worker's helmet and whether or not an alarm sound is output. An alarm system is known to determine whether or not (see, for example, Patent Document 2). Further, there is known a safety device that communicates between a forklift and an operator who works in the vicinity (around) of the forklift and controls the output of an alarm sound based on this communication (see, for example, Patent Document 3). ).

Japanese Unexamined Patent Publication No. 2008-179940 JP-A-2009-193494 JP-A-2007-310587

An object of the present invention is to provide an excavator capable of detecting an object existing in an area that becomes a blind spot of an operator.

The excavator according to the embodiment of the present invention is an excavator having a lower traveling body, an upper rotating body rotatably arranged on the lower traveling body, and a peripheral monitoring device for monitoring the surroundings. in a state that can manipulate, have a function of detecting an object before the operation is started, and outputs an alarm when it detects an object, the alarm outputted until an operation for stopping is performed by detecting an object The excavator is maintained in an inoperable state regardless of whether or not it is used.

By the above-mentioned means, it is possible to provide an excavator capable of detecting an object existing in an area that becomes a blind spot of an operator.

It is a block diagram which shows schematic structure example of the image generation apparatus which concerns on embodiment of this invention. It is a figure which shows the configuration example of the excavator which mounts an image generator. It is a figure which shows an example of the spatial model which the input image is projected. It is a figure (the 1) which shows an example of the relationship between a spatial model and an image plane to be processed. It is a figure for demonstrating the correspondence between the coordinates on the input image plane, and the coordinates on a space model. It is a figure for demonstrating the correspondence between coordinates by the coordinate correspondence means. It is a figure for demonstrating the action of a group of parallel lines. It is a figure for demonstrating the action of the auxiliary line group. It is a flowchart which shows the flow of the processing target image generation processing and output image generation processing. This is a display example of the output image (No. 1). It is a top view of the excavator on which the image generator is mounted. It is a figure (the 1) which shows the input image of each of three cameras mounted on a shovel, and the output image generated by using the input image. It is a figure for demonstrating the image disappearance prevention processing which prevents the disappearance of an object in the overlapping part of the imaging space of each of two cameras. It is a comparison diagram which shows the difference between the output image shown in FIG. 12 and the output image obtained by applying the image disappearance prevention processing to the output image of FIG. It is a figure (No. 2) which shows the input image of each of three cameras mounted on a shovel, and the output image generated by using the input image. It is a figure (the 2) which shows an example of the relationship between a spatial model and an image plane to be processed. It is a figure explaining the relationship of two output images which are switched by the 1st output image switching process. It is a figure explaining the relationship of three output images which are switched by the 2nd output image switching process. It is a correspondence table which shows the correspondence relationship between the determination result of the presence / absence determination means, and the content of an output image. It is a flowchart which shows the flow of the 1st alarm control processing. It is a figure which shows an example of the transition of the output image displayed in the 1st alarm control processing. It is a flowchart which shows the flow of the 2nd alarm control processing. It is a figure which shows an example of the transition of the output image displayed in the 2nd alarm control processing. It is a flowchart which shows the flow of the work machine state determination process. It is a flowchart which shows the flow of the alarm control processing at the start of work. It is a block diagram which shows the other configuration example of the peripheral monitoring apparatus schematicly. It is a block diagram which shows the other structural example of the peripheral monitoring apparatus schematicly.

Hereinafter, the best mode for carrying out the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram schematically showing a configuration example of an image generation device 100 according to an embodiment of the present invention.

The image generation device 100 is an example of a peripheral monitoring device for a work machine that monitors the periphery of the work machine, and is a control unit 1, a camera 2, an input unit 3, a storage unit 4, a display unit 5, a person detection sensor 6, and an alarm. It is composed of an output unit 7, a gate lock lever 8, and an ignition switch 9. Specifically, the image generation device 100 generates an output image based on the input image captured by the camera 2 mounted on the work machine, and presents the output image to the operator. Further, the image generation device 100 switches the content of the output image to be presented based on the output of the human detection sensor 6.

FIG. 2 is a diagram showing a configuration example of an excavator 60 as a work machine on which an image generator 100 is mounted. The excavator 60 is placed on a crawler-type lower traveling body 61 via a swivel mechanism 62. The swivel body 63 is mounted so as to be swivelable around the swivel shaft PV.

Further, the upper swivel body 63 is provided with a cab (driver's cab) 64 on the front left side thereof, an excavation attachment E on the front center portion thereof, and a camera 2 (right side camera 2R, rear camera 2B) on the right side surface and the rear surface thereof. ) And a person detection sensor 6 (right side person detection sensor 6R, rear person detection sensor 6B). The display unit 5 is installed at a position in the cab 64 that is easily visible to the operator. Further, in the cab 64, an alarm output unit 7 (right side alarm output unit 7R, rear alarm output unit 7B), a gate lock lever 8, and an ignition switch 9 are installed.

Next, each component of the image generation device 100 will be described.

The control unit 1 is a computer including a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), an NVRAM (Non-Volatile Random Access Memory), and the like. In this embodiment, the control unit 1 provides, for example, a program corresponding to each of the coordinate mapping means 10, the image generation means 11, the presence / absence determination means 12, the alarm control means 13, and the work machine state determination means 14, which will be described later. It is stored in a ROM or NVRAM, and the CPU is made to execute a process corresponding to each means while using the RAM as a temporary storage area.

The camera 2 is a device for acquiring an input image projecting the surroundings of the excavator 60. In this embodiment, the camera 2 is, for example, a right side camera 2R and a rear camera 2B attached to the right side surface and the rear surface of the upper swivel body 63 so as to be able to image an area that is a blind spot of the operator in the cab 64 (FIG. See 2.). Further, the camera 2 includes an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 2 may be attached to a position other than the right side surface and the rear surface (for example, the front surface and the left side surface) of the upper swing body 63, and a wide-angle lens or a fisheye lens is attached so as to be able to image a wide range. You may be.

Further, the camera 2 acquires an input image according to the control signal from the control unit 1, and outputs the acquired input image to the control unit 1. When the camera 2 acquires an input image using a fisheye lens or a wide-angle lens, the camera 2 sends the corrected input image corrected to the apparent distortion and tilt caused by using those lenses to the control unit 1. Output. Further, the camera 2 may output the input image without correcting the apparent distortion and tilt to the control unit 1 as it is. In that case, the control unit 1 corrects the apparent distortion and tilt.

The input unit 3 is a device for allowing the operator to input various information to the image generation device 100, and is, for example, a touch panel, a button switch, a pointing device, a keyboard, or the like.

The storage unit 4 is a device for storing various types of information, and is, for example, a hard disk, an optical disk, a semiconductor memory, or the like.

The display unit 5 is a device for displaying image information, for example, a liquid crystal display or a projector installed in the cab 64 (see FIG. 2) of the excavator 60, and various types output by the control unit 1. Display the image.

The human detection sensor 6 is a device for detecting a person existing around the excavator 60. In this embodiment, the human detection sensor 6 is attached to, for example, the right side surface and the rear surface of the upper swing body 63 so as to be able to detect a person existing in the area of the operator's blind spot in the cab 64 (see FIG. 2). ..

The human detection sensor 6 is a sensor that distinguishes and detects a person from an object other than the human, and is, for example, a sensor that detects an energy change in the corresponding monitoring space, and is a charcoal infrared sensor, a bolometer infrared sensor, or the like. Includes a motion detection sensor that uses an output signal from an infrared camera or the like. In this embodiment, the human detection sensor 6 uses a pyroelectric infrared sensor, and detects a moving object (moving heat source) as a human. Further, the monitoring space of the right-side person detection sensor 6R is included in the imaging space of the right-side camera, and the monitoring space of the rear person detection sensor 6B is included in the imaging space of the rear camera 2B.

Like the camera 2, the person detection sensor 6 may be attached to a position other than the right side surface and the rear surface of the upper swivel body 63 (for example, the front surface and the left side surface), and the front surface of the upper swivel body 63. It may be attached to any one of the left side surface, the right side surface, and the rear surface, or may be attached to all surfaces.

The alarm output unit 7 is a device that outputs an alarm to the operator of the excavator 60. For example, the alarm output unit 7 is an alarm device that outputs at least one of sound and light, and includes an audio output device such as a buzzer and a speaker, and a light emitting device such as an LED and a flashlight. In this embodiment, the alarm output unit 7 is a buzzer that outputs an alarm sound, and is a right side alarm output unit 7R attached to the right inner wall of the cab 64 and a rear alarm output unit attached to the rear inner wall of the cab 64. It is composed of 7B (see FIG. 2).

The gate lock lever 8 is a device for switching the state of the excavator 60. In this embodiment, the gate lock lever 8 has a locked state in which the excavator 60 is in a non-workable state and an unlocked state in which the excavator 60 is in a workable state. The "workable state" means a state in which the operator can operate the excavator 60, and the "workable state" means a state in which the operator cannot operate the excavator 60. The gate lock lever 8 repeatedly outputs a signal representing its current state to the control unit 1 at a predetermined cycle.

Specifically, the operator pulls up the gate lock lever 8 to make it substantially horizontal to unlock the gate lock lever 8, and pushes down the gate lock lever 8 to lock the gate lock lever 8.

In the unlocked state, the gate lock lever 8 blocks the entrance / exit of the cab 64 to prevent the operator from exiting the cab 64, while the operation levers and operation pedals (hereinafter, "operations") in the cab 64 (not shown) are used. The operation by "pedal etc.") is enabled so that the operator can operate the excavator 60.

On the other hand, in the locked state, the gate lock lever 8 opens the entrance / exit of the cab 64 to allow the operator to exit from the cab 64, while disabling the operation by the operation lever or the like in the cab 64 and enabling the operator. Prevents the excavator 60 from being operated.

More specifically, the gate lock lever 8 closes the gate lock valve in the locked state and opens the gate lock valve in the unlocked state. The gate lock valve is a switching valve provided between the oil passage between the control valve (not shown) and the operating lever or the like. The control valve is a flow rate control valve that controls the flow of hydraulic oil between a hydraulic pump (not shown) and various hydraulic actuators. In the closed state, the gate lock valve blocks the flow of hydraulic oil between the control valve and the operating lever, etc., and invalidates the operating lever, etc. Further, in the gate lock valve, in the open state, the hydraulic oil is communicated between the control valve and the operation lever or the like to enable the operation lever or the like.

The ignition switch 9 is a device for switching the state of the excavator 60. In this embodiment, the ignition switch 9 has an off state in which the excavator 60 is in a non-workable state and an on state in which the excavator 60 is in a workable state. The ignition switch 9 repeatedly outputs a signal representing its current state to the control unit 1 at a predetermined cycle.

Specifically, the operator switches the ignition switch 9 between the on state and the off state by pressing the ignition switch 9.

In the on state, the ignition switch 9 starts the control pump that supplies hydraulic oil to the oil passage between the control valve and the operation lever by starting the engine. Then, the ignition switch 9 enables the operator to operate the excavator 60 by enabling the operation by the operation lever or the like in the cab 64.

On the other hand, in the off state, the ignition switch 9 stops the control pump by stopping the engine. Then, the ignition switch 9 invalidates the operation by the operation lever or the like in the cab 64 and prevents the operator from operating the excavator 60.

Further, the image generation device 100 generates a processing target image based on the input image, and performs image conversion processing on the processing target image so that the positional relationship with the surrounding objects and the sense of distance can be intuitively grasped. After generating the output image to be used, the output image may be presented to the operator.

The “processed image” is an image that is generated based on the input image and is the target of image conversion processing (for example, scale conversion processing, affine transformation processing, distortion conversion processing, viewpoint conversion processing, and the like). Specifically, the "processed image" is, for example, an input image taken by a camera that captures the ground surface from above, and includes an image in the horizontal direction (for example, an empty portion) due to its wide angle of view. It is an image suitable for image conversion processing generated from. More specifically, the input image is projected onto a predetermined spatial model so that the horizontal image is not displayed unnaturally (for example, the empty part is not treated as if it is on the ground surface), and then the image is projected. It is generated by reprojecting a projected image projected onto a spatial model onto another two-dimensional plane. The image to be processed may be used as it is as an output image without performing image conversion processing.

The "spatial model" is the projection target of the input image. Specifically, the "spatial model" is composed of at least one or a plurality of planes or curved surfaces including a plane or a curved surface other than the processing target image plane which is the plane on which the processing target image is located. The plane or curved surface other than the processing target image plane, which is the plane on which the processing target image is located, is, for example, a plane parallel to the processing target image plane or a plane or curved surface forming an angle with the processing target image plane. ..

The image generation device 100 may generate an output image by performing an image conversion process on the projected image projected on the spatial model without generating the image to be processed. Further, the projected image may be used as it is as an output image without performing image conversion processing.

FIG. 3 is a diagram showing an example of the spatial model MD on which the input image is projected, and FIG. 3 left is a diagram showing the relationship between the excavator 60 and the spatial model MD when the excavator 60 is viewed from the side. The right figure of FIG. 3 shows the relationship between the excavator 60 and the spatial model MD when the excavator 60 is viewed from above.

As shown in FIG. 3, the spatial model MD has a semi-cylindrical shape, and has a plane region R1 inside the bottom surface thereof and a curved surface region R2 inside the side surface thereof.

Further, FIG. 4 is a diagram showing an example of the relationship between the spatial model MD and the processing target image plane, and the processing target image plane R3 is, for example, a plane including the plane region R1 of the spatial model MD. In FIG. 4, for clarification, the space model MD is shown in a cylindrical shape instead of the semi-cylindrical shape as shown in FIG. 3, but the space model MD is either a semi-cylindrical shape or a cylindrical shape. It may be. The same applies to the following figures. Further, as described above, the image plane R3 to be processed may be a circular region including the plane region R1 of the space model MD, or may be an annular region not including the plane region R1 of the space model MD.

Next, various means included in the control unit 1 will be described.

The coordinate mapping means 10 is a means for associating the coordinates on the input image plane on which the input image captured by the camera 2 is located, the coordinates on the spatial model MD, and the coordinates on the processing target image plane R3. In this embodiment, the coordinate mapping means 10 includes, for example, various parameters related to the camera 2 preset or input via the input unit 3, and a predetermined input image plane, spatial model MD, and the like. And, based on the mutual positional relationship of the processing target image plane R3, the coordinates on the input image plane, the coordinates on the spatial model MD, and the coordinates on the processing target image plane R3 are associated with each other. The various parameters related to the camera 2 are, for example, the optical center of the camera 2, the focal length, the CCD size, the optical axis direction vector, the camera horizontal direction vector, the projection method, and the like. Then, the coordinate mapping means 10 stores the correspondence between them in the input image / spatial model correspondence map 40 and the spatial model / processing target image correspondence map 41 of the storage unit 4.

When the coordinate mapping means 10 does not generate the processing target image, the coordinate mapping between the coordinates on the spatial model MD and the coordinates on the processing target image plane R3, and the spatial model / processing target of the corresponding relationship. The storage in the image correspondence map 41 is omitted.

The image generation means 11 is a means for generating an output image. In this embodiment, the image generation means 11 performs scale conversion, affine transformation, or distortion transformation on the image to be processed, so that the coordinates on the image plane R3 to be processed and the output image are located on the output image plane. Associate with coordinates. Then, the image generation means 11 stores the correspondence relationship in the processing target image / output image correspondence map 42 of the storage unit 4. Then, the image generation means 11 associates the value of each pixel in the output image with the value of each pixel in the input image while referring to the input image / spatial model correspondence map 40 and the spatial model / processing target image correspondence map 41. Generate an output image. The value of each pixel is, for example, a luminance value, a hue value, a saturation value, or the like.

Further, the image generation means 11 is an output image plane in which the coordinates on the processing target image plane R3 and the output image are located based on various parameters related to the virtual camera that are set in advance or input via the input unit 3. Associate with the above coordinates. The various parameters related to the virtual camera are, for example, the optical center of the virtual camera, the focal length, the CCD size, the optical axis direction vector, the camera horizontal direction vector, the projection method, and the like. Then, the image generation means 11 stores the correspondence relationship in the processing target image / output image correspondence map 42 of the storage unit 4. Then, the image generation means 11 associates the value of each pixel in the output image with the value of each pixel in the input image while referring to the input image / spatial model correspondence map 40 and the spatial model / processing target image correspondence map 41. Generate an output image.

The image generation means 11 may generate an output image by changing the scale of the image to be processed without using the concept of a virtual camera.

Further, when the image generation means 11 does not generate the image to be processed, the image generation means 11 associates the coordinates on the spatial model MD with the coordinates on the output image plane according to the image conversion process performed. Then, the image generation means 11 generates an output image by associating the value of each pixel in the output image with the value of each pixel in the input image while referring to the input image / spatial model correspondence map 40. In this case, the image generation means 11 omits associating the coordinates on the processing target image plane R3 with the coordinates on the output image plane and storing the correspondence in the processing target image / output image correspondence map 42. ..

Further, the image generation means 11 switches the content of the output image based on the determination result of the presence / absence determination means 12. The details of switching the output image by the image generation means 11 will be described later.

The presence / absence determination means 12 is a means for determining the presence / absence of a person in each of the plurality of monitoring spaces set around the work machine. In this embodiment, the presence / absence determination means 12 determines the presence / absence of people around the excavator 60 based on the output of the person detection sensor 6.

Further, the presence / absence determination means 12 may determine the presence / absence of a person in each of the plurality of monitoring spaces set around the work machine based on the input image captured by the camera 2. Specifically, the presence / absence determination means 12 may determine the presence / absence of people around the work machine by using image processing techniques such as optical flow and pattern matching. The presence / absence determination means 12 may determine the presence / absence of people around the work machine based on the output of an image sensor different from the camera 2.

Alternatively, the human presence / absence determining means 12 determines the presence / absence of a person in each of the plurality of monitoring spaces set around the work machine based on the output of the person detection sensor 6 and the output of the image sensor such as the camera 2. May be good.

The alarm control means 13 is a means for controlling the alarm output unit 7. In this embodiment, the alarm control means 13 outputs an alarm based on the determination result of the presence / absence determination means 12, or based on the determination result of the presence / absence determination means 12 and the determination result of the work machine state determination means 14. The unit 7 is controlled. The details of the control of the alarm output unit 7 by the alarm control means 13 will be described later.

The work machine state determination means 14 is a means for determining the state of the work machine. In this embodiment, the work machine state determination means 14 determines whether or not the excavator 60 is in a workable state. The details of the determination of the state of the excavator 60 by the work machine state determination means 14 will be described later.

Next, an example of specific processing by the coordinate mapping means 10 and the image generation means 11 will be described.

The coordinate mapping means 10 can map the coordinates on the input image plane and the coordinates on the spatial model by using, for example, Hamilton's quaternion.

FIG. 5 is a diagram for explaining the correspondence between the coordinates on the input image plane and the coordinates on the spatial model. The input image plane of the camera 2 is represented as one plane in the UVW Cartesian coordinate system with the optical center C of the camera 2 as the origin. The spatial model is represented as a three-dimensional plane in the XYZ Cartesian coordinate system.

First, the coordinate mapping means 10 translates the origin of the XYZ coordinate system to the optical center C (origin of the UVW coordinate system), and then shifts the X-axis to the U-axis, the Y-axis to the V-axis, and the Z-axis. Rotate the XYZ coordinate system so that each coincides with the -W axis. This is to convert the coordinates on the spatial model (coordinates on the XYZ coordinate system) into the coordinates on the input image plane (coordinates on the UVW coordinate system). The symbol "-" of "-W axis" means that the direction of the Z axis and the direction of the -W axis are opposite to each other. This is because the UVW coordinate system has the front of the camera in the + W direction, and the XYZ coordinate system has the vertical lower direction in the −Z direction.

When there are a plurality of cameras 2, each of the cameras 2 has an individual UVW coordinate system. Therefore, the coordinate mapping means 10 translates the XYZ coordinate system with respect to each of the plurality of UVW coordinate systems. Rotate.

In the above conversion, the XYZ coordinate system is translated so that the optical center C of the camera 2 becomes the origin of the XYZ coordinate system, and then the Z axis is rotated so as to coincide with the −W axis, and the X axis is U. This is achieved by rotating it so that it matches the axis. Therefore, the coordinate mapping means 10 can combine these two rotations into one rotation operation by describing this conversion in Hamilton's quaternion.

By the way, the rotation for matching one vector A with another vector B corresponds to the process of rotating by the angle formed by the vector A and the vector B about the normal of the surface stretched by the vector A and the vector B. .. Then, assuming that the angle is θ, the angle θ is calculated from the inner product of the vector A and the vector B.

で表される。

It is represented by.

Further, the unit vector N of the normal of the surface extending between the vector A and the vector B is obtained from the outer product of the vector A and the vector B.

で表されることとなる。

Will be represented by.

The quaternion is when i, j, and k are imaginary units, respectively.

を満たす超複素数であり、本実施例において、四元数Ｑは、実成分をｔ、純虚成分をａ、ｂ、ｃとして、

It is a hypercomplex number that satisfies, and in this embodiment, the quaternion Q has a real component as t and a pure imaginary component as a, b, and c.

で表され、四元数Ｑの共役四元数は、

The conjugate quaternion of the quaternion Q is represented by

で表される。

It is represented by.

The quaternion Q can express a three-dimensional vector (a, b, c) with pure imaginary components a, b, and c while setting the real component t to 0 (zero), and t, a, b. It is also possible to express a rotation motion about an arbitrary vector by each component of, c.

Further, the quaternion Q can be expressed as one rotation operation by integrating a plurality of consecutive rotation operations. Specifically, the quaternion Q is, for example, when an arbitrary point S (sx, sy, sz) is rotated by an angle θ with an arbitrary unit vector C (l, m, n) as an axis. The point D (ex, ey, ez) can be expressed as follows.

ここで、本実施例において、Ｚ軸を−Ｗ軸に一致させる回転を表す四元数をＱｚとすると、ＸＹＺ座標系におけるＸ軸上の点Ｘは、点Ｘ'に移動させられるので、点Ｘ'は、

Here, in this embodiment, assuming that the quaternion representing the rotation that makes the Z axis coincide with the −W axis is Qz, the point X on the X axis in the XYZ coordinate system is moved to the point X', so that the point X'is

で表される。

It is represented by.

Further, in this embodiment, assuming that the quaternion representing the rotation that makes the line connecting the point X'on the X axis and the origin coincide with the U axis is Qx, "the Z axis matches the -W axis, and further. , The quaternion R representing "rotation that makes the X-axis coincide with the U-axis"

で表される。

It is represented by.

From the above, the coordinates P'when the arbitrary coordinates P on the spatial model (XYZ coordinate system) are expressed by the coordinates on the input image plane (UVW coordinate system) are

で表される。また、四元数Ｒがカメラ２のそれぞれで不変であることから、座標対応付け手段１０は、以後、この演算を実行するだけで空間モデル（ＸＹＺ座標系）上の座標を入力画像平面（ＵＶＷ座標系）上の座標に変換することができる。

It is represented by. Further, since the quaternion R is invariant in each of the cameras 2, the coordinate mapping means 10 thereafter simply executes this calculation to input the coordinates on the spatial model (XYZ coordinate system) in the image plane (UVW). Coordinate system) It can be converted to the above coordinates.

After converting the coordinates on the spatial model (XYZ coordinate system) into the coordinates on the input image plane (UVW coordinate system), the coordinate mapping means 10 is formed by the line segment CP'and the optical axis G of the camera 2. Calculate the incident angle α. The line segment CP'is a line segment connecting the optical center C (coordinates on the UVW coordinate system) of the camera 2 and the coordinate P'representing an arbitrary coordinate P on the spatial model in the UVW coordinate system.

Further, the coordinate mapping means 10 calculates the declination φ and the length of the line segment EP'in the plane H parallel to the input image plane R4 (for example, the CCD plane) of the camera 2 and including the coordinates P'. The line segment EP'is a line segment connecting the intersection E of the plane H and the optical axis G and the coordinates P', and the deviation angle φ is formed by the U'axis and the line segment EP'on the plane H. The angle to do.

In the optical system of a camera, the image height h is usually a function of the incident angle α and the focal length f. Therefore, the coordinate mapping means 10 has a normal projection (h = ftanα), a normal projection (h = fsinα), a stereographic projection (h = 2ftan (α / 2)), and an equal stereographic projection (h = 2fsin (α / 2)). )), Equal distance projection (h = fα) and other appropriate projection methods are selected to calculate the image height h.

After that, the coordinate mapping means 10 decomposes the calculated image height h into U and V components on the UV coordinate system by the declination φ, and uses a numerical value corresponding to the pixel size per pixel of the input image plane R4. Divide. As a result, the coordinate mapping means 10 can associate the coordinates P (P') on the spatial model MD with the coordinates on the input image plane R4.

Assuming that the pixel size per pixel in the U-axis direction of the input image plane R4 is a _U and the pixel size per pixel in the V-axis direction of the input image plane R4 is a _V , the coordinates P on the spatial model MD The coordinates (u, v) on the input image plane R4 corresponding to (P') are

で表される。

It is represented by.

In this way, the coordinate mapping means 10 associates the coordinates on the spatial model MD with the coordinates on one or more input image planes R4 existing for each camera, and coordinates on the spatial model MD and the camera identifier. , And the coordinates on the input image plane R4 are associated and stored in the input image / spatial model correspondence map 40.

Further, since the coordinate mapping means 10 calculates the coordinate conversion using quaternions, it has an advantage that gimbal lock is not generated unlike the case where the coordinate conversion is calculated using Euler angles. .. However, the coordinate mapping means 10 is not limited to the one that calculates the coordinate conversion using quaternions, and may calculate the coordinate conversion using Euler angles.

When it is possible to associate the coordinates on the plurality of input image planes R4, the coordinate mapping means 10 inputs the coordinates P (P') on the spatial model MD with respect to the camera having the smallest incident angle α. It may be associated with the coordinates on the image plane R4, or may be associated with the coordinates on the input image plane R4 selected by the operator.

Next, among the coordinates on the spatial model MD, the process of reprojecting the coordinates on the curved surface region R2 (coordinates having a component in the Z-axis direction) onto the processing target image plane R3 on the XY plane will be described.

FIG. 6 is a diagram for explaining the correspondence between the coordinates by the coordinate matching means 10. F6A is a diagram showing the correspondence between the coordinates on the input image plane R4 of the camera 2 that employs normal projection (h = ftanα) as an example and the coordinates on the spatial model MD. The coordinate mapping means 10 is set so that each of the line segments connecting the coordinates on the input image plane R4 of the camera 2 and the coordinates on the spatial model MD corresponding to the coordinates passes through the optical center C of the camera 2. Associate both coordinates.

In the example of F6A, the coordinate mapping means 10 associates the coordinates K1 on the input image plane R4 of the camera 2 with the coordinates L1 on the plane region R1 of the spatial model MD, and the coordinates K2 on the input image plane R4 of the camera 2. Corresponds to the coordinates L2 on the curved surface region R2 of the spatial model MD. At this time, both the line segment K1-L1 and the line segment K2-L2 pass through the optical center C of the camera 2.

When the camera 2 employs a projection method other than the normal projection (for example, normal projection, stereographic projection, equal-angle projection, equal-distance projection, etc.), the coordinate mapping means 10 uses each projection method. Correspondingly, the coordinates K1 and K2 on the input image plane R4 of the camera 2 are associated with the coordinates L1 and L2 on the spatial model MD.

Specifically, the coordinate mapping means 10 has a predetermined function (for example, normal projection (h = fsinα), stereographic projection (h = 2ftan (α / 2)), equal-angle projection (h = 2fsin (α / 2)). 2)), equidistant projection (h = fα), etc.), the coordinates on the input image plane are associated with the coordinates on the spatial model MD. In this case, the line segments K1-L1 and the line segments K2-L2 do not pass through the optical center C of the camera 2.

F6B is a diagram showing the correspondence between the coordinates on the curved surface region R2 of the spatial model MD and the coordinates on the processing target image plane R3. The coordinate mapping means 10 introduces a parallel line group PL located on the XZ plane and forming an angle β with the image plane R3 to be processed. Then, in the coordinate mapping means 10, the coordinates on the curved surface region R2 of the spatial model MD and the coordinates on the processing target image plane R3 corresponding to the coordinates are both placed on one of the parallel line group PL. , Associate both coordinates.

In the example of F6B, the coordinate mapping means 10 associates the coordinates L2 on the curved surface region R2 of the spatial model MD with the coordinates M2 on the processing target image plane R3 on a common parallel line.

The coordinate mapping means 10 can also map the coordinates on the plane region R1 of the spatial model MD to the coordinates on the processing target image plane R3 using the parallel line group PL in the same manner as the coordinates on the curved surface region R2. is there. However, in the example of F6B, the plane region R1 and the processing target image plane R3 are common planes. Therefore, the coordinates L1 on the plane region R1 of the spatial model MD and the coordinates M1 on the processing target image plane R3 have the same coordinate values.

In this way, the coordinate mapping means 10 associates the coordinates on the spatial model MD with the coordinates on the processing target image plane R3, and associates the coordinates on the spatial model MD with the coordinates on the processing target image plane R3. It is stored in the spatial model / processing target image correspondence map 41.

F6C is a diagram showing the correspondence between the coordinates on the image plane R3 to be processed and the coordinates on the output image plane R5 of the virtual camera 2V that employs normal projection (h = ftanα) as an example. The image generation means 11 so that each of the line segments connecting the coordinates on the output image plane R5 of the virtual camera 2V and the coordinates on the processing target image plane R3 corresponding to the coordinates pass through the optical center CV of the virtual camera 2V. And associate both coordinates.

In the example of F6C, the image generation means 11 associates the coordinates N1 on the output image plane R5 of the virtual camera 2V with the coordinates M1 on the processing target image plane R3 (plane region R1 of the spatial model MD), and the virtual camera 2V The coordinates N2 on the output image plane R5 are associated with the coordinates M2 on the processing target image plane R3. At this time, both the line segment M1-N1 and the line segment M2-N2 pass through the optical center CV of the virtual camera 2V.

When the virtual camera 2V employs a projection method other than the normal projection (for example, normal projection, stereographic projection, equal-angle projection, equal-distance projection, etc.), the image generation means 11 uses each projection method. Correspondingly, the coordinates N1 and N2 on the output image plane R5 of the virtual camera 2V are associated with the coordinates M1 and M2 on the processing target image plane R3.

Specifically, the image generation means 11 has a predetermined function (for example, normal projection (h = fsinα), stereographic projection (h = 2ftan (α / 2)), equidistant angle projection (h = 2fsin (α / 2)). )), Equidistant projection (h = fα), etc.), the coordinates on the output image plane R5 and the coordinates on the processing target image plane R3 are associated with each other. In this case, the line segments M1-N1 and the line segments M2-N2 do not pass through the optical center CV of the virtual camera 2V.

In this way, the image generation means 11 associates the coordinates on the output image plane R5 with the coordinates on the processing target image plane R3, and obtains the coordinates on the output image plane R5 and the coordinates on the processing target image plane R3. It is associated and stored in the processing target image / output image correspondence map 42. Then, the image generation means 11 associates the value of each pixel in the output image with the value of each pixel in the input image while referring to the input image / spatial model correspondence map 40 and the spatial model / processing target image correspondence map 41. Generate an output image.

Note that F6D is a diagram in which F6A to F6C are combined, and shows the mutual positional relationship between the camera 2, the virtual camera 2V, the plane region R1 and the curved surface region R2 of the spatial model MD, and the image plane R3 to be processed.

Next, with reference to FIG. 7, the operation of the parallel line group PL introduced by the coordinate mapping means 10 for associating the coordinates on the spatial model MD with the coordinates on the processing target image plane R3 will be described.

The left figure of FIG. 7 is a view when an angle β is formed between the parallel line group PL located on the XZ plane and the image plane R3 to be processed. On the other hand, the right figure of FIG. 7 is a diagram when an angle β1 (β1> β) is formed between the parallel line group PL located on the XZ plane and the image plane R3 to be processed. Further, each of the coordinates La to Ld on the curved surface region R2 of the spatial model MD in the left figure of FIG. 7 and the right figure of FIG. 7 corresponds to the coordinates Ma to Md on the image plane R3 to be processed. Further, the respective intervals of the coordinates La to Ld in the left figure of FIG. 7 are equal to the respective intervals of the coordinates La to Ld in the right figure of FIG. 7. The parallel line group PL is assumed to exist on the XZ plane for the purpose of explanation, but actually exists so as to extend radially from all points on the Z axis toward the image plane R3 to be processed. To do. The Z axis in this case is referred to as a "reprojection axis".

As shown in the left view of FIG. 7 and the right view of FIG. 7, the distance between the coordinates Ma to Md on the processing target image plane R3 increases the angle between the parallel line group PL and the processing target image plane R3. It decreases linearly with. That is, it decreases uniformly regardless of the distance between the curved surface region R2 of the spatial model MD and each of the coordinates Ma to Md. On the other hand, in the example of FIG. 7, the coordinate group on the plane region R1 of the spatial model MD is not converted to the coordinate group on the image plane R3 to be processed, so that the distance between the coordinate groups does not change. ..

As for the change in the spacing of these coordinate groups, only the image portion corresponding to the image projected on the curved surface region R2 of the spatial model MD among the image portions on the output image plane R5 (see FIG. 6) is linearly enlarged or expanded. It means that it will be reduced.

Next, an alternative example of the parallel line group PL introduced by the coordinate mapping means 10 for associating the coordinates on the spatial model MD with the coordinates on the processing target image plane R3 will be described with reference to FIG.

FIG. 8 left is a diagram in which all of the auxiliary line groups AL located on the XZ plane extend from the start point T1 on the Z axis toward the image plane R3 to be processed. On the other hand, the right figure of FIG. 8 is a diagram in which all of the auxiliary line group AL extends from the start point T2 (T2> T1) on the Z axis toward the processing target image plane R3. Further, each of the coordinates La to Ld on the curved surface region R2 of the spatial model MD in the left figure of FIG. 8 and the right figure of FIG. 8 corresponds to the coordinates Ma to Md on the image plane R3 to be processed. In the example shown on the left of FIG. 8, the coordinates Mc and Md are not shown because they are outside the region of the image plane R3 to be processed. Further, the respective intervals of the coordinates La to Ld in the left figure of FIG. 8 are equal to the respective intervals of the coordinates La to Ld in the right figure of FIG. The auxiliary line group AL is assumed to exist on the XZ plane for the purpose of explanation, but actually exists so as to extend radially from an arbitrary point on the Z axis toward the image plane R3 to be processed. To do. As in FIG. 7, the Z axis in this case is referred to as a “reprojection axis”.

As shown in the left figure of FIG. 8 and the right figure of FIG. 8, the distance between the coordinates Ma to Md on the image plane R3 to be processed is the distance (height) between the start point and the origin O of the auxiliary line group AL. Decreases non-linearly as increases. That is, the larger the distance between the curved surface region R2 of the spatial model MD and each of the coordinates Ma to Md, the larger the decrease width of each interval becomes. On the other hand, in the example of FIG. 8, the coordinate group on the plane region R1 of the spatial model MD is not converted to the coordinate group on the processing target image plane R3, so that the distance between the coordinate groups does not change. ..

The change in the spacing of these coordinate groups corresponds to the image projected on the curved surface region R2 of the spatial model MD in the image portion on the output image plane R5 (see FIG. 6) as in the case of the parallel line group PL. It means that only the image portion is non-linearly enlarged or reduced.

In this way, the image generation device 100 does not affect the image portion (for example, a road surface image) of the output image corresponding to the image projected on the plane region R1 of the spatial model MD, and the spatial model MD. The image portion (for example, a horizontal image) of the output image corresponding to the image projected on the curved surface region R2 of the above can be enlarged or reduced linearly or non-linearly. Therefore, the image generator 100 does not affect the road surface image (virtual image when the excavator 60 is viewed from directly above) in the vicinity of the excavator 60, and the object located around the excavator 60 (horizontal direction from the excavator 60). The object in the image when the surroundings are viewed) can be quickly and flexibly enlarged or reduced, and the visibility of the blind spot region of the excavator 60 can be improved.

Next, referring to FIG. 9, a process in which the image generation device 100 generates a processing target image (hereinafter, referred to as “processing target image generation processing”), and an output image using the generated processing target image. The process of generating (hereinafter, referred to as "output image generation process") will be described. Note that FIG. 9 is a flowchart showing the flow of the processing target image generation processing (steps S1 to S3) and the output image generation processing (steps S4 to S6). Further, the arrangement of the camera 2 (input image plane R4), the spatial model (plane area R1 and curved surface area R2), and the image plane R3 to be processed is determined in advance.

First, the control unit 1 associates the coordinates on the processing target image plane R3 with the coordinates on the spatial model MD by the coordinate associating means 10 (step S1).

Specifically, the coordinate mapping means 10 acquires an angle formed between the parallel line group PL and the image plane R3 to be processed. Then, the coordinate mapping means 10 calculates a point where one of the parallel line group PLs extending from one coordinate on the processing target image plane R3 intersects the curved surface region R2 of the spatial model MD. Then, the coordinate mapping means 10 derives the coordinates on the curved surface region R2 corresponding to the calculated points as one coordinate on the curved surface region R2 corresponding to the one coordinate on the processing target image plane R3, and determines the correspondence relationship. It is stored in the spatial model / processing target image correspondence map 41. The angle formed between the parallel line group PL and the image plane R3 to be processed may be a value stored in advance in the storage unit 4 or the like, and the operator dynamically moves through the input unit 3. It may be a value to be entered.

Further, when one coordinate on the processing target image plane R3 matches one coordinate on the plane region R1 of the space model MD, the coordinate associating means 10 converts the one coordinate on the plane region R1 into the processing target image. It is derived as one coordinate corresponding to that one coordinate on the plane R3, and the correspondence relationship is stored in the spatial model / processing target image correspondence map 41.

After that, the control unit 1 associates one coordinate on the spatial model MD derived by the above process with the coordinate on the input image plane R4 by the coordinate associating means 10 (step S2).

Specifically, the coordinate mapping means 10 acquires the coordinates of the optical center C of the camera 2 that normally employs projection (h = ftanα). Then, the coordinate mapping means 10 is a line segment extending from one coordinate on the spatial model MD, and calculates a point at which the line segment passing through the optical center C intersects the input image plane R4. Then, the coordinate mapping means 10 derives the coordinates on the input image plane R4 corresponding to the calculated points as one coordinate on the input image plane R4 corresponding to the one coordinate on the spatial model MD, and determines the correspondence relationship. It is stored in the input image / spatial model correspondence map 40.

After that, the control unit 1 determines whether or not all the coordinates on the processing target image plane R3 are associated with the coordinates on the spatial model MD and the coordinates on the input image plane R4 (step S3). Then, when the control unit 1 determines that all the coordinates are not yet associated (NO in step S3), the control unit 1 repeats the processes of steps S1 and S2.

On the other hand, when it is determined that all the coordinates are associated with each other (YES in step S3), the control unit 1 ends the processing target image generation processing and then starts the output image generation processing. Then, the control unit 1 associates the coordinates on the processing target image plane R3 with the coordinates on the output image plane R5 by the image generation means 11 (step S4).

Specifically, the image generation means 11 generates an output image by performing scale transformation, affine transformation, or distortion transformation on the image to be processed. Then, the image generation means 11 determines the correspondence between the coordinates on the processing target image plane R3 and the coordinates on the output image plane R5, which are determined by the contents of the scale conversion, the affine transformation, or the distortion conversion applied to the processing target image. -Store in the output image correspondence map 42.

Alternatively, when the image generation means 11 generates an output image using the virtual camera 2V, the image generation means 11 calculates the coordinates on the output image plane R5 from the coordinates on the processing target image plane R3 according to the projection method adopted. The correspondence may be stored in the processing target image / output image correspondence map 42.

Alternatively, when the image generation means 11 generates an output image using a virtual camera 2V that employs normal projection (h = ftanα), the image generation means 11 acquires the coordinates of the optical center CV of the virtual camera 2V. Then, the image generation means 11 calculates a line segment extending from one coordinate on the output image plane R5, and a point at which the line segment passing through the optical center CV intersects the processing target image plane R3. Then, the image generation means 11 derives the coordinates on the processing target image plane R3 corresponding to the calculated points as one coordinate on the processing target image plane R3 corresponding to the one coordinate on the output image plane R5. In this way, the image generation means 11 may store the correspondence relationship in the processing target image / output image correspondence map 42.

After that, the image generation means 11 of the control unit 1 refers to the input image / spatial model correspondence map 40, the spatial model / processing target image correspondence map 41, and the processing target image / output image correspondence map 42. Then, the image generation means 11 has a correspondence relationship between the coordinates on the input image plane R4 and the coordinates on the space model MD, a correspondence relationship between the coordinates on the space model MD and the coordinates on the processing target image plane R3, and the processing target. The correspondence between the coordinates on the image plane R3 and the coordinates on the output image plane R5 is traced. Then, the image generation means 11 acquires the values (for example, luminance value, hue value, saturation value, etc.) possessed by the coordinates on the input image plane R4 corresponding to each coordinate on the output image plane R5. The acquired value is adopted as the value of each coordinate on the corresponding output image plane R5 (step S5). When a plurality of coordinates on the plurality of input image planes R4 correspond to one coordinate on the output image plane R5, the image generation means 11 has each of the plurality of coordinates on the plurality of input image planes R4. A statistical value based on the value may be derived, and the statistical value may be adopted as the value of the one coordinate on the output image plane R5. The statistical value is, for example, an average value, a maximum value, a minimum value, an intermediate value, or the like.

After that, the control unit 1 determines whether or not all the coordinate values on the output image plane R5 are associated with the coordinate values on the input image plane R4 (step S6). Then, when the control unit 1 determines that the values of all the coordinates are not yet associated (NO in step S6), the control unit 1 repeats the processes of steps S4 and S5.

On the other hand, when it is determined that the values of all the coordinates are associated with each other (YES in step S6), the control unit 1 generates an output image and ends this series of processing.

If the image generation device 100 does not generate the processing target image, the processing target image generation processing is omitted. In this case, the "coordinates on the processing target image plane" in step S4 in the output image generation processing are read as "coordinates on the spatial model".

With the above configuration, the image generation device 100 can generate a processing target image and an output image that allow the operator to intuitively grasp the positional relationship between the object around the excavator 60 and the excavator 60.

Further, the image generation device 100 executes coordinate mapping so as to trace back from the processing target image plane R3 to the input image plane R4 via the spatial model MD. As a result, the image generation device 100 can surely make each coordinate on the processing target image plane R3 correspond to one or a plurality of coordinates on the input image plane R4. Therefore, the image generation device 100 can quickly generate a higher quality processing target image as compared with the case where the coordinates are associated in the order from the input image plane R4 to the processing target image plane R3 via the spatial model MD. Can be done. When the coordinates are associated in the order from the input image plane R4 to the processing target image plane R3 via the spatial model MD, each coordinate on the input image plane R4 is converted into one or a plurality of coordinates on the processing target image plane R3. It is possible to surely correspond to the coordinates. However, some of the coordinates on the processing target image plane R3 may not be associated with any of the coordinates on the input image plane R4, and in that case, some of the coordinates on the processing target image plane R3. It is necessary to perform interpolation processing or the like.

Further, the image generation device 100 changes the angle formed between the parallel line group PL and the image plane R3 to be processed when only the image corresponding to the curved surface region R2 of the spatial model MD is enlarged or reduced. By rewriting only the portion related to the curved surface region R2 in the spatial model / processing target image correspondence map 41, the desired enlargement or reduction can be realized without rewriting the contents of the input image / spatial model correspondence map 40. ..

Further, when changing the appearance of the output image, the image generation device 100 simply rewrites the processing target image / output image correspondence map 42 by changing the values of various parameters related to scale conversion, affine conversion, or distortion conversion. It is possible to generate a desired output image (scale conversion image, affine conversion image or distortion conversion image) without rewriting the contents of the input image / spatial model correspondence map 40 and the spatial model / processing target image correspondence map 41.

Similarly, when the viewpoint of the output image is changed, the image generation device 100 simply changes the values of various parameters of the virtual camera 2V and rewrites the processing target image / output image correspondence map 42, and the input image / space. An output image (viewpoint conversion image) viewed from a desired viewpoint can be generated without rewriting the contents of the model correspondence map 40 and the spatial model / processing target image correspondence map 41.

FIG. 10 is a display example when the output image generated by using the input images of the two cameras 2 (right side camera 2R and rear camera 2B) mounted on the excavator 60 is displayed on the display unit 5. ..

The image generation device 100 generates an image to be processed by projecting the input images of the two cameras 2 onto the plane region R1 and the curved surface region R2 of the spatial model MD and then reprojecting them onto the processing target image plane R3. To do. Then, the image generation device 100 generates an output image by performing an image conversion process (for example, scale conversion, affine transformation, distortion transformation, viewpoint conversion processing, etc.) on the generated image to be processed. In this way, the image generator 100 looks down on the vicinity of the excavator 60 from the sky (image in the plane region R1) and an image in which the surroundings are viewed in the horizontal direction from the excavator 60 (image in the image plane R3 to be processed). Generates an output image that displays and at the same time. Hereinafter, such an output image will be referred to as a peripheral monitoring virtual viewpoint image.

When the image generation device 100 does not generate the image to be processed, the virtual viewpoint image for peripheral monitoring is subjected to image conversion processing (for example, viewpoint conversion processing) on the image projected on the spatial model MD. Generated by.

Further, the virtual viewpoint image for peripheral monitoring is trimmed into a circle so that the image when the excavator 60 performs a turning operation can be displayed without discomfort, and the center CTR of the circle is on the cylindrical center axis of the space model MD and. It is generated so as to be on the turning axis PV of the excavator 60. Therefore, the peripheral monitoring virtual viewpoint image is displayed so as to rotate about the central CTR according to the turning motion of the excavator 60. In this case, the central axis of the cylinder of the spatial model MD may or may not coincide with the reprojection axis.

The radius of the space model MD is, for example, 5 meters. Further, the angle formed by the parallel line group PL with the processing target image plane R3 or the starting point height of the auxiliary line group AL is the maximum reachable distance of the excavation attachment E (for example, 12 meters) from the turning center of the excavator 60. It is set so that when an object (for example, a worker) is present at a position separated by (for example, a worker), the object is displayed sufficiently large (for example, 7 mm or more) on the display unit 5. obtain.

Further, as the peripheral monitoring virtual viewpoint image, the CG image of the excavator 60 may be arranged so that the front of the excavator 60 coincides with the upper part of the screen of the display unit 5 and the turning center thereof coincides with the center CTR. .. This is to make it easier to understand the positional relationship between the excavator 60 and the object appearing in the output image. As the virtual viewpoint image for peripheral monitoring, a frame image including various information such as orientation may be arranged around the frame image.

Next, the details of the peripheral monitoring virtual viewpoint image generated by the image generation device 100 will be described with reference to FIGS. 11 to 14.

FIG. 11 is a top view of the excavator 60 equipped with the image generation device 100. In the embodiment shown in FIG. 11, the excavator 60 includes three cameras 2 (left side camera 2L, right side camera 2R, and rear camera 2B) and three person detection sensors 6 (left side person detection sensor 6L, right side). It is equipped with a person detection sensor 6R and a rear person detection sensor 6B). The regions CL, CR, and CB shown by the alternate long and short dash lines in FIG. 11 indicate the imaging spaces of the left side camera 2L, the right side camera 2R, and the rear camera 2B, respectively. Further, the regions ZL, ZR, and ZB shown by the dotted lines in FIG. 11 indicate the monitoring spaces of the left side person detection sensor 6L, the right side person detection sensor 6R, and the rear person detection sensor 6B, respectively. Further, the excavator 60 includes a display unit 5, three alarm output units 7 (left side alarm output unit 7L, right side alarm output unit 7R, and rear alarm output unit 7B), and a gate lock lever in the cab 64. 8 and an ignition switch 9 are provided.

In this embodiment, the monitoring space of the person detection sensor 6 is narrower than the imaging space of the camera 2, but the monitoring space of the person detection sensor 6 may be the same as the imaging space of the camera 2, and is larger than the imaging space of the camera 2. It may be wide. Further, the monitoring space of the human detection sensor 6 is located in the vicinity of the excavator 60 in the imaging space of the camera 2, but may be in a region farther from the excavator 60. Further, the monitoring space of the human detection sensor 6 has an overlapping portion in the portion where the imaging space of the camera 2 overlaps. For example, in the overlapping portion between the imaging space CR of the right side camera 2R and the imaging space CB of the rear camera 2B, the monitoring space ZR of the right side person detection sensor 6R overlaps with the monitoring space ZB of the rear person detection sensor 6B. However, the monitoring space of the human detection sensor 6 may be arranged so as not to cause duplication.

FIG. 12 is a diagram showing input images of each of the three cameras 2 mounted on the excavator 60 and output images generated by using the input images.

The image generation device 100 generates an image to be processed by projecting the input images of the three cameras 2 onto the plane area R1 and the curved surface area R2 of the spatial model MD and then reprojecting them onto the image plane R3 to be processed. .. Further, the image generation device 100 generates an output image by performing an image conversion process (for example, scale conversion, affine transformation, distortion transformation, viewpoint conversion processing, etc.) on the generated image to be processed. As a result, the image generation device 100 obtains an image of the vicinity of the excavator 60 looking down from the sky (an image in the plane region R1) and an image of the surroundings in the horizontal direction from the excavator 60 (an image in the image plane R3 to be processed). Generates a virtual viewpoint image for peripheral monitoring to be displayed at the same time. The image displayed in the center of the peripheral monitoring virtual viewpoint image is the CG image 60CG of the excavator 60.

In FIG. 12, the input image of the right side camera 2R and the input image of the rear camera 2B each capture a person in the overlapping portion between the image pickup space of the right side camera 2R and the image pickup space of the rear camera 2B (right side). See the area R10 surrounded by the two-point chain line in the input image of the square camera 2R and the area R11 surrounded by the two-point chain line in the input image of the rear camera 2B).

However, assuming that the coordinates on the output image plane are associated with the coordinates on the input image plane with respect to the camera having the smallest incident angle, the output image causes the person in the overlapping portion to disappear (one point in the output image). See region R12 surrounded by chain lines.)

Therefore, in the output image portion corresponding to the overlapping portion, the image generation device 100 associates the region on which the coordinates on the input image plane of the rear camera 2B are associated with the coordinates on the input image plane of the right camera 2R. The area is mixed to prevent the objects in the overlapping part from disappearing.

FIG. 13 is a diagram for explaining a stripe pattern process which is an example of an image loss prevention process for preventing the disappearance of an object in an overlapping portion of the imaging space of each of the two cameras.

F13A is a diagram showing an output image portion corresponding to an overlapping portion between the imaging space of the right-side camera 2R and the imaging space of the rear camera 2B, and corresponds to the rectangular region R13 shown by the dotted line in FIG.

Further, in F13A, the area PR1 filled with gray is an image area in which the input image portion of the rear camera 2B is arranged, and the input image of the rear camera 2B is set at each coordinate on the output image plane corresponding to the area PR1. Coordinates on the plane are associated.

On the other hand, the area PR2 filled with white is an image area in which the input image portion of the right-side camera 2R is arranged, and the input image plane of the right-side camera 2R is set at each coordinate on the output image plane corresponding to the partial PR2. The above coordinates are associated.

In this embodiment, the region PR1 and the region PR2 are arranged so as to form a striped pattern (striped pattern treatment), and the boundary line of the portion where the region PR1 and the region PR2 are alternately arranged in a striped pattern is the swivel of the excavator 60. It is defined by concentric circles on the horizontal plane centered on the center.

F13B is a top view showing the state of the space area diagonally rearward to the right of the excavator 60, and shows the current state of the space area imaged by both the rear camera 2B and the right side camera 2R. Further, F13B indicates that a rod-shaped three-dimensional object OB exists diagonally to the right and rearward of the excavator 60.

F13C shows a part of the output image generated based on the input image obtained by actually capturing the spatial region indicated by F13B with the rear camera 2B and the right side camera 2R.

Specifically, in the image OB1, the image of the three-dimensional object OB in the input image of the rear camera 2B extends in the extension direction of the line connecting the rear camera 2B and the three-dimensional object OB by the viewpoint transformation for generating the road surface image. Represents what has been done. That is, the image OB1 is a part of the image of the three-dimensional object OB displayed when the road surface image in the output image portion is generated by using the input image of the rear camera 2B.

Further, in the image OB2, the image of the three-dimensional object OB in the input image of the right-side camera 2R is extended in the extension direction of the line connecting the right-side camera 2R and the three-dimensional object OB by the viewpoint transformation for generating the road surface image. Represents an image. That is, the image OB2 is a part of the image of the three-dimensional object OB displayed when the road surface image in the output image portion is generated by using the input image of the right-side camera 2R.

As described above, in the overlapping portion, the image generation device 100 has the region PR1 to which the coordinates on the input image plane of the rear camera 2B are associated and the region PR2 to which the coordinates on the input image plane of the right camera 2R are associated. To mix. As a result, the image generation device 100 displays both the two images OB1 and the image OB2 relating to one three-dimensional object OB on the output image, and prevents the three-dimensional object OB from disappearing from the output image.

FIG. 14 is a comparison diagram showing the difference between the output image of FIG. 12 and the output image obtained by applying the image disappearance prevention process (stripe pattern process) to the output image of FIG. The output image of FIG. 12 is shown, and the lower figure of FIG. 14 shows the output image after applying the image loss prevention process (strip pattern process). While the person disappears in the area R12 surrounded by the alternate long and short dash line in the upper figure of FIG. 14, the person is displayed without disappearing in the area R14 surrounded by the alternate long and short dash line in FIG. 14 lower figure.

The image generation device 100 may apply a mesh pattern processing, an averaging process, or the like instead of the stripe pattern processing to prevent the disappearance of the object in the overlapping portion. Specifically, the image generation device 100 outputs the average value of the corresponding pixel values (for example, the luminance value) in the input images of the two cameras by the averaging process to correspond to the overlapping portion. It is used as the pixel value of the image part. Alternatively, the image generation device 100 performs mesh pattern processing to associate the pixel values in the input image of one camera with the pixel values in the input image of the other camera in the output image portion corresponding to the overlapping portion. Are arranged so as to form a mesh pattern with the area to which is associated with. As a result, the image generation device 100 prevents the object in the overlapping portion from disappearing.

Next, with reference to FIGS. 15 to 17, a process in which the image generation means 11 switches the content of the output image based on the determination result of the presence / absence determination means 12 (hereinafter referred to as “first output image switching process”). .) Will be described. Note that FIG. 15 is a diagram showing input images of each of the three cameras 2 mounted on the excavator 60 and output images generated by using the input images, and corresponds to FIG. Further, FIG. 16 is a diagram showing an example of the relationship between the spatial model MD used in the first output image switching process and the image plane R3 to be processed, and corresponds to FIG. Further, FIG. 17 is a diagram illustrating the relationship between the two output images switched by the first output image switching process.

As shown in FIG. 15, the image generation device 100 projects the input images of the three cameras 2 onto the plane region R1 and the curved surface region R2 of the spatial model MD, and then reprojects them onto the processing target image plane R3. To generate an image to be processed. Further, the image generation device 100 generates an output image by performing an image conversion process (for example, scale conversion, affine transformation, distortion transformation, viewpoint conversion processing, etc.) on the generated image to be processed. As a result, the image generation device 100 generates a virtual viewpoint image for peripheral monitoring that simultaneously displays an image of the vicinity of the excavator 60 looking down from the sky and an image of the surroundings viewed horizontally from the excavator 60.

Further, in FIG. 15, each input image of the left side camera 2L, the rear side camera 2B, and the right side camera 2R shows a state in which three workers are present. Further, the output image shows a state in which nine workers are present around the excavator 60.

The height of the plane region R1 of the space model MD is set to a height corresponding to the road surface which is the ground plane of the excavator 60. Therefore, in the image in the plane region R1 of FIG. 15, the distance between the ground contact position of each of the nine workers and the CG image 60CG of the excavator 60 is between each of the nine workers and the excavator 60. It accurately represents the actual distance. However, the image of the operator is displayed so as to increase as the distance from the ground contact position (foot) increases. In particular, as shown in FIG. 15, the head of the worker is displayed remarkably larger than the size of the CG image 60CG of the excavator 60. Therefore, the operator of the excavator 60 who sees the output image of FIG. 15 may think that the distance between the excavator 60 and the operator is larger than the actual distance.

Therefore, when the person presence / absence determining means 12 determines that a person exists in any of the left side monitoring space ZL, the rear monitoring space ZB, and the right side monitoring space ZR, the image generation means 11 determines the content of the output image. Switch. In the present embodiment, the image generation means 11 sets the height of the plane region R1 of the spatial model MD from the height corresponding to the road surface to the height corresponding to the height of the human head (hereinafter referred to as “head height”). To.). The head height is a preset value, for example, 150 cm. However, the head height may be a dynamically determined value. For example, when the image generator 100 can detect the height (height) of the worker existing around the excavator 60 based on the output of the person detection sensor 6 or the like, the image generator 100 determines the head height based on the detected height. You may. Specifically, the head height may be determined according to the height of the worker closest to the excavator 60, and the height statistics (maximum value, minimum value, etc.) of a plurality of workers existing around the excavator 60 may be determined. The head height may be determined according to the average value, etc.).

Here, with reference to FIGS. 16 and 17, the space model MD of FIG. 4 having a plane region R1 having a height corresponding to the road surface and the head height reference space model MDM having a plane region R1M of head height The relationship between The upper view of FIG. 16 shows the relationship between the spatial model MD and the image plane R3 to be processed including the plane area R1, and the lower figure of FIG. 16 shows the head height reference space model MDM and the head height reference plane area R1M. The relationship with the head height reference processing target image plane R3M including the above is shown. Further, in FIG. 17, the output screen D1 is a virtual viewpoint image for peripheral monitoring based on the road surface height generated by using the spatial model MD, and the output image D2 is the head height generated by using the spatial model MDM. This is a virtual viewpoint image for peripheral monitoring of the standard. The image D3 is an explanatory image showing the difference in size between the peripheral monitoring virtual viewpoint image based on the road surface height and the peripheral monitoring virtual viewpoint image based on the head height. Further, the size of the image portion D10 in the peripheral monitoring virtual viewpoint image based on the road surface height corresponds to the size of the peripheral monitoring virtual viewpoint image based on the head height.

As shown in FIG. 16, the heights of the head height reference plane region R1M and the head height reference processing target image plane R3M are compared with the heights of the plane region R1M corresponding to the height of the road surface and the processing target image plane R3. , Head height HT is high.

As a result, as shown in FIG. 17, the output image D2, which is a virtual viewpoint image for peripheral monitoring based on the head height, has a distance between the worker's body part (head) existing at the head height HT and the excavator 60. , Displayed to represent the actual distance between the operator and the excavator 60. As a result, the image generation device 100 can prevent the operator of the excavator 60 who sees the output image from perceiving the distance between the excavator 60 and the operator larger than the actual distance.

As shown in FIG. 16, the region D10M on the spatial model MDM corresponding to the image portion D10 is included in the head height reference plane region R1M. That is, the virtual viewpoint image for peripheral monitoring based on the head height does not use the input image portion reprojected on the annular portion (the portion other than the head height reference plane region R1M) of the head height reference processing target image plane R3M. .. Therefore, in this embodiment, the image generation device 100 omits the association of coordinates in the area other than the area D10M.

With the above configuration, the image generation device 100 switches the content of the output image based on the determination result of the human existence determination means 12. Specifically, when it is determined that a person exists around the excavator 60, the image generation device 100 switches the virtual viewpoint image for peripheral monitoring based on the road surface height to the virtual viewpoint image for peripheral monitoring based on the head height. .. As a result, when the image generation device 100 detects an operator around the excavator 60, the distance between the excavator 60 and the operator can be more accurately transmitted to the operator of the excavator 60. When the image generator 100 subsequently determines that there is no person around the excavator 60, the image generation device 100 changes the virtual viewpoint image for peripheral monitoring based on the head height into a virtual viewpoint image for peripheral monitoring based on the road surface height. Switch. This is to allow the operator of the excavator 60 to monitor the surroundings of the excavator 60 in a wider area.

Next, referring to FIG. 18, another process in which the image generation means 11 switches the content of the output image based on the determination result of the presence / absence determination means 12 (hereinafter referred to as “second output image switching process””. ) Will be explained. Note that FIG. 18 is a diagram illustrating the relationship between the three output images switched by the second output image switching process.

The image generation device 100 executes image loss prevention processing such as stripe pattern processing, mesh pattern processing, and averaging processing for preventing the disappearance of objects in the overlapping portions of the imaging spaces of the two cameras. Specifically, the image generation device 100 duplicates by mixing the regions in which the coordinates on the input image planes of the two cameras are associated with each other in the output image portion corresponding to the overlapping portion by the image disappearance prevention processing. Prevent the disappearance of objects in the part. However, the image loss prevention process has a problem that the visibility of the object in the overlapping portion is lowered.

Therefore, the image generation means 11 switches the content of the output image according to the determination result of the presence / absence determination means 12. In the present embodiment, the image generation means 11 uses the camera corresponding to the monitoring space determined to have a person as the priority camera, and the camera corresponding to the monitoring space determined to have no person to be the priority camera. The imaging space of the priority camera and the imaging space of the priority camera have overlapping portions. Then, when the priority camera and the priority camera cannot be determined, the image generation means 11 generates a virtual viewpoint image for peripheral monitoring to which the image loss prevention process is applied. On the other hand, when the priority camera and the priority camera can be determined, the image generation means 11 applies the image loss prevention process to the output image portion corresponding to the overlapping portion on the input image plane of the priority camera. A virtual viewpoint image for peripheral monitoring is generated by associating the coordinates of.

Specifically, the image generation means 11 applies image loss prevention processing to peripheral monitoring when it is determined that a person exists in all of the left side monitoring space ZL, the rear side monitoring space ZB, and the right side monitoring space ZR. Generate a virtual viewpoint image for use. This is because none of the cameras can be a priority camera and no camera can be a priority camera. That is, if any of the cameras is used as the priority camera, there is a risk that people in the overlapping portion will disappear.

Further, the image generation means 11 is for peripheral monitoring to which the image loss prevention process is applied even when it is determined that no person exists in all of the left side monitoring space ZL, the rear monitoring space ZB, and the right side monitoring space ZR. Generate a virtual viewpoint image. This is because there is no target to be displayed while using any camera as the priority camera. However, in this case, the image generation means 11 monitors the periphery by associating the coordinates on the input image plane of one of the cameras with the output image portion corresponding to the overlapping portion without applying the image loss prevention processing. A virtual viewpoint image for use may be generated. This is because there is no object that may disappear in the first place, and the processing load of the control unit 1 can be reduced by omitting the image loss prevention process.

The output image D11 of FIG. 18 is a peripheral area to which the image loss prevention process is applied, which is generated when it is determined that a person exists in all of the left side monitoring space ZL, the rear side monitoring space ZB, and the right side monitoring space ZR. A virtual viewpoint image for monitoring is shown. In this case, the image generation means 11 rearward the output image portion R15 corresponding to the overlapping portion of the left side imaging space CL and the rear imaging space CB with the region in which the coordinates on the input image plane of the left side camera 2L are associated with each other. The area on which the coordinates on the input image plane of the camera 2B are associated is mixed. Further, the image generation means 11 includes a region in which the output image portion R16 corresponding to the overlapping portion of the right side imaging space CR and the rear imaging space CB is associated with the coordinates on the input image plane of the right side camera 2R, and the rear camera. A region to which the coordinates on the input image plane of 2B are associated is mixed. This is to prevent the disappearance of the operator in the output image portions R15 and R16.

On the other hand, when it is determined that a person exists in the left side monitoring space ZL and it is determined that there is no person in the rear monitoring space ZB, the image generation means 11 sets the left side camera 2L as the priority camera. The rear camera 2B is used as the priority camera. That is, the image generation means 11 associates the coordinates on the input image plane of the left side camera 2L with the output image part R15 corresponding to the overlapping part of the left side image pickup space CL and the rear side image pickup space CB.

Further, when it is determined that a person exists in the right side monitoring space ZR and it is determined that there is no person in the rear monitoring space ZB, the image generation means 11 sets the right side camera 2R as the priority camera and rearward. Let the camera 2B be the priority camera. That is, the image generation means 11 associates the coordinates on the input image plane of the right-side camera 2R with the output image portion R16 corresponding to the overlapping portion of the right-side imaging space CR and the rear imaging space CB.

The output image D12 of FIG. 18 is a side camera generated when it is determined that a person exists in the left side monitoring space ZL and the right side monitoring space ZR, and it is determined that there is no person in the rear monitoring space ZB. A virtual viewpoint image for priority peripheral monitoring is shown. In this case, the image generation means 11 uses each of the left side camera 2L and the right side camera 2R as the priority camera, and the rear camera 2B as the priority camera. Then, the image generation means 11 associates the coordinates on the input image plane of the left side camera 2L with the output image part R15 corresponding to the overlapping part of the left side image pickup space CL and the rear side image pickup space CB, and sets the right side image pickup space CR. The coordinates on the input image plane of the right-side camera 2R are associated with the output image portion R16 corresponding to the overlapping portion of the rear imaging space CB. As a result, the image generation means 11 associates the output image portion R17P with the coordinates on the input image plane of the left camera 2L, and associates the output image portion R18P with the coordinates on the input image plane of the right camera 2R, and outputs the output image portion R18P. The coordinates on the input image plane of the rear camera 2B are associated with the image portion R19N.

Further, when it is determined that there is no person in the left side monitoring space ZL and it is determined that there is a person in the rear monitoring space ZB, the image generation means 11 sets the left side camera 2L as the priority camera. The rear camera 2B is the priority camera. That is, the image generation means 11 associates the coordinates on the input image plane of the rear camera 2B with the output image portion R15 corresponding to the overlapping portion of the left side imaging space CL and the rear imaging space CB.

Further, when it is determined that there is no person in the right side monitoring space ZR and it is determined that there is a person in the rear monitoring space ZB, the image generation means 11 sets the right side camera 2R as the priority camera. The rear camera 2B is the priority camera. That is, the image generation means 11 associates the coordinates on the input image plane of the rear camera 2B with the output image portion R16 corresponding to the overlapping portion of the right side imaging space CR and the rear imaging space CB.

The output image D13 of FIG. 18 is a rear camera generated when it is determined that there is no person in the left side monitoring space ZL and the right side monitoring space ZR, and it is determined that there is a person in the rear monitoring space ZB. A virtual viewpoint image for priority peripheral monitoring is shown. In this case, the image generation means 11 sets each of the left side camera 2L and the right side camera 2R as the priority camera, and the rear camera 2B as the priority camera. Then, the image generation means 11 associates the coordinates on the input image plane of the rear camera 2B with the output image portion R15 corresponding to the overlapping portion of the left side imaging space CL and the rear imaging space CB, and the right side imaging space CR and the rear. Coordinates on the input image plane of the rear camera 2B are also associated with the output image portion R16 corresponding to the overlapping portion of the imaging space CB. As a result, the image generation means 11 associates the output image portion R17N with the coordinates on the input image plane of the left camera 2L, and associates the output image portion R18N with the coordinates on the input image plane of the right camera 2R, and outputs the output image portion R18N. The coordinates on the input image plane of the rear camera 2B are associated with the image portion R19P.

As shown in FIG. 18, the output image portion R17P corresponds to a portion obtained by adding the output image portion R15 to the output image portion R17N, and the output image portion R18P is obtained by adding the output image portion R16 to the output image portion R18N. Corresponds to the part. Further, the output image portion R19P corresponds to a portion obtained by adding the output image portion R15 and the output image portion R16 to the output image portion R19N.

FIG. 19 is a correspondence table showing a correspondence relationship between the determination result of the presence / absence determination means 12 and the content of the output image. The ◯ mark indicates that the presence / absence determination means 12 has determined that a person exists, and the × mark indicates that the presence / absence of a person has been determined.

Pattern A is as shown in the output image D12 of FIG. 18 when it is determined that a person exists only in the left side monitoring space ZL and no person exists in the rear monitoring space ZB and the right side monitoring space ZR. Indicates that a virtual viewpoint image for peripheral monitoring that gives priority to the side camera is generated.

Pattern B is as shown in the output image D12 of FIG. 18 when it is determined that a person exists only in the right side monitoring space ZR and no person exists in the left side monitoring space ZL and the rear monitoring space ZB. Indicates that a virtual viewpoint image for peripheral monitoring that gives priority to the side camera is generated.

In the pattern C, when it is determined that a person exists in the left side monitoring space ZL and the right side monitoring space ZR, and no person exists in the rear monitoring space ZB, the side as shown in the output image D12 of FIG. Indicates that a virtual viewpoint image for peripheral monitoring that gives priority to the square camera is generated.

In the pattern D, when it is determined that a person exists in the left side monitoring space ZL and the rear side monitoring space ZB, and no person exists in the right side monitoring space ZR, the pattern D is as shown in the output image D13 of FIG. Indicates that a camera-priority virtual viewpoint image for peripheral monitoring is generated.

Patterns E to G prevent image disappearance as shown in the output image D11 of FIG. 18 when it is determined that a person exists in at least one of the left side monitoring space ZL and the right side monitoring space ZR and the rear monitoring space ZB. Indicates that a virtual viewpoint image for peripheral monitoring to which the processing is applied is generated.

In the pattern H, when it is determined that no person exists in all of the left side monitoring space ZL, the rear side monitoring space ZB, and the right side monitoring space ZR, the image disappearance prevention process as shown in the output image D11 of FIG. 18 is performed. Indicates that the applied peripheral monitoring virtual viewpoint image is generated.

With the above configuration, the image generation device 100 switches the content of the output image based on the determination result of the human existence determination means 12. Specifically, when the priority camera and the priority camera cannot be determined, the image generation device 100 generates a virtual viewpoint image for peripheral monitoring to which the image loss prevention process is applied. On the other hand, when the priority camera and the priority camera can be determined, the image generation means 11 applies the image loss prevention process to the output image portion corresponding to the overlapping portion on the input image plane of the priority camera. A virtual viewpoint image for peripheral monitoring is generated by associating the coordinates. As a result, the image generation device 100 can more clearly display the operator existing in the overlapping portion of the imaging space of each of the two cameras.

Further, in the above-described embodiment, the image generation device 100 generates a virtual viewpoint image for peripheral monitoring that prioritizes the side camera with the left side camera 2L and the right side camera 2R as priority cameras and the rear camera 2B as the priority camera. To do. Alternatively, the image generation device 100 generates a peripheral monitoring virtual viewpoint image in which the left side camera 2L and the right side camera 2R are the priority cameras and the rear camera 2B is the priority camera. However, the present invention is not limited to this configuration. For example, the image generation device 100 applies the image loss prevention process to the output image portion R16 while associating the output image portion R15 with the coordinates on the input image plane of either the left side camera 2L or the rear camera 2B. You may. Further, the image generation device 100 applies the image loss prevention process to the output image portion R15 while associating the output image portion R16 with the coordinates on the input image plane of either the right side camera 2R or the rear camera 2B. You may.

Further, the image generation device 100 may combine the first output image switching process and the second output image switching process.

Further, in the above-described embodiment, the image generation device 100 associates the monitoring space of one person detection sensor with the imaging space of one camera, but provides the monitoring space of one person detection sensor with the imaging space of a plurality of cameras. It may be made to correspond, and the monitoring space of a plurality of person detection sensors may correspond to the imaging space of one camera.

Further, in the above-described embodiment, the image generation device 100 switches the content of the output image at the moment when the determination result of the presence / absence determination means 12 changes. However, the present invention is not limited to this configuration. For example, the image generation device 100 may set a predetermined delay time from the change of the determination result of the presence / absence determination means 12 to the switching of the content of the output image. This is to prevent the contents of the output image from being frequently switched.

Next, referring to FIGS. 20 and 21, a process in which the alarm control means 13 controls the alarm output unit 7 based on the determination result of the presence / absence determination means 12 (hereinafter referred to as “first alarm control process”). ) Will be explained. Note that FIG. 20 is a flowchart showing the flow of the first alarm control process, and FIG. 21 shows an example of the transition of the output image displayed during the first alarm control process. The alarm control means 13 repeatedly executes this first alarm control process at a predetermined cycle.

First, the presence / absence determination means 12 determines whether or not there is a person around the excavator 60 (step S11). At this time, the image generation means 11 generates and displays, for example, a virtual viewpoint image for peripheral monitoring based on the road surface height as shown in the output image D21 of FIG.

When it is determined that a person exists around the excavator 60 (YES in step S11), the presence / absence determination means 12 outputs a detection signal to the alarm control means 13. Upon receiving the detection signal, the alarm control means 13 outputs an alarm start signal for starting an alarm to the alarm output unit 7, and outputs an alarm from the alarm output unit 7 (step S12).

In this embodiment, the alarm output unit 7 outputs an alarm sound. Then, for example, as shown in the output image D22 of FIG. 21, the image generation means 11 switches and displays the peripheral monitoring virtual viewpoint image based on the road surface height to the peripheral monitoring virtual viewpoint image based on the head height. Specifically, the presence / absence determining means 12 notifies the alarm control means 13 of the detection signal when it is determined that the worker P1 exists in the rear monitoring space ZB. Then, the alarm control means 13 outputs an alarm start signal to the left side alarm output unit 7L, the rear alarm output unit 7B, and the right side alarm output unit 7R, and outputs an alarm sound from all three alarm output units. Let me. Further, the image generation means 11 superimposes and displays the alarm stop button G1 on the peripheral monitoring virtual viewpoint image based on the head height. The alarm stop button G1 is a software button configured in cooperation with a touch panel as an input unit 3. The operator can stop the alarm sound by pressing the alarm stop button G1. The alarm stop button G1 may be a hardware button installed in the vicinity of the display unit 5. In this case, the image generation means 11 may superimpose and display a text message indicating that the alarm can be stopped by pressing the hardware button as the alarm stop button G1 on the peripheral monitoring virtual viewpoint image. Further, even when it is determined that a person exists around the excavator 60, the image generation means 11 keeps the virtual viewpoint image for peripheral monitoring based on the road surface height as it is, as shown in the output image D23 of FIG. The alarm stop button G1 may be superimposed and displayed while being used.

On the other hand, when it is determined that there is no person around the excavator 60 (NO in step S11), the presence / absence determination means 12 does not output a detection signal to the alarm control means 13. Therefore, the alarm control means 13 does not output the alarm start signal to the alarm output unit 7. Further, the alarm control means 13 does not output an alarm stop signal for stopping the alarm to the alarm output unit 7 even when the alarm has already been output.

After that, the alarm control means 13 determines whether or not the alarm stop button G1 is pressed (step S13). Then, when it is determined that the alarm stop button G1 is pressed (YES in step S13), the alarm control means 13 outputs an alarm stop signal to the alarm output unit 7, and outputs an alarm from the alarm output unit 7. Stop (step S14). Further, when the image generation means 11 displays the virtual viewpoint image for peripheral monitoring based on the head height, the virtual viewpoint image for peripheral monitoring based on the head height is used as the virtual viewpoint image for peripheral monitoring based on the road surface height. Switch to the image and display it.

On the other hand, when it is determined that the alarm stop button G1 is not pressed (NO in step S13), the alarm control means 13 does not output the alarm stop signal to the alarm output unit 7. That is, the alarm output by the alarm control means 13 does not stop until the alarm stop button G1 is pressed even if the presence / absence determination means 12 subsequently determines that there is no person around the excavator 60. .. Further, when the image generation means 11 is displaying the virtual viewpoint image for peripheral monitoring based on the head height, the image generation means 11 continues to display the virtual viewpoint image for peripheral monitoring based on the head height.

The alarm control means 13 changes the content of the alarm when the presence / absence determination means 12 determines that there is no person around the excavator 60 even before the alarm stop button G1 is pressed. You may. Specifically, the alarm control means 13 may change the intensity, height, output interval, etc. of the alarm sound that has already been output, or the intensity, emission color, emission interval, etc. of the alarm lamp that has already been emitted. Etc. may be changed. This is so that the operator who receives the alarm can distinguish between the case where the presence / absence determination means 12 determines that a person exists and the case where it is determined that there is no person.

With the above configuration, the image generation device 100 outputs an alarm when it is determined that a person exists in the monitoring space, and does not stop the alarm until the operator indicates the intention to stop the alarm. That is, the image generation device 100 stops the alarm even when a person existing in the monitoring space exits the monitoring space or fails to detect a person existing in the monitoring space. There is no such thing. Therefore, the image generation device 100 can prompt the operator to confirm that the operator existing in the monitoring space around the excavator 60 has left the monitoring space.

Further, the image generation device 100 has a configuration in which the presence / absence determining means 12 determines the presence / absence of a worker (moving object) based on a moving object detection sensor, and stops an alarm when the worker is stationary in the monitoring space. It can be prevented from being lost. Further, in the image generation device 100, in a configuration in which the presence / absence determination means 12 determines the presence / absence of a worker (moving object) using an optical flow, the alarm is stopped when the worker is stationary in the monitoring space. It can be prevented from being stowed. The moving object detection sensor and the optical flow do not detect a stationary object, and do not distinguish between the worker standing still in the monitoring space and the worker leaving the monitoring space. Therefore, in the configuration in which the alarm is stopped when it is determined that no person exists, the alarm may be stopped even though the worker is present in the monitoring space. On the other hand, the image generation device 100 can call the attention of the operator of the excavator 60 by not stopping the alarm when there is a possibility that an operator is present in the monitoring space.

Next, with reference to FIGS. 22 to 25, a process in which the alarm control means 13 controls the alarm output unit 7 based on the determination result of the presence / absence determination means 12 and the determination result of the work machine state determination means 14 ( Hereinafter, “second alarm control process”) will be described. Note that FIG. 22 is a flowchart showing the flow of the second alarm control process, and FIG. 23 shows an example of the transition of the output image displayed during the second alarm control process. Further, FIG. 24 is a flowchart showing a flow of a process of determining the state of the work machine by the work machine state determination means 14 (hereinafter, referred to as “work machine state determination process”), and FIG. 25 is a work impossible state. It is a flowchart which shows the flow of the process which the alarm control means 13 controls the alarm output part 7 (hereinafter, is referred to as "the alarm control process at the start of work") at the time of switching to the workable state. The alarm control means 13 repeatedly executes the second alarm control process in a predetermined cycle, and the work machine state determination means 14 repeatedly executes the work machine state determination process in a predetermined cycle. Further, the alarm control means 13 executes the work start alarm control process when switching from the work impossible state to the workable state.

First, the presence / absence determination means 12 determines whether or not there is a person around the excavator 60 (step S21). At this time, when the display unit 5 is activated, the image generation means 11 generates and displays, for example, a virtual viewpoint image for peripheral monitoring based on the road surface height as shown in the output image D31 of FIG. 23. ..

When it is determined that a person exists around the excavator 60 (YES in step S21), the presence / absence determination means 12 refers to the result of the determination by the work machine state determination means 14 (see FIG. 24 described later) (step S22). ). In this case, the presence / absence determination means 12 may fix the gate lock lever 8 in the locked state. Specifically, the presence / absence determination means 12 may prevent the operator from pulling up the gate lock lever 8, or may prevent the gate lock lever 8 from being unlocked even if the gate lock lever 8 is pulled up. ..

When the work machine state determination means 14 determines that the excavator 60 is in a workable state (YES in step S22), the presence / absence determination means 12 outputs a detection signal to the alarm control means 13. Upon receiving the detection signal, the alarm control means 13 outputs an alarm start signal for outputting an alarm to the alarm output unit 7, and outputs an alarm from the alarm output unit 7 (step S23).

In this embodiment, the alarm output unit 7 outputs an alarm sound. Specifically, the human presence / absence determining means 12 sends a left side detection signal to the alarm control means 13 when it is determined that a person (here, workers P10 to P12) exists in the left side monitoring space ZL. Notice. Then, the alarm control means 13 outputs an alarm start signal to the left side alarm output unit 7L, the rear alarm output unit 7B, and the right side alarm output unit 7R, and outputs an alarm sound from all three alarm output units. Let me. In this case, the control unit 1 activates the display unit 5 when the display unit 5 is not activated. Then, as shown in the output image D22 of FIG. 21, the image generation means 11 switches the virtual viewpoint image for peripheral monitoring based on the road surface height to the virtual viewpoint image for peripheral monitoring based on the head height, and presses the alarm stop button G1. Overlapping display. Further, as shown in the output image D23 of FIG. 21, the image generation means 11 may superimpose and display the alarm stop button G1 on the peripheral monitoring virtual viewpoint image based on the road surface height. In this case, the alarm control means 13 may prohibit the work by the excavator 60 until the alarm stop button G1 is pressed. Specifically, the alarm control means 13 may close the gate lock valve and block the flow of hydraulic oil between the control valve and the operation lever or the like to invalidate the operation lever or the like.

Further, when the work machine state determination means 14 determines that the excavator 60 is not in the workable state (NO in step S22), the presence / absence determination means 12 does not output the detection signal to the alarm control means 13. The value of the person detection flag prepared in NVRAM or the like is set to "1" (on) (step S24). The person detection flag has a value "0" (off) set as an initial value, and the detection flag value "1" (on) indicates that a person has been detected, and the detection flag value "0" ( Off) indicates that no person has been detected. At this point, since the alarm control means 13 does not receive the detection signal, the alarm start signal is not output to the alarm output unit 7, and the alarm output unit 7 does not output the alarm. Further, when the alarm control means 13 has already output the alarm start signal to the alarm output unit 7, the alarm control means 13 may output the alarm stop signal to the alarm output unit 7. Further, the image generation means 11 is the same as the case where it is determined that there is no person even though it is determined that there is a person around the excavator 60 when the display unit 5 is activated. Display the output image. Specifically, the image generation means 11 displays, for example, a virtual viewpoint image for peripheral monitoring based on the road surface height as shown in the output image D32 of FIG. 23. However, the image generation means 11 may switch the peripheral monitoring virtual viewpoint image based on the road surface height to the peripheral monitoring virtual viewpoint image based on the head height.

On the other hand, when it is determined that there is no person around the excavator 60 (NO in step S21), the presence / absence determination means 12 does not refer to the result of the determination by the work machine state determination means 14, and the alarm control means 13 It does not output a detection signal to.

In this case, the image generation means 11 is the same as the case where it is determined that there is a person even though it is determined that there is no person around the excavator 60 when the alarm has already been output. Display the output image. Specifically, the image generation means 11 displays, for example, a virtual viewpoint image for peripheral monitoring based on the head height as shown in the output image D33 of FIG. 23 together with the alarm stop button G1. However, the image generation means 11 may switch the head height-based peripheral monitoring virtual viewpoint image to a road surface height-based peripheral monitoring virtual viewpoint image as shown in the output image D34 of FIG. 23.

After that, if the alarm has already been output, the alarm control means 13 monitors whether or not the alarm stop button G1 is pressed (step S25). Then, when it is determined that the alarm stop button G1 is pressed (YES in step S25), the alarm control means 13 outputs an alarm stop signal to the alarm output unit 7, and outputs an alarm from the alarm output unit 7. Stop (step S26). Further, when the image generation means 11 displays a virtual viewpoint image for peripheral monitoring based on the head height (see output image D33), the virtual viewpoint image for peripheral monitoring based on the head height is used as the road surface height. It is displayed by switching to the reference peripheral monitoring virtual viewpoint image (see output image D31).

On the other hand, when it is determined that the alarm stop button G1 is not pressed (NO in step S25), the alarm control means 13 does not output the alarm stop signal to the alarm output unit 7. That is, the alarm output by the alarm control means 13 does not stop until the alarm stop button G1 is pressed even if the presence / absence determination means 12 subsequently determines that there is no person around the excavator 60. .. Further, when the image generation means 11 displays a virtual viewpoint image for peripheral monitoring based on the head height (see output image D33), the image generation means 11 displays the virtual viewpoint image for peripheral monitoring based on the head height. Let it continue.

The alarm control means 13 changes the content of the alarm when the presence / absence determination means 12 determines that there is no person around the excavator 60 even before the alarm stop button G1 is pressed. You may. Specifically, the alarm control means 13 may change the intensity, height, output interval, etc. of the alarm sound that has already been output, or the intensity, emission color, emission interval, etc. of the alarm lamp that has already been emitted. Etc. may be changed. This is so that the operator who receives the alarm can distinguish between the case where the presence / absence determination means 12 determines that a person exists and the case where it is determined that there is no person.

Next, with reference to FIG. 24, the work machine state determination process by the work machine state determination means 14 will be described.

First, the work machine state determining means 14 determines whether or not the ignition switch 9 is in the ON state based on the output of the ignition switch 9 (step S31).

When it is determined that the ignition switch 9 is in the ON state (YES in step S31), the work machine state determining means 14 determines whether or not the gate lock lever 8 is in the unlocked state based on the output of the gate lock lever 8. Determine (step S32).

When it is determined that the gate lock lever 8 is in the unlocked state (YES in step S32), the work machine state determining means 14 determines that the excavator 60 is in the workable state (step S33).

On the other hand, when it is determined that the ignition switch 9 is not in the ON state (NO in step S31), or when it is determined that the gate lock lever 8 is not in the unlocked state (NO in step S32), the work machine state determining means 14 Determines that the excavator 60 is not in the workable state, that is, the excavator 60 is in the non-workable state (step S34).

After that, the work machine state determination means 14 sets the value of the work enable flag prepared in the NVRAM or the like according to the determination result. The presence / absence determination means 12 refers to the value of the workable flag in step S22 of the second alarm control process shown in FIG. 22, and determines the presence / absence of an alarm output based on the referenced value. The workable flag has a value "0" (off) set as an initial value, and the workable flag value "1" (on) indicates that work is possible, and the workable flag value "0". "(Off) indicates that work is not possible.

The work machine state determining means 14 may determine whether or not the excavator 60 is in a workable state based on only one of the states of the gate lock lever 8 and the ignition switch 9.

Next, with reference to FIG. 25, the work start alarm control process by the alarm control means 13 will be described.

First, the alarm control means 13 refers to the value of the person detection flag (step S41). Then, when it is determined that the value of the person detection flag is "1" on (YES in step S41), the alarm control means 13 outputs an alarm start signal to the alarm output unit 7, and the alarm output unit 7 outputs the alarm start signal. An alarm is output (step S42). Specifically, the alarm control means 13 outputs an alarm even when it is determined by the presence / absence determination means 12 that there is no person around the excavator 60 at the present time. In this embodiment, the alarm control means 13 outputs an alarm start signal to the left side alarm output unit 7L, the rear alarm output unit 7B, and the right side alarm output unit 7R, and gives an alarm from all three alarm output units. Output sound. In this case, the alarm control means 13 may prohibit the work by the excavator 60 until the output of the alarm is stopped. Specifically, the alarm control means 13 may close the gate lock valve and block the flow of hydraulic oil between the control valve and the operation lever or the like to invalidate the operation lever or the like.

After that, the alarm control means 13 measures the elapsed time after the start of the alarm, and determines whether or not a predetermined time (for example, 2 seconds) has elapsed after the start of the alarm (step S43).

When it is determined that the predetermined time has not elapsed after the start of the alarm (NO in step S43), the alarm control means 13 waits until the predetermined time elapses.

On the other hand, when it is determined that a predetermined time has elapsed after the start of the alarm (YES in step S43), the alarm control means 13 outputs an alarm stop signal to the alarm output unit 7, and outputs an alarm from the alarm output unit 7. Is stopped (step S44). Further, the alarm control means 13 resets the value of the person detection flag to “0” (off) (step S45). The alarm control means 13 may not stop the alarm until the alarm stop button G1 is pressed.

With the above configuration, the image generation device 100 does not output an alarm when the excavator 60 is in an inoperable state even when it is determined that a person exists in the monitoring space. Therefore, the image generation device 100 can prevent an unnecessary alarm from being output when the excavator 60 is in an inoperable state.

On the other hand, even if the image generator 100 determines that there is no person in the monitoring space at the present time, the worker has entered the monitoring range while the excavator 60 is in the inoperable state. Outputs an alarm to. Therefore, the image generation device 100 can prompt the operator to confirm that the operator existing in the monitoring space around the excavator 60 has left the monitoring space.

Further, in the above-mentioned work start alarm control process, the alarm control means 13 outputs an alarm as long as the value of the person detection flag is "1" (on). However, the present invention is not limited to this configuration. For example, the alarm control means 13 may not output the alarm if the elapsed time from the time when the value of the person detection flag is set to "1" (on) is equal to or longer than a predetermined time. Alternatively, the person presence / absence determination means 12 sets the value of the person detection flag to “0” (off) when the elapsed time from the time when the value of the person detection flag is set to “1” (on) reaches a predetermined time. You may reset it. This is to prevent an alarm from being output based on a detection record that is too old.

Further, when the presence / absence determination means 12 determines the presence / absence of an operator (moving object) based on the moving object detection sensor, the image generation device 100 enters the monitoring space and stands still when the excavator 60 is in an inoperable state. It is possible to prevent the operator of the excavator 60 from failing to detect when the excavator 60 becomes workable and to start the work by the excavator 60 without noticing the operator.

Further, when the presence / absence determination means 12 determines the presence / absence of a worker (moving object) using an optical flow, the image generation device 100 enters the monitoring space and stands still when the excavator 60 is in an inoperable state. It is possible to prevent the operator of the excavator 60 from failing to detect when the excavator 60 becomes workable and to start the work by the excavator 60 without noticing the operator.

The moving object detection sensor and the optical flow do not detect a stationary object, and do not distinguish between the worker standing still in the monitoring space and the worker leaving the monitoring space. Therefore, in a configuration in which an alarm is not output in any case when it is determined that there is no person in the monitoring space at the present time, the work of entering the monitoring space and standing still before the excavator 60 becomes ready for work. There is a risk that the work of the excavator 60 will be allowed even though there is a person. On the other hand, the image generation device 100 can call the attention of the operator of the excavator 60 by outputting an alarm when there is a possibility that an operator is present in the monitoring space.

Further, in the above-mentioned second alarm control process, the image generation device 100 causes the display unit 5 to display the output image even when the excavator 60 is in the inoperable state, but prevents the display unit 5 from displaying the output image. You may. This is because when the excavator 60 is in an inoperable state, there is no possibility that the excavator 60 and the operator come into contact with each other, and the operator does not need to monitor the surroundings of the excavator 60 through the output image.

Although the preferred examples of the present invention have been described in detail above, the present invention is not limited to the above-mentioned examples, and various modifications and substitutions are made to the above-mentioned examples without departing from the scope of the present invention. Can be added.

For example, in the above-described embodiment, the image generation device 100 adopts a cylindrical space model MD as a space model, but a space model having another columnar shape such as a polygonal column may be adopted, and the bottom surface and the space model may be adopted. A space model composed of two side surfaces may be adopted, or a space model having only side surfaces may be adopted.

Further, the image generation device 100 is mounted on a self-propelled excavator equipped with movable members such as a bucket, an arm, a boom, and a swivel mechanism together with a camera and a human detection sensor. Then, the image generation device 100 constitutes an operation support system that supports the movement of the excavator and the operation of the movable members while presenting the surrounding image to the operator. However, the image generation device 100 may be mounted together with a camera and a person detection sensor on a work machine that does not have a swivel mechanism such as a forklift or an asphalt finisher. Alternatively, the image generator 100 may be mounted together with a camera and a human detection sensor on a work machine having a movable member but not self-propelled, such as an industrial machine or a fixed crane. Then, the image generation device 100 may configure an operation support system that supports the operation of these work machines.

Further, although the peripheral monitoring device has been described by taking the image generation device 100 including the camera 2 and the display unit 5 as an example, the peripheral monitoring device may be configured as a device that does not include the image display function by the camera 2, the display unit 5, and the like. For example, as shown in FIG. 26, the peripheral monitoring device 100A as a device that executes the first alarm control process includes a camera 2, a storage unit 4, a display unit 5, a gate lock lever 8, an ignition switch 9, and coordinate mapping means. 10. The image generating means 11 and the working machine state determining means 14 may be omitted. Further, as shown in FIG. 27, the peripheral monitoring device 100B as a device for executing the second voice transmission support process and the work start alarm control process includes a camera 2, an input unit 3, a storage unit 4, a display unit 5, and coordinates. The associating means 10 and the image generating means 11 may be omitted.

1 ... Control unit 2 ... Camera 2L ... Left side camera 2R ... Right side camera 2B ... Rear camera 3 ... Input unit 4 ... Storage unit 5 ... Display unit 6 ...・ Person detection sensor 6L ・ ・ ・ Left side person detection sensor 6R ・ ・ ・ Right side person detection sensor 6B ・ ・ ・ Rear person detection sensor 7 ・ ・ ・ Alarm output unit 7L ・ ・ ・ Left side alarm output unit 7B ・ ・ ・Rear alarm output unit 7R ・ ・ ・ Right side alarm output unit 8 ・ ・ ・ Gate lock lever 9 ・ ・ ・ Ignition switch 10 ・ ・ ・ Coordinate mapping means 11 ・ ・ ・ Image generation means 12 ・ ・ ・ Presence / absence judgment means 13・ ・ ・ Alarm control means 14 ・ ・ ・ Work machine state determination means 40 ・ ・ ・ Input image / spatial model correspondence map 41 ・ ・ ・ Spatial model / processing target image correspondence map 42 ・ ・ ・ Processing target image / output image correspondence map 60 ・ ・ ・ Excavator 61 ・ ・ ・ Lower traveling body 62 ・ ・ ・ Swivel mechanism 63 ・ ・ ・ Upper swivel body 64 ・ ・ ・ Cab 100 ・ ・ ・ Image generator 100A, 100B ・ ・ ・ Peripheral monitoring device

Claims (8)

With the lower running body,
An upper swivel body rotatably arranged on the lower traveling body and
An excavator that has a peripheral monitoring device that monitors the surroundings.
In a state capable of operating the excavator, have a function of detecting an object before the operation is started, until outputs an alarm when it detects an object, the operation to stop the alarm was output is performed, the object Keep the excavator inoperable regardless of whether it is detected or not.
Excavator. It has a function of detecting an object around the excavator in a state where the excavator cannot be operated.
The excavator according to claim 1. With the lower running body,
An upper swivel body rotatably arranged on the lower traveling body and
An excavator that has a peripheral monitoring device that monitors the surroundings.
It has a function of detecting an object in the surrounding of the shovel in a state that can not operate the excavator, when detecting an object, even as the gate lock lever is operated so as to be available to interact with the shovel, the shovel Keep inoperable,
Excavator. It has a function of detecting an object around the excavator while the excavator can be operated.
The excavator according to claim 3. It has an alarm output unit that outputs an alarm to the operator.
When an object exists around the excavator, an alarm is output by the alarm output unit.
The excavator according to any one of claims 1 to 4. The timing at which an alarm is output differs depending on whether or not the excavator can be operated when an object is present around the excavator.
The excavator according to claim 5. It is determined whether or not the excavator can be operated based on the state of at least one of the gate lock lever and the ignition switch.
The excavator according to any one of claims 1 to 6. It is determined whether or not the excavator can be operated based on the output of the device that switches the state of the excavator.
The excavator according to any one of claims 1 to 6.

Images