JP3139959B2 - Work machine operation support image system - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an operation support image system for a working machine, and more particularly to an image system for assisting an operator of a working machine to confirm surrounding safety.

[0002]

2. Description of the Related Art Conventionally, when operating a construction work machine (hereinafter simply referred to as a work machine) such as a hydraulic excavator shown in FIG. Because it is difficult to confirm the surroundings, the safety of the surroundings must be confirmed using a mirror such as a rearview mirror or a video or image (hereinafter simply referred to as an image) of a camera mounted outward on the working machine. I have. In the confirmation method using a camera image, since the field of view of an image is narrow only with a camera having a fixed shooting direction, the field of view of the image is widened using a camera with a variable shooting direction.

[0003]

However, the conventional method of confirming the surroundings using a mirror has a problem that a blind spot can be formed. In addition, the safety confirmation method using a camera with a variable shooting direction is complicated because the operator has to operate both the work machine and adjust the shooting direction of the camera at the same time. Therefore, there is a problem that a time delay occurs in the safety check and instantaneous judgment becomes difficult.

[0004] In order to solve these problems, the present inventors have focused on fisheye lenses. If a fisheye lens is used, an image with a wide field of view can be obtained without changing the shooting direction of the camera. However fisheye lens has a large barrel distortion, the image is wide-angle image I _a was distorted as shown in FIG. 5 (A) e.g.
Therefore, there is a problem that a large distortion occurs at the peripheral portion of the image and the image becomes difficult to see. Figure 5 shows a digital image processing method for correcting the distortion of the wide-angle image I _a. The image processing method, Figure 5 and digitizes wide-angle image I _a shown in (A) a plurality of compartments I
By then partitioned into _bi, fisheye lens to store the unique aberrations per section I _bi, it corrected based on the distortion of one or more compartments I _bi selected by some kind of method to the aberration of each compartment 5 The distortion-free section _Igi shown in (B) is obtained. FIG.
An example in which nine sections are simultaneously corrected to distortion-free sections will be described. Using this image processing method, it is possible to use a wide-angle image I _a fisheye lens around safety check during operation of the work machine.

An object of the present invention is to provide an operation support image system for a working machine using a fisheye lens.

[0006]

Referring to FIGS. 1 to 3, an operation support image system for a working machine according to the present invention is mounted on a working machine 3 (see FIG. 3A) outward in a predetermined direction. An image display device 16 having a display surface 15 provided at a predetermined position O in an operation room 5 of the working machine 3;
An eye position measuring device 18 attached to the head of the operator 2 facing the display surface 15 in the operation room 5 and measuring the position M of the eyes of the operator 2, at a position facing the eyes of the head of the operator 2. A line-of-sight detection device 20 that is attached and detects a line-of-sight direction I by forming a gazing point corresponding to the direction of the eye of the operator 2, is connected to the eye position measurement device 18 and the line-of-sight detection device 20, and is provided on the display surface 15. Attention point detection means 22 for calculating the position of attention point P where the line of sight of operator 2 intersects display surface 15 from position O, eye position M of operator 2 and line-of-sight direction I, imager 6 and image display device 16
And output wide-angle image I _a (FIG. 2 (A) see) one section there is a point of interest P with a split into a plurality of sections I _b1 ~I _b5 of being connected to the target-point detecting means 22 imaging device 6 (e.g., I _b1 ) _Is converted into a distortion-free section (for example, I _g1 ), and the remaining sections (for example, I _b2 to
Image processing device 13 for outputting to image display device 16 together with I _b5 )
It is provided with.

[0007]

1 to 3 show an embodiment in which the working machine 3 is a hydraulic shovel. However, the present invention is not limited to application to a hydraulic excavator. Referring to FIGS. 1-3, the present invention is a working machine 3 attached a fisheye lens 7 with the imaging device 6 in a predetermined direction outwardly on an image processing apparatus provided with the output angle image I _a, for example, in the operation chamber 15 Enter in 13. The image processing apparatus 13 shown in FIG. 1 converts the input wide-angle image _Ia into a plurality of sections I
_bi sectioning means 10; a selecting means 11 for selecting a specific section (for example, I _b1 ); and a selected one section (for example, I _b1 ) having no distortion (for example, I _b1 ).
_g1 ). If the wide-angle image _Ia is not _a digital image, A
A D converter is provided to perform digital conversion. 2 (A) is, the image processing apparatus classification means 10 wide-angle image I _a 5 compartments I _b1 of 13 ~I _b5
Indicates that the classification is made. However, the wide-angle image in the present invention
The method of dividing _Ia and the number of sections are not limited to the example of FIG.

An eye position measuring device 18 for measuring the position M of the eye and a visual line detecting device 20 for detecting the visual line direction I are mounted on the head of the operator 2. Eye position measuring device 18 of FIG.
Are located on the side and top of the helmet 24 at position sensors 19a and 19b, respectively, and on the helmet 24 at a position facing the front of the operator 2's eyes and in a fixed relative relationship to the position sensors 19a and 19b. It is formed by the attached visual line detection panel 24. Each of the position sensors 19a and 19b measures a three-dimensional position in a predetermined coordinate system in the operation room 5 of the work machine 3. The gaze detection panel 21 forms a gaze point corresponding to the gaze direction I of the operator 2 on the panel 21 by infrared rays, detects the gaze point, and further detects a deviation between a reference point on the panel 21 and the gaze point. Is output. The reference point of the line-of-sight detection panel 21 is maintained in a fixed relative relationship with the position sensors 19a and 19b. The operator 2 wears the helmet 24 horizontally so that, for example, the reference point on the line-of-sight detection panel 21 is at a fixed distance D in front of the eye (see FIG. 1). The eye position measuring device 18 shown in FIG. 1 obtains a reference point position of the line-of-sight detection panel 21 from the measured positions of the position sensors 19a and 19b and the above-mentioned relative relationship, and obtains an eye based on a certain distance D from the reference point position to the eye. Is measured. An optical system for measuring the opposing distance may be provided between the line-of-sight detection panel 21 and the eye, and the position M of the eye may be measured from the opposing distance. The gaze detection device 20 obtains the position of the gazing point on the panel 21 based on the reference point position of the gaze detection panel 21 and the deviation from the reference point obtained by the method described above, and determines the gazing point position and the eye position. The gaze direction I is detected from the position M.

The eye position measuring device 18 and the line of sight detecting device 20 are connected to the point of interest detecting means 22. The attention point detecting means 22 has a memory 23 for storing a predetermined position O of the display surface 15 in the operation room 5, and passes through the eye position M according to the input of the eye position M and the line-of-sight direction I of the operator 2. In addition, a straight line having the line-of-sight direction I is obtained, the position of the point of interest P is calculated as an intersection between the line and the display surface 15 at the predetermined position O, and the calculated position of the point of interest P is output to the selecting means 11 of the image processing device 13. I do. One example of the attention point detecting means 22 is a computer in which necessary programs are built.

The selecting means 11 of the image processing device 13 is a sorting means
5 sections I divided by 10_b1~ I _b5Attention points out of
Select one section including the position of P, and convert the one section
It is sent to stage 12 and converted to a distortion-free section. Fig. 2 (B) shows five sections I
_b1~ I_b5Middle section I_b1Indicates that there is an attention point P
And the section I_b1The distortion-free section I_g1Is shown. Strange
Non-distorted section I after replacement_g1Is the other partition I without conversion_b2~ I_b5With
It is displayed on the display surface 15. An example of the conversion means 12 is shown in FIG.
Image processing hardware capable of performing image correction processing of the prior art
Or software. Middle parcel I_b1Small distortion
In this case, the conversion by the conversion means 12 is omitted, and
Block I_b1~ I_b5May be displayed on the display surface 15 as it is. Figure
2 (C) is section I_b2The display surface 15 when there is an attention point P
And the conversion means 12_b2The distortion-free section I_g2Converted to
FIG. 2 (D) shows section I_b3If there is an attention point P
On the display surface 15 of the display.

As shown in FIG. 2, according to the image system of the present invention, the section I _bi having the point of interest P is converted into the distortion-free section I _gi and displayed according to the movement of the point of interest P on the display surface 15. Thus, it is possible to provide a wide-angle safety confirmation image in which the periphery of the point of interest P is easy to see. In addition, since the distortion-free section is switched, the operator 2 only needs to move his or her line of sight and does not need any special operation, so that safety confirmation in all directions can be quickly performed. FIG. 2 shows a case where only one section I _bi is converted into a distortion-free section I _gi , respectively.
It can also be converted together a plurality of compartments I _bi adjacent the _bi-free strain compartment I _gi.

Thus, it is possible to provide the "working machine operation support image system using a fisheye lens" which is the object of the present invention.

[0013]

FIG. 3 shows different radii from the center position of the imager 6.
Fish-eye radially outward at multiple angular positions
An embodiment in which a lens 7 is provided is shown. That is, the imaging device 6 of FIG.
Are three in the radial direction at 90 ° intervals from the center position of the imaging device 6.
Fisheye lens 7_R, 7_B, 7_L, Facing the front of the work machine 3
Against fisheye lens 7_B, 7_RAnd 7_LEach facing backward, right
Place the imaging device 6 on the work machine 3 so that
I have. Fisheye lens 7_BAnd 7_RPartially overlap the field of view of
Also fisheye lens 7_BAnd 7_LBy partially overlapping the field of view
To obtain a wide-field image without blind spots by using three wide-angle images
be able to. FIG. 4 shows three fisheye lenses 7_L, 7_B, 7_RTo
Three display surfaces 15 for displaying wide-angle images through _L, 15_B, 15_RTo
1 shows a display device 16 having the same. In this case, the attention inspection of FIG.
Each display surface 15 is stored in the memory 23 of the output means 22._L, 15_B, 15_RPredetermined position
Of the operator 2 and each display surface 15_L, 1
Five_B, 15_RBy detecting the intersection with
Of the display surface and the point of interest P on the display surface
be able to. For example, the display device 16 shown in FIG.
If it is set, the operator 2 can use any display surface as necessary.
Fifteen_L, 15_B, 15_RLook sideways and side by side
Can see the distortion-free image in any direction in real time.
Wear.

The switching of the distortion-free section I _gi in accordance with the movement of the position of the point of interest P of the operator 2 has been described above. A manual coordinate setting machine (not shown) is connected to the point of interest detection means 22 in FIG. Then, by inputting the output of the manual coordinate setting device as the output of the point of interest detection means 22 to the image conversion means 13, the position of the distortion-free section I _gi on the display surface 15 can also be moved manually. The operation support image system of the present invention can also be used when remotely controlling the work machine 3.

[0015]

As described above, the operation support image system for a working machine according to the present invention divides an output wide-angle image of an imaging machine with a fish-eye lens attached to a working machine outward in a predetermined direction into a plurality of sections, Since the position of the point of interest on the display surface of the display device is detected and one section having the point of interest is converted into a distortion-free section and displayed on the display surface together with the remaining sections, the following remarkable effects are achieved.

(A) Since the undistorted section of the section where the point of interest is located is displayed without time delay in accordance with the movement of the point of interest on the display surface, a wide-angle safety confirmation image in which the periphery of the point of interest is easy to see is displayed. Can be provided. (B) A plurality of wide-angle lenses in different directions can be provided so that the fields of view partially overlap, and the occurrence of blind spots can be avoided. (C) Compared to the method of moving the imaging device, there is no rotating part or the like, so that there is less possibility of occurrence of a failure.

[Brief description of the drawings]

FIG. 1 is a block diagram of one embodiment of the present invention.

FIG. 2 is an explanatory diagram of a display surface of the present invention.

FIG. 3 is an explanatory diagram of an example of a work machine and an imaging device.

FIG. 4 is an explanatory diagram of an example of a display device.

FIG. 5 is an explanatory diagram of a conventional image correction processing method for a wide-angle image.

[Explanation of symbols]

2 ... operator 3 ... work machine 5 ... operating room 6 ... imaging machine 7 ... fisheye lens 10 ... dividing means 11 ... selecting means 12 ... conversion means 13 ... image processing device 15 ... display surface 16 ... image display device 18 ... eye position measurement Apparatus 19 Position sensor 20 Line-of-sight direction detecting apparatus 21 Line-of-sight detecting panel 22 Point-of-interest detection means 23 Memory 24 Helmet _Ia Wide-angle image _Ibi Section _Igi Section without distortion.

Claims (3)

(57) [Claims] 1. An image pickup apparatus with a fish-eye lens mounted outward on a work machine in a predetermined direction, an image display device having a display surface provided at a predetermined position in an operation room of the work machine, and facing a display surface in the operation room. An eye position measurement device attached to the operator's head for measuring the position of the operator's eyes, a gaze point attached to a position facing the eyes of the operator's head and corresponding to the direction of the eyes of the operator; A line-of-sight detection device that detects the line-of-sight direction by forming the line-of-sight line from the predetermined position of the display surface and the position and line-of-sight direction of the operator's eyes connected to the line-of-sight measuring device and the line-of-sight detection device Point-of-interest detection means for calculating the position of a point of interest that intersects with the image-capturing device, the image display device, and the point-of-interest detection means. There Work machine operation support imaging system comprising an image processing apparatus for output to the image display apparatus together with the remaining compartments on converted into unstrained compartment a section. 2. An imaging system according to claim 1, wherein said position sensors are respectively attached to the side and top of the helmet, and are located on said helmet, facing the front of the operator's eye and fixed to the position of said position sensor. The eye position measuring device is formed by a line-of-sight detection panel attached to a position in a relative relationship, a reference point on the line-of-sight detection panel is kept in the relative relationship to the position sensor, and a reference point on the line-of-sight detection panel is The operator wears the helmet so as to be at a fixed distance in front of the eye, and the position of the reference point on the line-of-sight detection panel in the line-of-sight measurement device and the deviation of the point of fixation formed on the panel from the reference point An operation support image system for a working machine, wherein the gaze direction is detected from the position of the eye and the position of the eye. 3. The image system according to claim 1, wherein
A fisheye lens is provided radially outward at a plurality of angular positions in different radial directions from the center position of the imaging device, and a wide-angle image of a field of view partially overlapped with a field of view of an adjacent fisheye lens via each fisheye lens is taken, and the image is taken. A display device is provided with a plurality of display surfaces for image display via each fish-eye lens, and the point-of-interest detection means detects a display surface that intersects the operator's line of sight and calculates the position of the point of interest on the display surface. Operation support image system for work machines.