JP7657176B2 - Display device and display program - Google Patents

Description

The present invention relates to a display device and a display program.

2. Description of the Related Art In order to improve the workability of an operator of a work machine such as a crane, technology is being developed that captures images of the work machine and its surroundings with a camera and displays the images on a display screen located near the operator, for example.
Patent Literature 1 also discloses a technique in which a monitor is equipped with a photographing device, and a map including a monitored area and a forward area of each monitor stationed in the monitored area are displayed on a display screen. Patent Literature 1 also discloses a technique in which a monitor can be selected, and an image captured by the photographing device worn by the selected monitor is displayed small on, for example, the upper right part of the display screen.

JP 2018-137514 A

If one considers applying the above-mentioned technology described in Patent Document 1 to a technology for photographing a work machine and its surroundings with a camera and displaying the image on a display screen, the image captured by the camera would be displayed, for example, in the upper right corner of the display screen.
In that case, however, the images of the work machine and its surroundings captured by the camera will be displayed small, making them difficult for the operator to see and resulting in poor visibility.

The present invention has been made in consideration of the above points, and aims to provide a display device and a display program that can display a work machine and its surroundings on a display screen with good visibility.

According to one aspect of the display device of the present invention,
A display device that displays an overhead view of the surroundings of a work machine and information indicating the positions of image capture devices present around the work machine,
an overhead view display area for displaying the overhead view;
a photographed image display area, separate from the overhead view display area, for displaying a photographed image photographed by the photographing device;
an acquisition unit that acquires a display range to be enlarged and displayed in the captured image;
Equipped with
The display range acquired by the acquisition unit is enlarged and displayed in the captured image display area.

According to another aspect of the display device of the present invention,
The display program causes the display device to execute the above display.

The present invention makes it possible to display the work machine and its surroundings on the display screen with good visibility.

FIG. 2 is a side view illustrating a configuration example of a crane. FIG. 2 is a block diagram showing a configuration of the imaging device. FIG. 1 is a block diagram illustrating a configuration of a display device. FIG. 11 is a diagram illustrating an example of a display method on a display screen of a display device. FIG. 13 is a diagram illustrating an example of a display on a display screen when there are multiple workers. 1A is a diagram showing a state in which a display range is specified on a captured image display area, and FIG. 1B is a diagram showing a state in which the display range is enlarged and displayed. FIG. 13 is a diagram illustrating a method for enlarging a display range. 11 is a diagram showing a photographing range corresponding to a display range displayed in a photographed image display area. FIG. 13A and 13B are diagrams for explaining a method of displaying a shooting range corresponding to a display range on an overhead view display area. FIG. 13 is a diagram showing a point of interest set on the bird's-eye view display area. FIG. 1A is a diagram showing a state in which the point of interest is not within the shooting range corresponding to the display range, and FIG. 1B is a diagram showing a state in which the shooting range has been changed so that the point of interest is within the shooting range corresponding to the display range. 13A and 13B are diagrams illustrating a method for changing a shooting range corresponding to a display range so that a point of interest approaches the center of the display range.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of a display device and a display program according to the present invention will be described with reference to the accompanying drawings.
In the following, the working machine is a crane, and a crawler crane is used as an example of a crane, but the present invention is not limited to this and can be applied to any type of cranes, including other mobile cranes such as wheel cranes, truck cranes, rough terrain cranes, and all-terrain cranes, as well as tower cranes, overhead cranes, jib cranes, retractable cranes, stacker cranes, gantry cranes, and unloaders. The present invention is also applicable not only to cranes equipped with a load hook, but also to cranes that suspend attachments such as magnets and earth drill buckets. Of course, the present invention is not limited to cranes, and can also be applied to working machines such as hydraulic excavators and forklifts.

As shown in FIG. 1, the crane 1 is configured to include a self-propelled crawler-type lower traveling body 2 and an upper rotating body 3 rotatably mounted on the lower traveling body 2.
A boom 4 is attached to the front side of the upper rotating body 3 so as to be capable of being raised and lowered. A counterweight 5 is attached to the rear part of the upper rotating body 3 to balance the weight of the boom 4 and the suspended load.

The boom 4 is raised and lowered by a boom winch (not shown) which winds up and unwinds a boom rope 6. A hoisting rope 8 having a hook 7 at its tip is wound up and unwinded by a hoisting winch (not shown).
A cab 9 is disposed on the right front side of the upper rotating body 3. The cab 9 is a control seat where an operator (not shown) sits and operates the crane 1.

The cab 9 is equipped with a display device 20, which will be described in detail later.
In this embodiment, the display device 20 is provided in the cab 9 of the crane 1, but it may be a computer used for monitoring by a site manager or the like, or may be located in a management center or the like away from the site. It may be a mobile terminal such as a tablet carried by a worker or manager at the site, or, in the case of remote operation, it may be a monitor in a remote operation room.

Around the crane 1, a worker A photographs the crane 1 etc. with a camera device 10 carried by the worker A. As will be described later, there may be a plurality of workers A who photograph the crane 1 etc.
The photographing device 10 may be a camera, or may be a smartphone or a portable information terminal having a camera function.

In this embodiment, as shown in FIG. 2, the photographing device 10 includes a control unit 11, a photographing unit 12, a GNSS (Global Navigation Satellite System) receiving unit 13, a sensor unit 14, and a communication unit 15.
In addition, other components such as memory may be provided as necessary.

The control unit 11 includes a CPU (Central Processing Unit) and controls the operation of each part of the image capturing device 10 according to a control program.
The image capturing unit 12 has a camera function and can capture moving images. The image capturing unit 12 transmits image data of the moving images to the display device 20 via the communication unit 15 together with information such as the angle of view when the captured image was captured.

The GNSS receiving unit 13 is equipped with a receiver that receives radio waves transmitted from GPS (Global Positioning System) satellites, etc., and determines its own position, i.e., the position (latitude, longitude) of the imaging device 10, based on the information superimposed on the received radio waves.
The sensor unit 14 includes, for example, a gyro sensor and a geomagnetic sensor, and transmits data measured by the sensors to the control unit 11 .

The control unit 11 calculates its own orientation, i.e., the orientation (direction) of the imaging device 10, based on the transmitted data.
The control unit 11 then transmits information on the position and orientation of the image capturing device 10 to the display device 20 via the communication unit 15 .

Next, the display device 20 according to the present embodiment will be described.
As shown in Figure 3, the display device 20 includes a control unit 21 consisting of a CPU etc., a communication unit 22, a GNSS receiving unit 23, a sensor unit 24 equipped with a gyro sensor etc., an input unit 25, a display unit 26 having a display screen 30, and a memory 27.

The control unit 11 is configured by a computer equipped with a CPU and the like, and controls the operation of each part of the display device 20 according to a control program.
The control unit 11 also includes a display program, and controls the display unit 26 to display on a display screen 30, which will be described below, in accordance with the display program.

When the communication unit 22 receives information on the position and orientation of the image capturing device 10 from the image capturing device 10 , the communication unit 22 stores the information in the memory 27 .
The GNSS receiving unit 23 determines its own position, that is, the position (latitude, longitude) of the display device 20 based on the information superimposed on the received radio waves.
The sensor unit 24 transmits data measured by the gyro sensor and the like to the control unit 21. The control unit 21 then calculates its own orientation, i.e., the orientation (direction) of the display device 20, based on the transmitted data, similarly to the control unit 11 of the imaging device 10.

In this embodiment, the input unit 25 is configured as a touch panel, and an operator (user) can touch the display screen 30 of the display device 20 to input instructions to the display device 20. The input unit 25 may be one that inputs instructions by cursor operation or the like.
The display unit 26 displays information on a display screen 30 under the control of the control unit 21. The display screen 30 is formed of, for example, a liquid crystal display.

The method of displaying on the display screen 30 of the display device 20 and the method of processing in the control unit 21 will be specifically described below.
Fig. 4 is a diagram showing an example of how to display on the display screen of a display device. In Fig. 4, the crane 1 (work machine) is depicted in a simplified form as if it were a combination of rectangular parallelepipeds.

The display screen 30 of the display device 20 includes an overhead view display area 31 and a captured image display area 32 .
In this embodiment, the upper right corner of the overhead view display area 31 is set as a captured image display area 32 separate from the overhead view display area 31. In this manner, the captured image display area 32 has a smaller area displayed on the display screen 30 than the overhead view display area 31.

In the overhead view display area 31, an overhead view of the surroundings of the crane 1 (work machine) is displayed.
In this case, the overhead view may be, for example, a photographed image of the surroundings of the crane 1 (work machine) from above by attaching a camera to the hook 7 (see FIG. 1) of the crane 1. Also, the overhead view may be, for example, an around-view image composed of photographed images taken by a plurality of cameras attached to the crane 1 for photographing the surroundings.

As shown in FIG. 4, the overhead view also includes symbols such as squares, circles, and dots representing the crane 1 (working machine), worker A, and surrounding objects.
Furthermore, the bird's-eye view also includes a combination of an actually photographed image and symbols such as squares, circles, and dots.

For example, in the overhead view display area 31 , the crane 1 is displayed near the center, and the forward direction of the crane 1 faces upward on the display screen 30 .
The bird's-eye view display area 31 also displays a worker A who is present around the crane 1. The position of the worker A is also the position of the image capturing device 10.

In this case, as mentioned above, the image of the actual worker A captured in the image actually taken of the crane 1 as described above may be displayed as is in the overhead view display area 31.
Furthermore, as shown in Figure 4, when displaying the crane 1 and worker A in graphical form, it is possible to configure the position and orientation of the aforementioned camera 10 to be displayed as the position and orientation of worker A.

In this case, the control unit 21 of the display device 20 calculates the relative position and orientation between the crane 1 and the camera device 10 based on the position and orientation of the camera device 10 transmitted from the camera device 10 and the position and orientation of the display device 20.
Then, as described above, when the overhead view display area 31 is displayed so that the forward direction of the crane 1 faces upward on the display screen 30, it is calculated where the photographing device 10 is located on the overhead view display area 31 and in which direction it is facing.

As another method for calculating the relative position and orientation of the crane 1 and the camera device 10, it is possible to analyze images captured by multiple cameras capturing images of the area around the crane 1 to obtain the position and facial orientation of photographer A, and then calculate the position of the camera device 10 on the overhead view display area 31 and the direction in which it is facing.

In the example of Figure 4, the position of the imaging device 10 calculated by the control unit 21 is displayed as a circle, and an illustration of the face of worker A is displayed within the circle so that worker A is facing the direction of the imaging device 10.
By displaying in this manner, the worker A around the crane 1 is displayed in a pseudo manner.

In this way, the overhead view display area 31 is configured to display information indicating the positions of the camera devices 10 present around the crane 1 (work machine). Note that the information indicating the position of the camera devices 10 does not necessarily have to indicate both the position of the camera devices 10 and the method of photographing them, but may indicate either one of them. For example, it may indicate the direction in which the camera devices 10 located outside the overhead view are positioned relative to the work machine. Furthermore, the information indicating the position of the camera devices 10 may be displayed in an area other than the overhead view display area 31, as long as the positional relationship with the work machine is known.
4, the shooting range R of the image capturing device 10 is displayed as an isosceles triangle (hereinafter simply referred to as a triangle) on the overhead view. The shooting range R of the image capturing device 10 may be displayed as a sector shape or the like.

In the example of FIG. 4, the center line of the triangle representing the imaging range R (see the dashed line in the figure) represents the imaging direction of the imaging device 10 and coincides with the direction of the face of worker A.
Moreover, the photographing range R (i.e., the length of the base of the triangle) is calculated based on the information on the angle of view transmitted from the photographing device 10 as described above.

That is, if the angle of view is large, the length of the base of the triangle is displayed as being long, and if the angle of view is small, the length of the base of the triangle is displayed as being short.
In this way, by configuring the shooting range R of the photographing device 10 to be displayed on the overhead view display area 31, a person who sees it (an operator) can easily intuitively grasp not only the shooting direction of the photographing device 10 but also what range is being photographed.

As shown in Figure 5, when there are multiple camera devices ₁₀₁ , ₁₀₂ , ... around the crane 1 (i.e., when workers _A1 , _A2 , ... are present), the position, shooting direction and shooting range R for each camera device ₁₀₁ , ₁₀₂ , ... are calculated and displayed.
In addition, when displaying actual photographed images of the crane 1 and worker A as an overhead view, it is also possible to configure the system to calculate and display the photographing direction and photographing range R of the photographing device 10 using a triangle or the like in the same manner as described above.

In the photographed image display area 32, a photographed image photographed by the photographing device 10 is displayed.
5, in this embodiment, when there are a plurality of photographing devices 10 around the crane 1 (work machine), when the operator selects one of them on the display screen 30 by, for example, touching it, the control unit 21 displays the image photographed by the selected photographing device 10 in the photographed image display area 32. Therefore, the operator can select on the display screen 30 the photographing device 10 photographing the part of the crane 1 that he or she wishes to see, and easily display the photographed image of the part that he or she wishes to see.

On the other hand, when the operator inputs and specifies a display range ROI on the photographed image display area 32, the control unit 21 acquires the display range ROI to be enlarged and displayed in the photographed image displayed in the photographed image display area 32. That is, in this embodiment, the control unit 21 functions as the acquisition unit in the present invention.
Specifically, as shown in FIG. 6(a), the control unit 21 acquires information of the specified display range ROI (i.e., the pixel positions at the top, bottom, left and right ends of the display range ROI, etc.) from the captured image displayed in the captured image display area 32.

Then, the control unit 21 is adapted to enlarge and display the acquired display range ROI in the photographed image display area 32, as shown in FIG. 6(b).
At this time, it is possible to configure so that the display range ROI is enlarged and displayed in accordance with the captured image display area 32 .

Specifically, as shown in FIG. 7, the control unit 21 calculates how many times the vertical width (number of pixels) Wa of the captured image display area 32 is compared to the vertical width wa of the display range ROI (vertical magnification ra), and calculates how many times the horizontal width Wb of the captured image display area 32 is compared to the horizontal width wb of the display range ROI (horizontal magnification rb).

Then, the control unit 21 enlarges the display area ROI by the smaller of the vertical magnification ra and the horizontal magnification rb, and displays it in the captured image display area 32 as shown in FIG. 6B.
By enlarging and displaying the display range ROI in this manner, it is possible to greatly enlarge and display the display range ROI within the captured image display area 32 .

As described above, according to the display device 20 of this embodiment, it is possible to greatly enlarge and display the display range ROI that the operator wishes to see in the captured image display area 32 of the display screen 30 .
Therefore, it is possible to display the crane 1 (working machine) and its surroundings on the display screen 30 with good visibility.

It is also possible to configure the display area 32 so that the operator can pinch out on the captured image display area 32 to enlarge and display the display range ROI in the captured image display area 32 .
In this case, the control unit 21 simultaneously performs the process of obtaining the display range ROI specified by the operator from the captured image displayed in the captured image display area 32, and the process of enlarging and displaying the obtained display range ROI in the captured image display area 32.

Furthermore, when the display range ROI is enlarged and displayed in the captured image display area 32, it may become difficult to tell which part of the crane 1 (work machine) is actually being enlarged and displayed.
Therefore, for example, as shown in FIG. 8, the control unit 21 can be configured to display an imaging range Rroi corresponding to the display range ROI on the overhead view display area 31.

In this case, for example, as shown in Figure 9 (a), the width (number of pixels) wbr from the right end of the captured image display area 32 to the right end of the display range ROI is calculated, and the width wbl from the left end of the captured image display area 32 to the left end of the display range ROI is calculated.
Then, the ratios rr and rl of the width Wb in the left-right direction of the photographed image display area 32 to the widths wbr and wbl are calculated.

9B, when the length of the base of a triangle representing the shooting range R of the image capture device 10 in the overhead view display area 31 is set to 1, a straight line is drawn from the right end of the base of the triangle to the position of ratio rr and the apex of the triangle (i.e., the circle representing the image capture device 10). Similarly, a straight line is drawn from the left end of the base of the triangle to the position of ratio rl and the apex of the triangle.
The triangular area between these two lines is the imaging range Rroi corresponding to the display range ROI.

In this way, by displaying the shooting range Rroi corresponding to the display range ROI on the bird's-eye view display area 31, when the operator enlarges the display range ROI in the captured image display area 32, it becomes possible to easily understand which part of the crane 1 (work machine) is being enlarged, thereby improving comprehension.

As described above, in this embodiment, the video captured by worker A around the crane 1 and transmitted from the imaging device 10 is displayed in the captured image display area 32 of the display device 20.
When the operator specifies a display range ROI in the photographed image displayed in the photographed image display area 32 , the display range ROI is enlarged and displayed in the photographed image display area 32 .

In this case, if Worker A moves while taking pictures, changes the shooting direction, or moves crane 1, the point that the operator is focusing on within the display range ROI (hereinafter referred to as the point of focus) may move within the display range ROI, or may move out of the display range ROI and no longer be visible within the display range ROI.
However, if the operator has to follow the moving part of interest within the display range ROI, it may become difficult to see and the operator's eyes may become tired. Also, if the part of interest moves outside the display range ROI, it is necessary to re-specify the display range ROI or to give instructions to the operator A, which is troublesome.

Therefore, when the operator sets the area of interest, i.e., the focus area, on the overhead view display area 31 of the display device 20, the control unit 21 of the display device 20 can be configured to automatically change the display range ROI to follow this.
In this case, for example, when the operator sets a point of interest P on the overhead view display area 31 as shown in Fig. 10 by touching a point of interest on the overhead view display area 31, the control unit 21 of the display device 20 acquires information about the point of interest P on the overhead view display area 31 (such as a pixel position on the display screen 30). That is, the control unit 21 functions as a point of interest acquisition unit.

Then, as shown in FIG. 11(a), if the point of interest P is not within the shooting range Rroi corresponding to the display range ROI described above, the point of interest P falls outside the display range ROI of the captured image display area 32 and is no longer reflected within the display range ROI.
Therefore, when such a situation occurs, the shooting range Rroi is changed based on the point of interest P on the overhead view display area 31 and the position of the shooting device 10 (the circular position in the overhead view display area 31) so that the point of interest P falls within the shooting range Rroi corresponding to the display range ROI, as shown in Figure 11 (b).

By changing the shooting range Rroi in this manner, even if the position of at least one of the crane 1 (work machine) and the shooting device 10 moves, or the shooting direction of the shooting device 10 changes, the control unit 21 can change the display range ROI based on the position of the attention point P on the overhead view display area 31 and the position of the shooting device 10 so that the attention point P is displayed within the display range ROI, and can enlarge and display the display range ROI in the captured image display area 32.
Therefore, even if the position of the crane 1 (work machine) or the imaging device 10 moves or the imaging direction of the imaging device 10 changes, it is possible to prevent at least the focus point P from moving out of the display range ROI and no longer being captured within the display range ROI.

Furthermore, when changing the display range ROI as described above, it is also possible to configure the display range ROI to be changed so that the attention location P approaches the center of the display range ROI.
That is, when the positional relationship between the set point of interest P and the shooting range Rroi corresponding to the display range ROI on the overhead view display area 31 is, for example, as shown in FIG. 12(a), the shooting range Rroi is rotated and moved around the position of the shooting device 10 (the circular position in the overhead view display area 31) so that the center line of the triangle representing the shooting range Rroi (see the dotted line in the figure) passes through the point of interest P, as shown in FIG. 12(b).

Then, the display range ROI corresponding to the rotated and moved photographing range Rroi is enlarged and displayed in the photographed image display region 32 of the display device 20.
With this configuration, it is possible to change the display range ROI so that the attention point P approaches the center of the display range ROI. Therefore, the attention point P does not move around within the display range ROI, making the attention point P easier to see and improving comprehension.

It goes without saying that the present invention is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit of the present invention.
For example, while the above embodiment shows a case where only an overhead view of the work machine is displayed, it is also possible to configure the system to display a front view, side view, etc. of the work machine, and to perform each process based not only on the planar positional relationship between the work machine and the worker, etc., based on the overhead view, but also on the vertical positional relationship between the work machine and the worker, based on the front view, etc.

1. Crane (working machine)
10: Image capture device 10 ₁ , 10 _2: Multiple image capture devices 20: Display device 21: Control unit (acquisition unit, attention point acquisition unit)
31 Bird's-eye view display area 32 Photographed image display area P Point of interest R Photographed range Rroi Photographed range ROI corresponding to the display range Display range

Claims (8)

A display device that displays an overhead view of the surroundings of a work machine and information indicating the positions of image capture devices present around the work machine,
an overhead view display area for displaying the overhead view;
a photographed image display area, separate from the overhead view display area, for displaying a photographed image photographed by the photographing device;
an acquisition unit that acquires a display range to be enlarged and displayed in the captured image;
Equipped with
the display range acquired by the acquisition unit is enlarged and displayed in the captured image display area.
Display device. a photographing range of the photographing device is displayed on the bird's-eye view display area;
The display device according to claim 1 . displaying the shooting range corresponding to the display range on the overhead view display area;
The display device according to claim 2 . when one of the plurality of photographing devices present around the work machine is selected, the photographed image captured by the selected photographing device is displayed in the photographed image display area.
The display device according to claim 1 . The acquisition unit acquires the display range designated on the captured image display area.
The display device according to claim 1 . a point of interest acquisition unit for acquiring a point of interest on the overhead view,
when the position of at least one of the work machine and the photographing device has moved, or when the photographing direction of the photographing device has changed, the display range is changed based on the position of the attention point and the photographing device acquired by the attention point acquisition unit so that the attention point is displayed within the display range, and the display is enlarged and displayed in the photographed image display area.
The display device according to claim 1 . changing the display range so that the attention point approaches the center of the display range;
The display device according to claim 6. A display program for causing the display device according to claim 1 to execute the display according to claim 1.

Images