JP7810097B2 - Display System - Google Patents

Description

This disclosure relates to a display system.

The forklift disclosed in Patent Document 1 includes a camera, a display, a lifting device, a control device, and forks. Images captured by the camera are displayed on the display. The forklift operator can check the situation ahead of the forklift by checking the image displayed on the display. The lifting device raises and lowers the camera. The control device lowers the camera when a load is placed on the fork. This prevents the load from entering the camera's imaging range when a load is placed on the fork.

Japanese Patent Application Laid-Open No. 2003-128395

If a forklift is driven with the camera lowered, there is a risk that the camera may be hit by foreign objects. For example, flying stones may hit the camera.

A display system that solves the above problem displays the view ahead of a forklift equipped with forks on which cargo can be placed. The display system includes a camera positioned to capture an image of the view ahead of the forklift from the space between the cargo placed on the forks and the road surface, a lifting device that raises and lowers the camera, a display unit that displays the image captured by the camera, and a control device. The control device calculates the height of the camera from the road surface, and sets the camera to its normal position when certain conditions are not met, including that the cargo is placed on the forks and the fork's lift height is less than the low lift height threshold. If the certain conditions are met, the control device adjusts the camera's position to within a low position range lower than the normal position based on the camera's height.

When certain conditions are met, luggage is captured in the image captured by the camera. In this case, the control device can lower the camera to a position lower than its normal position, thereby reducing the proportion of luggage in the image. The control device adjusts the camera position within the low position range based on the camera's height. This prevents the camera from being lowered excessively. Preventing the distance between the camera and the road surface from becoming excessively short prevents foreign objects from hitting the camera.

In the above display system, the forklift is equipped with a height sensor that detects the lift height and a tilt sensor that detects the tilt angle of the forks, and the control device may calculate the height from the road surface to the tips of the forks based on the lift height, the tilt angle, and the length of the forks, and calculate the height of the camera from the road surface based on the height from the road surface to the tips of the forks and the displayed position of the tips of the forks in the image.

In the display system, the specific condition may include a vehicle speed of the forklift being less than a speed threshold.
In the display system, the specific condition may include that the traveling direction of the forklift is a forward direction.

In the above display system, the forklift may be equipped with an accelerator operation unit, and the specific condition may include the amount of operation of the accelerator operation unit being less than an operation amount threshold.

This invention prevents foreign objects from hitting the camera.

FIG. 1 is a side view of a forklift. FIG. 2 is a front view of an outer mast provided in the forklift truck of FIG. 1 . FIG. 2 is a schematic diagram of the forklift truck of FIG. 1. 4 is a flowchart showing the lift control performed by the control device of FIG. 3 . FIG. 10 is a side view showing the forklift with the camera in a fixed position. 6 is a diagram showing an image captured by a camera provided on the forklift truck of FIG. 5. FIG. FIG. 4 is a side view showing the forklift in which the position of the camera is adjusted by the lifting device of FIG. 3 . 8 is a diagram showing an image captured by a camera provided on the forklift of FIG. 7. FIG.

An embodiment of a display system will now be described.
<Forklift>
As shown in Figure 1, the forklift 10 is a counter type. The forklift 10 may be a reach type. The forklift 10 is manually operated by an operator on board the forklift 10. The forklift 10 may be configured to be switchable between automatic and manual operation. In the following description, front, rear, left, right and front are determined based on the forklift 10. The forklift 10 is equipped with a display system DS.

The forklift 10 includes a vehicle body 11, drive wheels 12, steering wheels 13, and a loading device 14.
The cargo handling device 14 is provided at the front of the vehicle body 11. The cargo handling device 14 includes a mast 15. The mast 15 includes an outer mast 21 and an inner mast 17.

As shown in Figure 2, two outer masts 21 are provided, spaced apart from each other in the left-right direction. The cargo handling device 14 is equipped with an outer mast bar 22 that connects the outer masts 21 together. The outer mast bar 22 connects the lower ends of the two outer masts 21 together. Two inner masts 17 are provided, spaced apart from each other in the left-right direction. The inner masts 17 can be raised and lowered relative to the outer masts 21.

1, the cargo handling device 14 includes a lift bracket 18. The lift bracket 18 is provided on the inner mast 17 so as to be able to move up and down.
The cargo handling device 14 includes a fork 19. The fork 19 is attached to a lift bracket 18. The fork 19 moves up and down together with the lift bracket 18. A cargo W1 to be handled is placed on the fork 19.

The loading device 14 is equipped with a lift cylinder 31. The lift cylinder 31 is a hydraulic cylinder. The lift cylinder 31 raises and lowers the forks 19. The loading device 14 is equipped with a tilt cylinder 32. The tilt cylinder 32 is a hydraulic cylinder. The tilt cylinder 32 tilts the forks 19.

As shown in FIG. 3, the display system DS includes a control device 41. The control device 41 includes a processor 42 and a storage unit 43. The processor 42 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), or a digital signal processor (DSP). The storage unit 43 includes random access memory (RAM) and read-only memory (ROM). The storage unit 43 stores a program for operating the forklift 10. The storage unit 43 can be said to store program code or instructions configured to cause the processor 42 to execute processing. The storage unit 43, i.e., a computer-readable medium, includes any available medium accessible by a general-purpose or dedicated computer. The control device 41 may be configured with hardware circuits such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA). The control device 41, which is a processing circuit, may include one or more processors operating according to a computer program, one or more hardware circuits such as an ASIC or FPGA, or a combination thereof.

The forklift 10 is equipped with a height sensor 51. The height sensor 51 detects the height of the forks 19. The height of the forks 19 is the height from the road surface to the forks 19. The height sensor 51 outputs an electrical signal corresponding to the height of the forks 19 to the control device 41. The control device 41 can recognize the height of the forks 19 based on the electrical signal from the height sensor 51.

The forklift 10 is equipped with a load sensor 52. The load sensor 52 outputs an electrical signal to the control device 41 that corresponds to the weight of the cargo W1 placed on the forks 19. The control device 41 can recognize the weight of the cargo W1 from the electrical signal from the load sensor 52.

The forklift 10 is equipped with a vehicle speed sensor 53. The vehicle speed sensor 53 detects the vehicle speed of the forklift 10. If the forklift 10 is electric, the vehicle speed sensor 53 may be, for example, a rotation speed sensor that detects the rotation speed of the travel motor. The control device 41 can calculate the vehicle speed of the forklift 10 from the detection result of the vehicle speed sensor 53.

The forklift 10 is equipped with a tilt sensor 54. The tilt sensor 54 detects the tilt angle of the forks 19. The tilt angle of the forks 19 is the angle of the forks 19 relative to the road surface, with 0° being the angle when the road surface and the top surface of the forks 19 are parallel. The tilt sensor 54 may be, for example, a potentiometer. The control device 41 can recognize the tilt angle of the forks 19 from the detection results of the tilt sensor 54.

The forklift 10 is equipped with a direction control unit 55. The direction control unit 55 is, for example, a lever. The direction control unit 55 tilts forward or backward from a neutral position. The direction of travel of the forklift 10 can be determined by the operation direction of the direction control unit 55. In this embodiment, tilting the direction control unit 55 forward causes the forklift 10 to travel forward. Tilting the direction control unit 55 backward causes the forklift 10 to travel backward.

The forklift 10 is equipped with a direction sensor 56. The direction sensor 56 detects the operation direction of the direction operation unit 55. The direction sensor 56 outputs an electrical signal corresponding to the operation direction of the direction operation unit 55 to the control device 41. The control device 41 recognizes the operation direction of the direction operation unit 55.

The forklift 10 is equipped with an accelerator operation unit 57. The accelerator operation unit 57 is, for example, a pedal. The accelerator operation unit 57 is operated to accelerate the forklift 10.

The forklift 10 is equipped with an accelerator sensor 58. The accelerator sensor 58 detects the amount of operation of the accelerator operation unit 57. The accelerator sensor 58 outputs an electrical signal corresponding to the amount of operation of the accelerator operation unit 57 to the control device 41. The control device 41 recognizes the amount of operation of the accelerator operation unit 57.

The display system DS includes a camera 59. The camera 59 is a digital camera. The camera 59 includes an imaging element. The imaging element is, for example, a charge-coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor. The camera 59 is positioned so that it can capture an image of the area ahead of the forklift 10 from the space between the luggage W1 and the road surface when the luggage W1 is loaded on the forks 19. For example, as shown in FIG. 2, the camera 59 is positioned on the outer mast bar 22.

The display system DS includes a lifting device 60. The lifting device 60 raises and lowers the camera 59. The lifting device 60 includes, for example, an electric motor and a speed reduction mechanism. The camera 59 is switched between being raised and lowered by switching the direction of rotation of the electric motor. The lifting device 60 is disposed, for example, on the outer mast bar 22.

As shown in FIG. 1, the highest position that the camera 59 can assume by being raised or lowered by the lifting device 60 is designated as the normal position P1. Of the positions that the camera 59 can assume by being raised or lowered by the lifting device 60, the range that is lower than the normal position P1 and is set so that the camera 59 does not come into contact with the road surface is designated as the low position range A1. The lifting device 60 can stop the camera 59 at any position within the low position range A1.

As shown in FIG. 3, the display system DS includes a display unit 71. The display unit 71 is provided in a position visible to the operator of the forklift 10. The display unit 71 is, for example, a liquid crystal display or an organic electroluminescence display.

The control device 41 updates the display content of the display unit 71. The control device 41 displays an image captured by the camera 59 on the display unit 71.
The control device 41 controls the elevator device 60 to perform elevation control for raising and lowering the camera 59. The elevation control is repeatedly executed at a predetermined control period. The elevation control will be described below.

<Lift control>
As shown in FIG. 4, in step S<b>1 , the control device 41 acquires the lift height of the forks 19 from the lift height sensor 51 .

Next, in step S2, the control device 41 acquires the load from the load sensor 52.
Next, in step S3, the control device 41 acquires the tilt angle of the fork 19 from the tilt sensor 54.

Next, in step S4, the control device 41 calculates the coordinates of the tips of the forks 19 based on the elevation height, tilt angle, and length of the forks 19. These coordinates are in a coordinate system with a predetermined position as the origin. The predetermined position is, for example, a position where the horizontal direction is the normal position P1 and the vertical direction is the road surface. The coordinates of the tips of the forks 19 need only include a coordinate in the vertical direction. The coordinates of the tips of the forks 19 include the height from the road surface to the tips of the forks 19. The height from the road surface to the tips of the forks 19 varies depending on the elevation height, tilt angle, and length of the forks 19. Therefore, the control device 41 can calculate the height from the road surface to the tips of the forks 19 based on these parameters. The length of the forks 19 can be stored in advance in the memory unit 43, for example.

Next, in step S5, the control device 41 calculates the height of the camera 59 from the road surface. The height of the camera 59 from the road surface can be calculated based on the coordinates of the tips of the forks 19 and the displayed position of the tips of the forks 19 in the image. The displayed position of the tips of the forks 19 in the image is the coordinate of the tips of the forks 19 in the coordinate system of the image. The displayed position of the tips of the forks 19 in the image only needs to include at least the coordinate in the height direction in the image. The displayed position of the tips of the forks 19 in the image differs depending on the relative heights of the forks 19 and the camera 59. Therefore, the control device 41 can calculate the relative heights of the forks 19 and the camera 59 based on the displayed position of the tips of the forks 19 in the image. Because the control device 41 can recognize the height of the forks 19 from the road surface, it can calculate the height of the camera 59 from the relative heights of the camera 59 and the forks 19.

Next, in step S6, the control device 41 determines whether the forklift 10 is holding the load W1 and whether the lift height of the forks 19 is low. If the determination result in step S6 is negative, the control device 41 performs the process in step S7. If the determination result in step S6 is positive, the control device 41 performs the process in step S8.

The control device 41 determines that the forklift 10 is holding the cargo W1 when the weight of the cargo W1 placed on the fork 19 is equal to or greater than the load determination threshold. The weight of the cargo W1 placed on the fork 19 can be recognized from the detection result of the load sensor 52. The load determination threshold is a predetermined value. The load determination threshold is set to a value that can determine that the cargo W1 is placed on the fork 19. For example, the load determination threshold is set to a value slightly higher than the weight detected by the load sensor 52 when the cargo W1 is not placed on the fork 19. As a result, when the cargo W1 is placed on the fork 19, it is determined that the forklift 10 is holding the cargo W1.

When the lift height of the forks 19 is less than the low lift height threshold, the control device 41 determines that the lift height of the forks 19 is low. The lift height of the forks 19 can be determined from the detection results of the lift height sensor 51. The low lift height threshold is a predetermined value. The lower the lift height of the forks 19, the higher the proportion of the luggage W1 in the imaging range of the camera 59 at the normal position P1. The low lift height threshold is set so that the proportion of the luggage W1 in the imaging range of the camera 59 at the normal position P1 is acceptable. The low lift height threshold may be set to a lift height such that, when luggage W1 is loaded on the forks 19, the luggage W1 does not enter the imaging range of the camera 59 at the normal position P1.

In step S7, the control device 41 controls the lifting device 60 to move the camera 59 to the normal position P1. More specifically, if the position of the camera 59 is normal position P1 at the time of performing the processing of step S7, the control device 41 maintains the position of the camera 59 at normal position P1. If the position of the camera 59 is not normal position P1 at the time of performing the processing of step S7, the control device 41 moves the camera 59 to the normal position P1 by raising the camera 59 using the lifting device 60. After completing the processing of step S7, the control device 41 ends the lifting control.

In step S8, the control device 41 determines whether the vehicle speed of the forklift 10 is equal to or greater than the speed threshold. The speed threshold is set to a value that can determine whether the forklift 10 is stopped. If the determination result in step S8 is positive, the control device 41 performs the processing of step S7. If the determination result in step S8 is negative, the control device 41 performs the processing of step S9. If the determination result in step S8 is negative, the vehicle speed of the forklift 10 is less than the speed threshold.

In step S9, the control device 41 adjusts the position of the camera 59 within the low position range A1 by controlling the lifting device 60. For example, the control device 41 adjusts the position of the camera 59 so that the height of the camera 59 is not excessively low. Furthermore, if the lift height of the forks 19 is less than the low lift height threshold, the closer the lift height of the forks 19 is to the low lift height threshold, the lower the proportion of the luggage W1 in the image. Therefore, while recognizing the height of the camera 59 from the road surface, the closer the lift height of the forks 19 is to the low lift height threshold, the higher the camera 59 moves to. In other words, while recognizing the height of the camera 59 from the road surface, the lower the lift height of the forks 19 is, the lower the camera 59 moves to.

If the specific condition is not met, the control device 41 sets the camera 59 to the normal position P1. If the specific condition is met, the control device 41 adjusts the position of the camera 59 within the low position range A1. If the determination result in step S6 is negative, the position of the camera 59 is set to the normal position P1. Therefore, the specific condition includes a situation in which a load W1 is placed on the fork 19 and the lift height of the fork 19 is less than the low lift height threshold. If the determination result in step S8 is positive, the position of the camera 59 is set to the normal position P1. If the determination result in step S8 is negative, the position of the camera 59 is adjusted within the low position range A1. Therefore, the specific condition includes a situation in which the vehicle speed of the forklift 10 is less than the speed threshold.

[Operation of this embodiment]
As shown in Figure 5, the forklift 10 is holding a load W1, and the lift height of the forks 19 is set to a low lift height. If the camera 59 were in a fixed position, the camera 59 would capture an image of a fixed range. As a result, as shown in Figure 6, the proportion of the load W1 in the image IM1 is high, making it difficult for the operator of the forklift 10 to see the situation ahead of the load W1 even when the image IM1 is displayed on the display unit 71. For example, even if an obstacle M1 is present ahead of the load W1, it is difficult for the operator of the forklift 10 to see the obstacle M1. The obstacle M1 includes both a person and a non-personal object.

As shown in FIG. 7 , in the forklift 10 of this embodiment, the camera 59 can be raised and lowered by the lifting device 60. When the forklift 10 is holding a load W1 and the lifting height of the forks 19 is low, the camera 59 is lowered to a position lower than the normal position P1. Lowering the camera 59 lowers the imaging range of the camera 59, thereby preventing the load W1 from entering the imaging range of the camera 59. As a result, as shown in FIG. 8 , the proportion of the load W1 in the image IM1 is reduced. This reduces the proportion of the load W1 in the image IM1, making it easier for the operator of the forklift 10 to confirm the situation ahead of the load W1 when checking the image IM1 displayed on the display unit 71. For example, it is easier for the operator of the forklift 10 to confirm that an obstacle M1 is present ahead of the load W1.

[Effects of this embodiment]
(1) When a specific condition is met, the control device 41 adjusts the position of the camera 59 within the low position range A1 based on the height of the camera 59. When the specific condition is met, the baggage W1 appears in the image IM1 captured by the camera 59. In this case, the control device 41 lowers the camera 59, thereby reducing the proportion of the baggage W1 in the image IM1. The control device 41 adjusts the position of the camera 59 within the low position range A1 based on the height of the camera 59. This prevents the camera 59 from being lowered excessively. By preventing the distance between the camera 59 and the road surface from becoming excessively short, it is possible to prevent foreign objects from hitting the camera 59.

(2) The control device 41 calculates the height of the forks 19 from the road surface to the tips of the forks 19 based on the lift height of the forks 19, the tilt angle of the forks 19, and the length of the forks 19. The control device 41 calculates the height of the camera 59 from the road surface based on the height of the forks 19 from the road surface and the displayed position of the tips of the forks 19 in image IM1. The height of the camera 59 from the road surface can be calculated using the lift height sensor 51, tilt sensor 54, and camera 59. The height of the camera 59 from the road surface can be calculated without using a dedicated device for calculating the height of the camera 59 from the road surface. Therefore, manufacturing costs can be reduced compared to when a dedicated device for calculating the height of the camera 59 from the road surface is provided on the forklift 10.

(3) The specific condition includes the forklift 10's vehicle speed being less than the speed threshold. When a stopped forklift 10 is started, the forklift 10 is likely to check what is ahead. If the forklift 10's vehicle speed is equal to or greater than the speed threshold, the forklift 10 is moving. Therefore, by setting the position of the camera 59 to the normal position P1 while the forklift 10 is moving, it is possible to further prevent foreign objects from hitting the camera 59.

[Example of change]
The embodiment can be modified as follows: The embodiment and the following modifications can be combined with each other to the extent that they are not technically inconsistent.

The specific condition may include the traveling direction of the forklift 10 being a forward direction. The control device 41 determines whether the traveling direction of the forklift 10 is a forward direction. The control device 41 can determine whether a forward movement is instructed by the direction operation unit 55 from the detection result of the direction sensor 56. The control device 41 determines that the traveling direction of the forklift 10 is a forward direction when a forward movement is instructed by the direction operation unit 55. The control device 41 may determine whether the traveling direction of the forklift 10 is a forward direction in addition to or instead of step S8.

If the specific condition includes the forklift 10 traveling in a forward direction, the specific condition is not met if the forklift 10 is not traveling in a forward direction. When the direction operation unit 55 is in the neutral position or tilted backward, the camera 59 is positioned in the normal position P1. Checking the front of the forklift 10 is often performed when the forklift 10 is moving forward. By positioning the camera 59 in the normal position P1 when the forklift 10 is not moving forward, it is possible to prevent the camera 59 from being lowered when checking the front of the forklift 10 is not necessary. This further prevents foreign objects from hitting the camera 59.

The specific condition may include the operation amount of the accelerator operation unit 57 being less than the operation amount threshold. The control device 41 recognizes the operation amount of the accelerator operation unit 57 from the detection result of the accelerator sensor 58. The control device 41 then determines whether the operation amount of the accelerator operation unit 57 is less than the operation amount threshold. The operation amount threshold is set to a value that can determine whether the forklift 10 is traveling. For example, the operation amount threshold is the operation amount of the accelerator operation unit 57 required to start the forklift 10 from a stopped state. The control device 41 may determine whether the operation amount of the accelerator operation unit 57 is less than the operation amount threshold in addition to or instead of step S8. In this case, the same effect as effect (3) of the embodiment can be obtained.

The specific condition may include both that the traveling direction of the forklift 10 is a forward direction and that the operation amount of the accelerator operation unit 57 is less than the operation amount threshold.
The specific condition may include that the load W1 is placed on the forks 19 and that the lift height of the forks 19 is less than the low lift height threshold. Therefore, the specific condition does not necessarily include that the vehicle speed of the forklift 10 is less than the speed threshold.

The forklift 10 may be provided with a dedicated device for calculating the height of the camera 59 from the road surface.
The forklift 10 may include an adjustment member that can be operated by the operator of the forklift 10 to adjust the height of the camera 59. In this case, it may be possible to set whether the adjustment of the height of the camera 59 by the control device 41 or the adjustment of the height of the camera 59 by the operator of the forklift 10 is to be given priority.

The control device 41 may calculate the length of the fork 19 by image processing.
The control device 41 may determine whether or not the package W1 is placed on the fork 19 from the image IM1.

○ When luggage W1 is placed on the forks 19, the brightness of the area in image IM1 where luggage W1 appears decreases. For this reason, the area in image IM1 where there is a large change in brightness may be used as the display position for the tips of the forks 19.

- A switch that switches on and off depending on whether or not the luggage W1 is placed on the fork 19 may be used as a sensor for detecting whether the luggage W1 is placed on the fork 19. A rangefinder that measures the distance above the fork 19 may be used as a sensor for detecting whether the luggage W1 is placed on the fork 19. When the luggage W1 is placed on the fork 19, the distance measured by the rangefinder becomes shorter, and therefore the control device 41 can determine whether or not the luggage W1 is placed on the fork 19.

The forklift 10 may be remotely operated. In this case, the operator operates the forklift 10 from a remote location away from the forklift 10. The operator operates a remote control device installed in the remote location. The remote control device may be a dedicated device or a mobile communication terminal such as a tablet terminal. The remote control device includes an operation unit for operating the forklift 10, a communication device, a terminal control device, and a terminal display unit. The communication device is configured to be able to communicate with the forklift 10. The terminal control device can acquire an image IM1 captured by the camera 59 via the communication device. The terminal control device displays the image IM1 on the terminal display unit. In this case, the terminal display unit is the display unit. If the terminal control device can give instructions to the control device 41 via the communication device, the terminal control device may raise and lower the camera 59 by performing lifting and lowering control. In this case, the terminal control device is the control device.

A1...low position range, DS...display system, IM1...image, P1...normal position, W1...baggage, 10...forklift, 19...fork, 41...control device, 51...height sensor, 54...tilt sensor, 57...accelerator operation unit, 59...camera, 60...lifting device, 71...display unit.

Claims (5)

A display system that displays the front of a forklift equipped with forks on which a load is to be loaded,
a camera disposed to capture an image of a front of the forklift from a space between the load placed on the fork and a road surface;
an elevator device for raising and lowering the camera;
a display unit on which an image captured by the camera is displayed;
a control device;
The control device
Calculating the height of the camera from the road surface;
When the load is loaded on the fork and a specific condition including that the lifting height of the fork is less than a low lifting height threshold is not satisfied, the position of the camera is set to a normal position;
A display system that adjusts the position of the camera within a low position range that is lower than the normal position based on the height of the camera when the specific condition is met. The forklift
a height sensor for detecting the height;
a tilt sensor that detects the tilt angle of the fork,
The control device
calculating a height from the road surface to the tip of the fork based on the lift height, the tilt angle, and the length of the fork;
The display system according to claim 1 , wherein the height of the camera from the road surface is calculated based on the height of the tips of the forks from the road surface and the display position of the tips of the forks in the image. The display system described in claim 1 or claim 2, wherein the specific condition includes the forklift's vehicle speed being less than a speed threshold. The display system described in claim 1 or claim 2, wherein the specific condition includes the forklift's traveling direction being a forward direction. The forklift includes an accelerator operation unit,
The display system according to claim 1 or 2, wherein the specific condition includes a condition in which the operation amount of the accelerator operation unit is less than an operation amount threshold value.