JP7831246B2 - Display system - Google Patents

Description

This disclosure relates to a display system.

The forklift disclosed in Patent Document 1 comprises a camera, a display unit, a control device, and forks. Images captured by the camera are displayed on the display unit. The forklift operator can confirm the situation in front of the forklift by checking the image displayed on the display unit. The control device changes the shooting conditions according to the fork's lifting height. As a result, an image corresponding to the fork's lifting height is displayed on the display unit.

Japanese Patent Publication No. 2011-37536

In Patent Document 1, when a load is placed on the forks, the load is included in the image captured by the camera. Because the display shows the load on the forks, it is difficult to see the situation in front of the forklift.

The display system that solves the above problem is a display system for displaying the front of a forklift equipped with forks on which a load is placed, comprising: a camera positioned to capture images of the front of the forklift from the space between the load placed on the forks and the road surface; a display unit that displays the image captured by the camera; and a control device, wherein the control device enlarges the high-brightness range of the image, excluding pixels with brightness below a threshold, and displays it on the display unit.

When a load is loaded onto the forks, the brightness of the area of the image captured by the camera that includes the load or the area that is in the load's shadow will be below a threshold. When a load is loaded onto the forks, the high-brightness area is the area visible in front of the forklift through the space between the load and the road surface. The control device enlarges this high-brightness area and displays it on the display unit, making it easier to see the situation in front of the load.

Regarding the above display system, the control device may divide the image into multiple ranges, calculate the average brightness value for each of the multiple ranges, and, if the difference in the average brightness values between adjacent ranges is less than the average brightness threshold, increase the difference in brightness between those adjacent ranges.

Regarding the above display system, the control device calculates the color difference between adjacent ranges among the multiple ranges. If the difference in the average values of the luminances between adjacent ranges is less than the average luminance threshold, and the color difference is also less than the color threshold, the control device may increase the difference in luminance between those adjacent ranges.

According to this invention, it becomes easier to check the situation in front of the cargo.

This is a side view of a forklift. Figure 1 is a front view of the outer mast of the forklift. Figure 1 is a schematic diagram of the forklift configuration. Figure 3 is a flowchart showing the display control performed by the control device. This figure shows the high-brightness range extracted from the image by the display control in Figure 4. This figure shows the high-luminance range extracted by the display control in Figure 4, which is divided into multiple ranges. This figure shows the image displayed on the display unit in Figure 3. This flowchart shows an example of changes to the display control. This figure shows the high-brightness range before the brightness adjustment process is performed. This figure shows the high-brightness range after the brightness adjustment process has been performed.

One embodiment of the display system will be described.
<Forklift>
As shown in Figure 1, the forklift 10 is a counterbalanced forklift. The forklift 10 may also be a reach forklift. The forklift 10 is operated manually by an operator riding on the forklift 10. The forklift 10 may be configured to allow switching between automatic and manual operation. In the following description, front, back, left, and right refer to the front, back, left, and right relative to the forklift 10. The forklift 10 is equipped with a display system DS.

The forklift 10 comprises a body 11, drive wheels 12, steering wheels 13, and a cargo handling device 14.
The cargo handling device 14 is located at the front of the vehicle body 11. The cargo handling device 14 includes a mast 15. The mast 15 includes an outer mast 21 and an inner mast 17.

As shown in Figure 2, two outer masts 21 are provided, spaced apart from each other in the left-right direction. The cargo handling device 14 includes an outer mast bar 22 that connects the two outer masts 21. The outer mast bar 22 connects the lower ends of the two outer masts 21. Two inner masts 17 are provided, spaced apart from each other in the left-right direction. The inner masts 17 are vertically movable relative to the outer masts 21.

As shown in Figure 1, the cargo handling device 14 includes a lift bracket 18. The lift bracket 18 is mounted so as to be able to move up and down relative to the inner mast 17.
The cargo handling device 14 is equipped with forks 19. The forks 19 are attached to a lift bracket 18. The forks 19 move up and down together with the lift bracket 18. The cargo W1 to be handled is placed on the forks 19.

The cargo handling device 14 includes a lift cylinder 31. The lift cylinder 31 is a hydraulic cylinder. The lift cylinder 31 raises and lowers the forks 19. The cargo handling device 14 also includes a tilt cylinder 32. The tilt cylinder 32 is a hydraulic cylinder. The tilt cylinder 32 tilts the forks 19.

As shown in Figure 3, the display system DS includes a control device 41. The control device 41 comprises a processor 42 and a storage unit 43. The processor 42 may be, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a DSP (Digital Signal Processor). The storage unit 43 includes RAM (Random Access Memory) and ROM (Read Only Memory). The storage unit 43 stores a program for operating the forklift 10. It can be said that the storage unit 43 stores program code or instructions configured to cause the processor 42 to execute processing. The storage unit 43, i.e., the computer-readable medium, includes any available medium accessible by a general-purpose or dedicated computer. The control device 41 may be composed of hardware circuits such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control device 41, which is a processing circuit, may include one or more processors operating according to a computer program, one or more hardware circuits such as an ASIC or FPGA, or a combination thereof.

The display system DS includes a camera 51. The camera 51 is a digital camera. The camera 51 includes an image sensor. The image sensor is, for example, a CCD image sensor (Charge Coupled Device image sensor) or a CMOS image sensor (Complementary Metal Oxide Semiconductor image sensor). The camera 51 is positioned to capture images of the area in front of the forklift 10 from the space between the load W1 and the road surface when the load W1 is loaded on the forks 19. For example, as shown in Figure 2, the camera 51 is positioned on the outer mast bar 22. In this embodiment, the image captured by the camera 51 is in RGB format.

The display system DS includes a display unit 61. The display unit 61 is positioned so that it can be seen by the operator of the forklift 10. The display unit 61 is, for example, a liquid crystal display or an organic electroluminescent display.

The control device 41 updates the display content of the display unit 61. The control device 41 displays the image obtained by the camera 51 on the display unit 61. The display control performed by the control device 41 will now be described. The display control is repeatedly executed at a predetermined control cycle.

<Display Control>
As shown in Figure 4, in step S1, the control device 41 converts the image acquired from the camera 51 from RGB format to YCbCr format. Y represents luminance. Cb represents the blue color difference. Cr represents the red color difference.

Next, in step S2, the control device 41 extracts the high-luminance range.
As shown in Figure 5, the high-brightness range A1 is the range in which the brightness of pixels constituting the image IM1 is equal to or greater than a threshold. The control device 41 can extract the high-brightness range A1 by excluding pixels from the image IM1 whose brightness is less than the threshold. The brightness of a pixel is the value of Y. In this embodiment, the high-brightness range A1 is the range in which the width dimension is the same as that of the image IM1 and the height dimension is shorter than that of the image IM1. The width direction of the image IM1 is the X-axis direction in the coordinate system of the image IM1. The height direction of the image IM1 is the Y-axis direction in the coordinate system of the image IM1.

In this embodiment, when extracting a band-shaped high-luminance area A1 from image IM1, for example, the control device 41 calculates the average luminance for multiple pixels in image IM1 that are at the same height. Then, if the average luminance is below a threshold, the control device 41 determines that the luminance for all of the multiple pixels at the same height is below the threshold. That is, for each row with the same Y-coordinate, the control device 41 uses the average luminance to determine whether the luminance is below the threshold.

As shown in Figure 4, in step S3, the control device 41 divides the high-brightness range A1 into multiple ranges.
As shown in Figure 6, in this embodiment, the high-luminance range A1 is divided into eight ranges A21 to A28. The high-luminance range A1 may be divided into multiple ranges, and may be divided into fewer than eight ranges or into nine or more ranges.

As shown in Figure 4, in step S4, the control device 41 calculates the average value of the brightness for each of the multiple ranges A21 to A28.
Next, in step S5, the control device 41 calculates the color difference between adjacent ranges A21 to A28. The color difference is the difference in color. The control device 41 can recognize the color of each of the multiple ranges A21 to A28 from the values of Cb and Cr. The control device 41 derives the color represented by the values of Cb and Cr for each of the multiple ranges A21 to A28. Then, the control device 41 calculates the color difference between adjacent ranges A21 to A28 from the difference in color represented by the values of Cb and Cr. The control device 41 may calculate the color difference as the difference in the Cb values between adjacent ranges A21 to A28, or it may calculate the color difference as the difference in the Cr values between adjacent ranges A21 to A28. The control device 41 may calculate the color difference based on the difference in Cb values between adjacent ranges A21 to A28 and the difference in Cr values between adjacent ranges A21 to A28.

Next, in step S6, the control device 41 increases the difference in luminance between adjacent ranges A21 to A28 if the difference in average luminance between those ranges A21 to A28 is less than the average luminance threshold and the color difference is less than the color threshold. If the difference in average luminance between adjacent ranges A21 to A28 is small, it is difficult for a person to perceive the difference between them. The average luminance threshold is set to a value that allows a person to perceive the difference between adjacent ranges A21 to A28. Similarly, if the color difference between adjacent ranges A21 to A28 is small, it is difficult for a person to perceive the difference between them. The color threshold is set to a value that allows a person to perceive the difference between adjacent ranges A21 to A28.

The control device 41 performs a process to increase the brightness difference between each of the ranges A21 to A28 and the adjacent ranges A21 to A28. The control device 41 sequentially changes the ranges A21 to A28 to which the process of increasing the brightness difference between adjacent ranges A21 to A28 is applied. As a result, the process of increasing the brightness difference between each of the ranges A21 to A28 and the adjacent ranges A21 to A28 is performed. The following explanation will use the example of increasing the brightness difference between the brightness of range A26, which is the target of the process of increasing the brightness difference between adjacent ranges A21 to A28, and the brightness of the range adjacent to range A26.

The ranges adjacent to range A26 are ranges A22, A25, and A27. The control device 41 determines whether the difference between the average luminance of range A26 and the average luminance of range A22 is less than the average luminance threshold, and whether the color difference between range A26 and range A22 is less than the color threshold. If the difference between the average luminance of range A26 and the average luminance of range A22 is less than the average luminance threshold, and the color difference between range A26 and range A22 is less than the color threshold, the control device 41 increases the difference between the luminance of range A26 and the luminance of range A22. For example, the control device 41 may add an additive value to the luminance of the pixels constituting range A26. The control device 41 may subtract a subtractive value from the luminance of the pixels constituting range A22. The additive value and the subtractive value are positive values. The additive value and the subtractive value may be the same value or different values. The control device 41 may subtract a subtraction value from the brightness of the pixels constituting range A26 and add a summation value to the brightness of the pixels constituting range A22. The control device 41 performs the same processing for ranges A25 and A27. This increases the difference between the brightness of range A26 and the brightness of adjacent ranges A22, A25, and A27.

Next, in step S7, the control device 41 displays the image IM1 obtained by the camera 51 on the display unit 61. The control device 41 enlarges the high-brightness range A1, whose brightness was adjusted in step S6, and displays it on the display unit 61. As a result, the high-brightness range A1 is stretched in the height direction compared to when the entire image IM1 is displayed on the display unit 61. After completing the process in step S7, the control device 41 terminates the display control.

[Operation of this embodiment]
As shown in Figure 1, assume that a load W1 is placed on the forks 19. In this case, the area of the image IM1 captured by the camera 51 in which the load W1 is visible and the area in the shadow of the load W1 will have a brightness below the threshold. In the example shown in Figure 5, the load W1 is visible in the upper area A2, which is higher than the high-brightness area A1. The lower area A3, which is lower than the high-brightness area A1, is in the shadow of the load W1. The upper area A2 and the lower area A3 are excluded from the image IM1 as pixels whose brightness is below the threshold. The high-brightness area A1 is the area that shows the front of the forklift 10 through the space between the load W1 placed on the forks 19 and the road surface.

As shown in Figure 7, if an obstacle M1 is present in front of the forklift 10, the display unit 61 will show an enlarged view of the obstacle M1 that can be captured from between the load W1 and the road surface. In the example shown in Figure 7, a person's foot captured from between the load W1 and the road surface will be displayed enlarged on the display unit 61.

If the load W1 is not loaded onto the forks 19, the load W1 will not be visible in image IM1. Furthermore, no shadow of the load W1 will be cast. Therefore, the entire image IM1 can become a high-brightness area A1. When the entire image IM1 becomes a high-brightness area A1, the control device 41 displays image IM1 on the display unit 61 without partially enlarging the image IM1 obtained by the camera 51. That is, the control device 41 displays the entire image IM1 obtained by the camera 51 on the display unit 61. As a result, when the load W1 is not loaded onto the forks 19, a wide area in front of the forklift 10 can be displayed on the display unit 61.

[Effects of this embodiment]
(1) The control device 41 enlarges the high-brightness range A1 and displays it on the display unit 61. When the control device 41 displays the entire image IM1 on the display unit 61, the area in which the load W1 is visible and the area in shadow of the load W1 are displayed on the display unit 61. These areas do not contribute, or contribute very little, to confirming the situation in front of the load W1, even if the operator of the forklift 10 can see them. The high-brightness range A1 is the area in front of the forklift 10 that is visible through the space between the load W1 placed on the forks 19 and the road surface. By the control device 41 enlarging the high-brightness range A1 and displaying it on the display unit 61, the operator of the forklift 10 can more easily confirm the situation in front of the load W1. The aspect ratio when the high-brightness range A1 is enlarged and displayed on the display unit 61 will be different from the aspect ratio when the entire image IM1 is displayed on the display unit 61. As a result, the shape of the obstacle M1 displayed on the display unit 61 will differ from the shape of the obstacle M1 when the entire image IM1 is displayed on the display unit 61. However, for the operation of the forklift 10, it is sufficient to determine whether or not an obstacle M1 is present in front of the load W1. Therefore, changes in the shape of the obstacle M1 are acceptable.

(2) The control device 41 increases the difference in brightness between adjacent ranges A21 to A28 if the difference in the average brightness between those adjacent ranges A21 to A28 is less than the average brightness threshold. When the difference in the average brightness between adjacent ranges A21 to A28 is less than the average brightness threshold, it is difficult for a person to perceive the difference between adjacent ranges A21 to A28. By increasing the difference in brightness between adjacent ranges A21 to A28, the operator of the forklift 10 can more easily confirm the situation in front of the forklift 10.

(3) The control device 41 increases the difference in brightness between adjacent ranges A21 to A28 if the difference in the average brightness between those ranges A21 to A28 is less than the average brightness threshold, and the color difference is also less than the color threshold. If the color difference between adjacent ranges A21 to A28 is small, it is difficult for a person to perceive the difference between them. By increasing the difference in brightness between adjacent ranges A21 to A28, the operator of the forklift 10 can more easily check the situation in front of the forklift 10.

[Example of changes]
The embodiment can be implemented with the following modifications. The embodiment and the following modifications can be combined with each other to the extent that they do not contradict each other technically.

The control device 41 may identify the area in image IM1 where the luggage W1 is visible. The control device 41 may then exclude the area in image IM1 where the luggage W1 is visible. In this case, the control device 41 may exclude the area in image IM1 where the luggage W1 is visible after excluding pixels in image IM1 whose brightness is below a threshold. Alternatively, the control device 41 may exclude the area in image IM1 where the luggage W1 is visible before excluding pixels in image IM1 whose brightness is below a threshold. For example, machine learning can be used as a method for identifying the area in image IM1 where the luggage W1 is visible.

The control device 41 may identify areas of significant change by comparing images of the forks 19 without the load W1 with images IM1 acquired from the camera 51. The control device 41 may then exclude these areas of significant change. Images of the forks 19 without the load W1 are provided for each predetermined lifting height. The control device 41 then compares images corresponding to the lifting height of the forks 19 with images IM1 acquired from the camera 51. The control device 41 may exclude areas of significant change from image IM1 after excluding pixels in image IM1 whose brightness is below a threshold. Alternatively, the control device 41 may exclude areas of significant change from image IM1 before excluding pixels in image IM1 whose brightness is below a threshold.

○ After performing the processing in steps S4 to S6, the high-luminance area A1 may be extracted. In this case, the control device 41 may divide the image IM1 into multiple areas and perform the processing in steps S4 to S6.

As shown in Figure 8, the control device 41 may perform the processing in step S14 instead of steps S4 to S6. As an example, as shown in Figure 9, suppose that in step S3 the high-brightness range A1 is divided into a first range A11 and a second range A12. The high-brightness range A1 may be divided into three or more ranges. The first range A11 and the second range A12 are adjacent in the width direction of the image IM1. For example, the first range A11 and the second range A12 are ranges of the same size.

In step S14, the control device 41 performs a brightness adjustment process. The brightness adjustment process is a process that increases the brightness difference between pixels that make up the image IM1 and whose color difference is less than the color difference threshold, when the difference between the maximum and minimum brightness values of multiple pixels that make up the image IM1 is less than the brightness threshold. If the brightness difference between two adjacent pixels is small, it is difficult for a person to perceive the difference between the two pixels. The brightness threshold is set to a value that a person can perceive the difference between two pixels. If the color difference between two adjacent pixels is small, it is difficult for a person to perceive the difference between the two pixels. The color difference threshold is set to a value that a person can perceive the difference between two pixels. Thus, the pixels that are targeted for increasing the brightness difference are pixels that are difficult for a person to perceive the difference between when viewed by a person. The control device 41 then sets the brightness difference between multiple pixels that make up the image IM1 and whose color difference is less than the color difference threshold to a predetermined value or greater. For example, if the color difference between two adjacent pixels is less than the color difference threshold, the control device 41 may set the difference in brightness between those two pixels to be greater than or equal to a predetermined value. The predetermined value is set to be at least greater than or equal to the brightness threshold. In this embodiment, the control device 41 calculates the difference between the maximum brightness and the minimum brightness between multiple divided ranges.

As shown in Figure 9, the minimum brightness value in the first range A11 is 134. The maximum brightness value in the second range A12 is 136. In this case, the difference between the maximum and minimum brightness values in image IM1 is less than the brightness threshold. The control device 41 lowers the brightness in the first range A11. The control device 41 raises the brightness in the second range A12. For example, as shown in Figure 10, the control device 41 sets the brightness of the first range A11 to 119. The control device 41 sets the brightness of the second range A12 to 151. The first range A11 may contain pixels with brightness values different from 134. The second range A12 may contain pixels with brightness values different from 136. The control device 41 may use the midpoint between the minimum brightness value in the first range A11 and the maximum brightness value in the second range A12 as the boundary value, and set the brightness to 119 for pixels with brightness lower than the boundary value, and to 151 for pixels with brightness equal to or greater than the boundary value. Furthermore, the control device 41 may add a value to the brightness of pixels whose brightness is lower than the boundary value. The control device 41 may also subtract a value from the brightness of pixels whose brightness is equal to or greater than the boundary value. The added value and the subtracted value are positive values. The added value and the subtracted value may be the same or different.

In step S7, the control device 41 enlarges the high-brightness range A1, whose brightness has been adjusted by the brightness adjustment process, and displays it on the display unit 61. As a result, the high-brightness range A1 is displayed stretched in the height direction compared to when the entire image IM1 is displayed on the display unit 61.

○The control device 41 does not need to divide the high-brightness range A1 into multiple parts when performing brightness adjustment processing.
○The control device 41 may extract the high-brightness range A1 after performing brightness adjustment processing. In this case, the control device 41 may divide the image IM1 into multiple ranges and perform brightness adjustment processing, or it may perform brightness adjustment processing without dividing the image IM1 into multiple ranges.

○The control device 41 may increase the difference in brightness between adjacent ranges A21 to A28, regardless of color difference, if the difference in the average brightness values between adjacent ranges A21 to A28 is less than the average brightness threshold.

○The forklift 10 may be remotely operated. In this case, the operator operates the forklift 10 from a remote location. The operator operates a remote control device installed at the remote location. A dedicated device may be used as the remote control device, or a portable communication terminal such as a tablet may be used. The remote control device comprises an operating unit for operating the forklift 10, a communication device, a terminal control device, and a terminal display unit. The communication device is configured to communicate with the forklift 10. The terminal control device can acquire the image IM1 captured by the camera 51 via the communication device. The terminal control device displays the image IM1 on the terminal display unit by performing display control similar to that in the embodiment. In this case, the terminal control device is the control device, the terminal display unit is the display unit, and the forklift 10 and the remote control device constitute the display system.

A1…High-brightness range, A11…First range, A12…Second range, DS…Display system, IM1…Image, W1…Luggage, 10…Forklift, 19…Fork, 41…Control device, 51…Camera, 61…Display unit.

Claims (3)

A display system that shows the front of a forklift equipped with forks that can carry loads,
A camera positioned to capture images of the area in front of the forklift from the space between the load placed on the forks and the road surface,
A display unit that displays the image captured by the aforementioned camera,
A control device is provided,
The control device is a display system that displays on the display unit an enlarged high-brightness range of the image, excluding pixels whose brightness is below a threshold. The control device is
The aforementioned image is divided into multiple regions,
The average value of the brightness is calculated for each of the aforementioned multiple ranges,
The display system according to claim 1, wherein if the difference in the average values of the luminances between adjacent ranges among the plurality of ranges is less than the average luminance threshold, the difference in luminances between the adjacent ranges is increased. The control device is
The color difference between adjacent ranges among the aforementioned ranges is calculated,
The display system according to claim 2, wherein if the difference in the average values of the luminances between adjacent ranges among the plurality of ranges is less than the average luminance threshold, and the color difference is less than the color threshold, the difference in luminances between the adjacent ranges is increased.