JP7775779B2 - Cargo Handling System - Google Patents

Description

This disclosure relates to a cargo handling system.

Patent Document 1 discloses a cargo handling vehicle that automatically retrieves cargo under the control of a control device. The cargo handling vehicle disclosed in Patent Document 1 moves to a cargo retrieval position while estimating its own position. Once at the retrieval position, the cargo handling vehicle retrieves the cargo.

Japanese Patent Application Laid-Open No. 2021-160885

When a loading vehicle is to pick up a load, it needs to move close to the pallet. In order to move the loading vehicle close to the pallet, it is necessary to detect the position of the pallet.

A loading and unloading system that solves the above problem is equipped with an external sensor that detects the position of an object using coordinates in a three-dimensional coordinate system, a loading and unloading vehicle equipped with a loading and unloading device, and a control device. The control device detects the position of a pallet from point cloud data, which is a collection of points representing the position of the object, and determines the pallet to be unloaded.

The points in the point cloud data represent the positions of objects, so the control device can detect the position of the pallet from the point cloud data.
In the above-mentioned cargo handling system, the control device may create a side view of the pallet from the point cloud data and detect the position and number of the pallet by extracting the pallet from the side view using image recognition.

In the above-mentioned cargo handling system, the control device may extract the driver's seat of the transport vehicle carrying the pallet from the side view using image recognition, and determine that the pallet farthest from the driver's seat is the pallet to be unloaded.

The present invention makes it possible to detect the position of a pallet.

1 is a schematic diagram of an area in which a loading vehicle operates; FIG. 2 is a perspective view of a transport vehicle and a loading vehicle. FIG. 1 is a schematic configuration diagram of a cargo handling vehicle. 10 is a flowchart showing palette detection control. FIG. 1 is a schematic diagram of a point cloud map. FIG.

An embodiment of the cargo handling system will be described below.
As shown in FIG. 1 , a stop position PS1 is set in the area A1. A transport vehicle 10 stops at the stop position PS1. The area of the stop position PS1 is larger than the area of the transport vehicle 10 when viewed from above. No structures such as pillars are present at the stop position PS1. The area A1 may be, for example, the entire area or a portion of a location such as a factory, port, airport, commercial facility, or public facility. A cargo handling vehicle 20 is operated in the area A1. The cargo handling vehicle 20 performs loading and unloading. Loading is the operation of loading a load C1 placed on a pallet PA1 onto the transport vehicle 10. Unloading is the operation of removing the pallet PA1 loaded on the transport vehicle 10 and the load C1 placed on the pallet PA1 from the transport vehicle 10. In the following description, the front, back, left, right, top, and bottom of the transport vehicle 10 refer to the front, back, left, right, top, and bottom when the transport vehicle 10 is used as the reference. The left, right, top, and bottom of the cargo handling vehicle 20 refer to the left, right, top, and bottom of the cargo handling vehicle 20 as the reference point.

As shown in FIG. 2, the transport vehicle 10 is a wing truck. The transport vehicle 10 may be any type of truck, such as a flatbed truck. The transport vehicle 10 includes a driver's seat 11, a front panel 12, a rear door 13, a cargo bed 14, wing side panels 16, and a tailgate 17.

The driver's seat 11 is a position where the driver of the transport vehicle 10 sits. The front panel 12 is located further rearward of the transport vehicle 10 than the driver's seat 11. The front panel 12 is located adjacent to the driver's seat 11. The rear door 13 is located further rearward of the transport vehicle 10 than the front panel 12. The front panel 12 and the rear door 13 are located at a distance from each other in the fore-and-aft direction of the transport vehicle 10. The loading platform 14 extends in the fore-and-aft direction of the transport vehicle 10 between the front panel 12 and the rear door 13. The loading platform 14 has a loading surface 15. The loading surface 15 is the upper surface of the loading platform 14. Loads C1 placed on pallets PA1 are loaded on the loading surface 15. The wing side panels 16 are located between the front panel 12 and the rear door 13. The wing side panels 16 are rotatable in the vertical direction of the transport vehicle 10 around the center position of the transport vehicle 10 in the vehicle width direction. One wing side panel 16 is provided on each side of the transport vehicle 10 in the vehicle width direction. The tailgate 17 is provided to extend in the front-to-rear direction of the transport vehicle 10. The tailgate 17 is provided along the edge of the loading platform 14, which is the edge extending in the front-to-rear direction of the transport vehicle 10. One tailgate 17 is provided on each side of the transport vehicle 10 in the vehicle width direction.

<Loading vehicle>
The cargo handling vehicle 20 includes a vehicle body 21, drive wheels 22, steering wheels 23, and a cargo handling device 24. The cargo handling device 24 is provided at the front of the vehicle body 21. The cargo handling device 24 includes a mast 25, a lift cylinder 28, a lift bracket 29, and a fork 30.

The mast 25 comprises an outer mast 26 and an inner mast 27. The inner mast 27 is movable up and down relative to the outer mast 26. The forks 30 are fixed to lift brackets 29. The lift brackets 29 and forks 30 move up and down together with the inner mast 27. The lift cylinders 28 raise and lower the inner mast 27. The lift cylinders 28 are hydraulic cylinders.

As shown in FIG. 3, the cargo handling vehicle 20 includes an external sensor 51, a control device 52, an auxiliary memory device 55, a vehicle control device 56, a travel actuator 59, a cargo handling actuator 60, and a camera 61. The cargo handling vehicle 20 is a cargo handling system.

The external sensor 51 detects the position of an object using coordinates in a three-dimensional coordinate system. The external sensor 51 is provided on the top of the cargo handling vehicle 20. For example, the external sensor 51 is provided on the head guard of the cargo handling vehicle 20.

Examples of the external sensor 51 include millimeter-wave radar, a stereo camera, a ToF (Time of Flight) camera, and a LIDAR (Laser Imaging Detection and Ranging). In this embodiment, a LIDAR is used as the external sensor 51. The external sensor 51 irradiates the surrounding area with a laser and derives the distance to the point by receiving the light reflected from the point where the laser hits. The point where the laser hits represents a portion of the surface of the object. The position of the point can be expressed by coordinates in a polar coordinate system. The coordinates of the point in the polar coordinate system are converted to coordinates in a Cartesian coordinate system. The conversion from the polar coordinate system to the Cartesian coordinate system may be performed by the external sensor 51 or by the control device 52. In this embodiment, it is assumed that the conversion from the polar coordinate system to the Cartesian coordinate system is performed by the external sensor 51. The external sensor 51 derives the coordinates of the point in the sensor coordinate system. The sensor coordinate system is a three-axis Cartesian coordinate system with the external sensor 51 as the origin. The external sensor 51 outputs the coordinates of multiple points obtained by irradiating the laser to the control device 52 as point cloud data.

The control device 52 includes a processor 53 and a memory unit 54. The memory unit 54 includes RAM (Random Access Memory) and ROM (Read Only Memory). The memory unit 54 stores program code or instructions configured to cause the processor 53 to execute processes. The memory unit 54, i.e., computer-readable medium, includes any available medium accessible by a general-purpose or special-purpose computer. The control device 52 may be configured with hardware circuits such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control device 52, which is a processing circuit, may include one or more processors that operate according to a computer program, one or more hardware circuits such as an ASIC or FPGA, or a combination thereof.

The auxiliary storage device 55 stores information that can be read by the control device 52. Examples of the auxiliary storage device 55 include a hard disk drive and a solid state drive. The auxiliary storage device 55 stores an environmental map D1. The auxiliary storage device 55 stores a first image recognition model D2. The auxiliary storage device 55 stores a second image recognition model D3.

Environmental map D1 is information about the physical structure of area A1, such as the shapes of objects present in area A1 and the size of area A1. In this embodiment, environmental map D1 is data that represents the structure of area A1 using coordinates in a map coordinate system. The map coordinate system is a three-axis Cartesian coordinate system. The map coordinate system is a coordinate system with any one point in area A1 as its origin. In the map coordinate system, the horizontal direction is defined by the X-axis and Y-axis, which are perpendicular to each other. The XY plane defined by the X-axis and Y-axis can be said to represent the horizontal plane. In the map coordinate system, the up-down direction is defined by the Z-axis, which is perpendicular to the X-axis and Y-axis. Coordinates in the map coordinate system will be referred to as map coordinates where appropriate. The map coordinate system is a three-dimensional coordinate system that represents three-dimensional positions.

The control device 52 estimates the self-position of the cargo handling vehicle 20. The control device 52 outputs the self-position of the cargo handling vehicle 20 to the vehicle control device 56. The self-position is the position of the cargo handling vehicle 20 on the environmental map D1. The self-position is a coordinate indicating a point on the cargo handling vehicle 20 in the map coordinate system. The point on the cargo handling vehicle 20 can be any point, but could be, for example, the center position of the cargo handling vehicle 20 in the horizontal direction.

The self-location is estimated by comparing the detection results of the external sensor 51 with the environmental map D1. The control device 52 extracts landmarks from the environmental map D1 that have the same shape as the landmarks obtained from the point cloud data. The control device 52 recognizes the positions of the landmarks from the environmental map D1. The positional relationship between the landmark positions and the loading vehicle 20 can be determined from the detection results of the external sensor 51. Therefore, the control device 52 can estimate the self-location by recognizing the positions of the landmarks. A landmark is an object that has characteristics that can be identified by the external sensor 51. A landmark is a physical structure whose position is unlikely to change. Examples of landmarks include walls and pillars. The self-location may be estimated by combining self-location estimation using the external sensor 51 with dead reckoning using an internal sensor. The self-location may be estimated by combining self-location estimation using the external sensor 51 with self-location estimation using satellite signals transmitted from GNSS (Global Navigation Satellite System) satellites.

The vehicle control device 56 has, for example, the same hardware configuration as the control device 52. The vehicle control device 56 has a processor 57 and a storage unit 58.
The travel actuator 59 is an actuator that causes the cargo handling vehicle 20 to travel. The travel actuator 59 includes, for example, a motor that rotates the drive wheels 22 and a steering mechanism. The vehicle control device 56 controls the travel actuator 59 while determining its own position to cause the cargo handling vehicle 20 to travel.

The cargo handling actuator 60 is an actuator that causes the cargo handling vehicle 20 to handle cargo. The cargo handling actuator 60 includes, for example, a motor that drives a pump that supplies hydraulic oil to the hydraulic equipment, and a control valve that controls the supply of hydraulic oil. The hydraulic equipment includes the lift cylinder 28. The vehicle control device 56 raises and lowers the fork 30 by controlling the cargo handling actuator 60.

The cargo handling vehicle 20 travels automatically by controlling the travel actuator 59 with the vehicle control device 56. The cargo handling vehicle 20 automatically handles cargo by controlling the cargo handling actuator 60 with the vehicle control device 56. The cargo handling vehicle 20 is an autonomous forklift.

The camera 61 is positioned to capture an image in front of the cargo handling vehicle 20. The camera 61 is a monocular camera. The camera 61 is installed, for example, between the two forks 30 in the vehicle width direction of the cargo handling vehicle 20. The camera 61 is installed so as to rise and fall together with the forks 30. For example, the camera 61 is attached to the lift bracket 29.

The pallet detection control performed by the control device 52 will now be described. Pallet detection control is performed to move the cargo handling vehicle 20 closer to the pallet PA1 when the cargo handling vehicle 20 picks up cargo. The cargo handling vehicle 20 approaches the transport vehicle 10, for example, based on commands from a higher-level control device. Then, when the distance between the cargo handling vehicle 20 and the transport vehicle 10 becomes less than a predetermined distance, pallet detection control is initiated. The predetermined distance is, for example, the distance that the transport vehicle 10 can detect using the external sensor 51.

<Pallet detection control>
As shown in FIGS. 4 and 5 , in step S1, the control device 52 creates a point cloud map PM1. The point cloud map PM1 is obtained by overlaying point cloud data obtained from the detection results of the external sensor 51. The control device 52 acquires point cloud data from the external sensor 51 multiple times while the loading vehicle 20 is moving. The control device 52 converts the coordinates of each point P1 in the point cloud data from the coordinates of the sensor coordinate system to map coordinates based on its own position. The control device 52 can recognize the origin of the sensor coordinate system in the map coordinate system based on its own position estimated by the control device 52. The control device 52 can recognize the deviation between the coordinate axes of the map coordinate system and the coordinate axes of the sensor coordinate system based on the origin of the sensor coordinate system in the map coordinate system and the deviation between the coordinate axes of the map coordinate system and the coordinate axes of the sensor coordinate system. The control device 52 creates the point cloud map PM1 by overlaying each point P1 converted to map coordinates each time point cloud data is acquired. The point cloud map PM1 is a collection of point cloud data. Compared to the points P1 of the point cloud data, the points P1 of the point cloud map PM1 are denser.

Figure 5 shows the point cloud map PM1 obtained by the processing of step S1. Each point P1 included in the point cloud map PM1 represents the map coordinates of an object. For ease of explanation, the points P1 included in the point cloud map PM1 will be classified into a first point P11, a second point P12, a third point P13, and a fourth point P14. The first point P11 is a point P1 obtained by irradiating the gate 17 with a laser. The second point P12 is a point P1 obtained by irradiating the driver's seat 11 with a laser. The third point P13 is a point P1 obtained by irradiating the pallet PA1 and cargo C1 loaded on the loading surface 15 with a laser. The fourth point P14 is a point P1 that does not fall into any of the first point P11, second point P12, or third point P13.

As shown in Figures 4 and 6, in step S2, the control device 52 creates a side view IM1 from the point cloud map PM1. The side view IM1 is a view in which point P1 is projected in the vehicle width direction of the transport vehicle 10. This makes it possible to make the point cloud map PM1 two-dimensional, so the side view IM1 can be treated as image data. The side view IM1 can also be considered a side view of the pallet PA1. An example of the processing performed by the control device 52 when creating the side view IM1 will be described below.

The control device 52 calculates the normal vector of each point P1. A normal vector is a vector oriented in a direction perpendicular to a plane surrounded by multiple points P1. Methods for deriving a normal vector include a method of determining a curved surface from each point P1 and deriving the normal vector of each point P1 from the curved surface, and a method using the cross product of vectors. For example, when the control device 52 determines the normal vector of one point P1, it determines the cross product of the vectors pointing from this point P1 to each of two points P1 located within a specified range. This cross product is the normal vector.

The control device 52 extracts point P1 where the normal vector is oriented horizontally. For example, the control device 52 determines whether the angle of the normal vector with respect to the XY plane of the map coordinate system is within a predetermined range. If the normal vector is oriented horizontally, the normal vector is parallel to the XY plane of the map coordinate system. Taking into account the inclination of the transport vehicle 10 and measurement errors, a predetermined range is set so that point P1 where the normal vector can be considered to be oriented horizontally can be extracted.

The control device 52 derives a plane equation from point P1, where the normal vector is oriented horizontally. The plane equation can be derived using, for example, a robust estimation method such as RANSAC (Random Sample Consensus) or the least squares method. The plane represented by the plane equation is a surface that extends vertically. In the case of the transport vehicle 10 of this embodiment, the plane can be obtained from the first point P11 defined by the gate 17. Furthermore, when loading goods, since the transport vehicle 10 is loaded with a pallet PA1 and a load C1, the plane can be obtained from the third point P13 defined by the pallet PA1 and the load C1.

The control device 52 creates a side view IM1 by projecting point P1 in a direction perpendicular to the plane represented by the plane equation. In this case, the control device 52 creates the side view IM1 by projecting point P1 within a predetermined range horizontally from the plane represented by the plane equation. As shown in FIG. 1, pallets PA1 may be loaded on the transport vehicle 10, lined up in the width direction of the transport vehicle 10. The predetermined range PT1 is set so as not to include pallets PA1 lined up in the width direction of the transport vehicle 10 that are far from the cargo handling vehicle 20. The predetermined range PT1 is set so as to include the surfaces of pallets PA1 lined up in the width direction of the transport vehicle 10 that are close to the cargo handling vehicle 20, facing the cargo handling vehicle 20. When the cargo handling vehicle 20 retrieves cargo, it retrieves the pallets PA1 lined up in the width direction of the transport vehicle 10 that are close to the cargo handling vehicle 20. By setting the predetermined range PT1 as described above, it is possible to obtain a side view IM1 that excludes pallets PA1 that are not to be picked up by the cargo handling vehicle 20.

As shown in Figures 4 and 6, next, in step S3, the control device 52 detects the position of pallet PA1 and the number of pallets PA1. The position of pallet PA1 is the position of pallet PA1 that is the target of loading by the cargo handling vehicle 20. The number of pallets PA1 is the number of pallets PA1 that are the target of loading by the cargo handling vehicle 20. The position of pallet PA1 is the coordinates of pallet PA1 in the map coordinate system. The position of pallet PA1 can be obtained by converting the coordinates of pallet PA1 in the side view IM1 into coordinates in the map coordinate system. A detailed explanation will be provided below.

The control device 52 determines the position of pallet PA1 in the side view IM1 by extracting pallet PA1 from the side view IM1. The position of pallet PA1 in the side view IM1 can be represented by an image coordinate system. The image coordinate system is a coordinate system in which the horizontal direction of the side view IM1 is the X axis and the vertical direction is the Y axis. The position of pallet PA1 is determined using image recognition. In this embodiment, the case where image recognition is performed using the first image recognition model D2 is described, but image recognition may also be performed by pattern matching.

The first image recognition model D2 is a trained model generated by machine learning. The first image recognition model D2 uses an algorithm capable of determining the class of an object on a region-by-region basis. The class is set to "palette." Examples of machine learning algorithms include SSD (Single Shot Multibox Detector), R-CNN (Regional Convolutional Neural Network), fast R-CNN, faster R-CNN, and YOLO (You Only Look Once). The first image recognition model D2 is generated by supervised learning or semi-supervised learning using training data. The training data includes image data containing an object corresponding to the class, as well as the position and label of the object in the image data. The training data can be generated, for example, by surrounding the object in the image data with a frame and labeling the image data. In this embodiment, the training data can be generated by surrounding the palette PA1 of the image data with a frame and labeling the image data as "palette." The image data used as training data may be obtained from the point cloud map PM1, as with the side view IM1, or may be obtained by capturing images using an imaging device.

The first image recognition model D2 identifies the area containing pallet PA1 from the input side view IM1. The area containing pallet PA1 is represented by bounding boxes B1 and B2. In the example shown in Figure 6, two bounding boxes B1 and B2 are extracted from the side view IM1. The bounding boxes B1 and B2 surround pallet PA1, which is represented by the third point P13. The number of bounding boxes B1 and B2 represents the number of pallets PA1. The positions of the bounding boxes B1 and B2 in the side view IM1 represent the position of pallet PA1 in the image coordinate system. Because the side view IM1 is created by projecting point P1 in the map coordinate system, coordinates in the map coordinate system are associated with point P1 constituting the side view IM1. The control device 52 can detect the position of pallet PA1 in the map coordinate system from the position of pallet PA1 in the side view IM1. Because the side view IM1 is created from point cloud data, it can be said that the control device 52 detects the positions and number of pallets PA1 from the point cloud data.

Next, in step S4, the control device 52 determines the position of the driver's seat 11 in the side view IM1. The position of the driver's seat 11 in the side view IM1 is determined using image recognition. In this embodiment, the case where image recognition is performed using the second image recognition model D3 is described, but image recognition may also be performed by pattern matching.

The second image recognition model D3 is a trained model generated by machine learning. The second image recognition model D3 is generated using the same method as the first image recognition model D2. In the second image recognition model D3, "driver's seat" is set as the class. As a result, the second image recognition model D3 identifies the area that includes the driver's seat 11 from the input side view IM1. In the example shown in FIG. 6, a bounding box B3 is extracted from the side view IM1. In the example shown in FIG. 6, the driver's seat 11, represented by the second point P12, is surrounded by the bounding box B3. The bounding box B3 represents the position of the driver's seat 11 in the side view IM1.

Next, in step S5, the control device 52 determines the pallet PA1 to be picked up. If there are multiple pallets PA1, the control device 52 determines the pallet PA1 to be picked up first. When the cargo handling vehicle 20 picks up cargo, it sequentially picks up the pallets PA1 loaded on the loading platform 14 of the transport vehicle 10 from the rear to the front of the transport vehicle 10. The control device 52 determines that the pallet PA1 farthest from the driver's seat 11 is the pallet PA1 to be picked up first. In this embodiment, the pallet PA1 represented by the bounding box B1 is the pallet PA1 to be picked up first. The pallet PA1 represented by the bounding box B2 is the pallet PA1 to be picked up second.

Next, in step S6, the control device 52 transmits the coordinates of pallet PA1 in the map coordinate system, i.e., the position of pallet PA1, to the vehicle control device 56. The control device 52 may transmit only the position of the pallet PA1 that is to be first picked up to the vehicle control device 56. The control device 52 may also transmit the positions of all detected pallets PA1 to the vehicle control device 56.

After completing the process of step S6, the control device 52 ends the palette detection control.
The vehicle control device 56 controls the travel actuator 59 in accordance with the position of the pallet PA1 transmitted from the control device 52. The vehicle control device 56 generates a route to the pallet PA1 from which the first load is to be taken. At this time, the vehicle control device 56 generates a route to an approach start position that is a predetermined distance before the pallet PA1. The predetermined distance is set to a distance that allows the camera 61 to capture an image of the pallet PA1. The vehicle control device 56 controls the travel actuator 59 so that the cargo handling vehicle 20 moves along the route. When the cargo handling vehicle 20 reaches the approach start position, the vehicle control device 56 moves the cargo handling vehicle 20 closer to the pallet PA1 while checking the position of the pallet PA1 using image data obtained by the camera 61. When the cargo handling vehicle 20 reaches a position where the pallet PA1 is to be taken, the vehicle control device 56 controls the cargo handling actuator 60 to take the load. The vehicle control device 56 checks the position of the pallet PA1 using image data obtained by the camera 61 and adjusts the position of the fork 30 to match the pallet PA1. This allows the cargo handling vehicle 20 to take the load.

[Effects of this embodiment]
(1) The control device 52 can detect the position of the pallet PA1 from the point cloud data. This allows the control device 52 to determine the pallet PA1 to be picked up.

(2) The control device 52 can detect the position of pallet PA1 from the point cloud data. This allows the vehicle control device 56 to generate a route toward pallet PA1. The camera 61 is used to move the cargo handling vehicle 20 from the approach start position to the position where pallet PA1 is unloaded. Because the detection range of the camera 61 is limited, it is necessary to move the cargo handling vehicle 20 to the approach start position with high accuracy. By detecting the position of pallet PA1 from the point cloud data and generating a route, the cargo handling vehicle 20 can be moved to the approach start position with high accuracy.

It is also possible to set a route in advance and move the loading vehicle 20 to a position where the pallet PA1 is to be unloaded. However, in this case, it would be necessary to set a route for each size of the transport vehicle 10, size of the pallet PA1, and type of pallet PA1, which would require a huge amount of work. In contrast, by detecting the position of the pallet PA1 from point cloud data and generating a route, it is possible to generate a route that corresponds to the position of the pallet PA1. This eliminates the need to set a route in advance.

(3) The control device 52 detects the position of pallet PA1 and the number of pallets PA1 from the side view IM1 through image recognition. By converting the point cloud data into the side view IM1, the side view IM1 can be treated as image data. This allows the control device 52 to detect the position of pallet PA1 in the side view IM1 and the number of pallets PA1 through image recognition. Then, from the position of pallet PA1 in the side view IM1, the position of pallet PA1 in the map coordinate system can be detected.

(4) The control device 52 extracts the driver's seat 11 from the side view IM1 using image recognition. When the cargo handling vehicle 20 retrieves cargo, it sequentially retrieves cargo C1 loaded on the loading platform 14 of the transport vehicle 10 from the rear to the front of the transport vehicle 10. Because the driver's seat 11 of the transport vehicle 10 is located at the front of the transport vehicle 10, by extracting the driver's seat 11 of the transport vehicle 10, it can determine that the pallet PA1 farthest from the driver's seat 11 is the pallet PA1 to be retrieved first.

(5) The external sensor 51 used to estimate the self-position can be used to detect the position of pallet PA1 and the number of pallets PA1. This reduces the number of parts compared to using dedicated sensors to detect the position of pallet PA1 and the number of pallets PA1.

[Example of change]
The embodiment can be modified as follows: The embodiment and the following modifications can be combined with each other to the extent that they are not technically inconsistent.

The control device 52 may detect the position of pallets PA1 and the number of pallets PA1 from the point cloud map PM1. In this case, for example, the position of pallets PA1 and the number of pallets PA1 are detected from the point cloud map PM1 using a model that has learned the characteristics of the three-dimensional data representing pallets PA1.

In step S3, the control device 52 only needs to detect the position of the pallet PA1, and does not need to detect the number of pallets PA1.
The control device 52 does not have to create the point cloud map PM1. In this case, the control device 52 converts the point cloud data acquired from the external sensor 51 into map coordinates and performs the processes from step S2 onwards. In other words, the control device 52 may detect the position and number of pallets PA1 from a single point cloud data, without creating the point cloud map PM1 which is a collection of multiple point cloud data.

The vehicle control device 56 may generate a route to the position where the cargo handling vehicle 20 will pick up the pallet PA1. In this case, the image data obtained by the camera 61 is used to adjust the position of the forks 30.

○ If there is a space adjacent to the pallet PA1 in the horizontal direction, the control device 52 may determine that the pallet PA1 is the first pallet PA1 to be loaded. The pallet PA1 to be loaded first does not have an adjacent pallet PA1 behind the transport vehicle 10. Therefore, the pallet PA1 to be loaded first has a space adjacent to it in the horizontal direction that is equal to or greater than the threshold. By detecting this space, the pallet PA1 to be loaded first can be determined. If cushioning material is present between adjacent pallets PA1, the threshold is set taking into account the thickness of the cushioning material. If there is no cushioning material between adjacent pallets PA1, the threshold may be set based on the width of the pallet PA1. For example, half the width of the pallet PA1 may be used as the threshold. The control device 52 may detect the space from the side view IM1 or from the point cloud map PM1. Because point P1 does not exist in the space, the control device 52 can detect the space based on the presence or absence of point P1.

The second image recognition model D3 may be an algorithm capable of determining the class of an object for each side view IM1. In this case, "right-facing" and "left-facing" may be set as classes. An example of a machine learning algorithm is a convolution neural network (CNN). The training data may be data in which image data showing a transport vehicle 10 facing right is labeled "right-facing" and data in which image data showing a transport vehicle 10 facing left is labeled "left-facing."

By inputting the side view IM1 into the second image recognition model D3, the control device 52 determines whether the transport vehicle 10 shown in the side view IM1 is facing right or left. If the transport vehicle 10 is facing right, the control device 52 determines that the pallet PA1 located at the leftmost position in the image coordinate system is the pallet PA1 to be loaded. If the transport vehicle 10 is facing left, the control device 52 determines that the pallet PA1 located at the rightmost position in the image coordinate system is the pallet PA1 to be loaded.

○ If the stopping direction of the transport vehicle 10 is constant, the pallet PA1 to be picked up may be determined from the map coordinates of the pallet PA1. If the stopping direction of the transport vehicle 10 is constant, the pallet PA1 loaded further rearward on the transport vehicle 10 will have a larger or smaller horizontal coordinate in the map coordinate system. Whether the pallet PA1 loaded further rearward on the transport vehicle 10 will have a larger or smaller horizontal coordinate in the map coordinate system depends on the positional relationship between the origin of the map coordinate system and the transport vehicle 10. The control device 52 can determine the pallet PA1 to be picked up based on the magnitude of the map coordinates. For example, if the pallet PA1 loaded further rearward on the transport vehicle 10 has a larger X coordinate in the map coordinate system, the control device 52 can determine that the pallet PA1 with the largest X coordinate in the map coordinate system is the pallet PA1 to be picked up.

Part of the pallet detection control process may be performed by the vehicle control device 56. In this case, the vehicle control device 56 can also be considered a control device.
A part of the processing of the pallet detection control may be performed by a host control device that issues commands to the cargo handling vehicle 20. In this case, the host control device can also be considered a control device. All of the processing of the pallet detection control may be performed by a host control device that issues commands to the cargo handling vehicle 20. In this case, the host control device is the control device. In this way, when at least a part of the processing of the pallet detection control is performed by the host control device, the control device 52 transmits data required for the processing to the host control device via wireless communication. For example, when all of the processing of the pallet detection control is performed by the host control device, the control device 52 transmits point cloud data to the host control device. The host control device transmits the results obtained by performing the pallet detection control to the control device 52 via wireless communication. The host control device is installed, for example, in area A1. In this case, the cargo handling system is made up of the cargo handling vehicle 20 and the host control device.

The cargo handling vehicle 20 may also pick up pallets PA1 placed on a shelf or on the ground.
The transport vehicle 10 may be any vehicle capable of transporting goods, such as an AGV (Automatic Guided Vehicle) or an AMR (Autonomous Mobile Robot).

The cargo handling vehicle 20 may be provided with a laser range finder instead of the camera 61 as a sensor for adjusting the position of the fork 30 .
The cargo handling device 24 may include, for example, a robot arm.

P1...point, PA1...pallet, 10...transport vehicle, 11...driver's seat, 20...cargo vehicle which is a cargo handling system, 24...cargo handling device, 51...external sensor, 52...control device.

Claims (2)

a cargo handling vehicle equipped with an external sensor that detects the position of an object in coordinates of a three-dimensional coordinate system and a cargo handling device;
A cargo handling system comprising:
The control device
extracting the points whose normal vectors are oriented horizontally from point cloud data, which is a set of points representing the positions of the objects;
Deriving a plane equation from the point where the normal vector is oriented horizontally;
creating a side view by projecting the points of the point cloud data within a predetermined range from the plane represented by the plane equation in a direction perpendicular to the plane;
extracting pallets from the side view by image recognition to detect the positions of the pallets and the number of the pallets;
A material handling system that determines the pallet to be picked up. The control device
extracting a driver's seat of the transport vehicle carrying the pallet from the side view by image recognition;
The cargo handling system according to claim 1 , wherein the pallet farthest from the driver's seat is determined to be the pallet to be picked up.