JP7748811B2 - Information processing device, control method, program, and storage medium - Google Patents

Description

This disclosure relates to processing when a ship comes to berth.

Technologies for assisting ships in docking (berthing) have been known for some time. For example, Patent Document 1 describes a method for controlling an automatic docking device that automatically docks ships by changing the ship's attitude so that light emitted from a lidar is reflected by objects around the docking position and received by the lidar.

Japanese Patent Application Laid-Open No. 2020-59403

When assisting ships that dock based on sensor output without using self-position estimation or map data, it is necessary to accurately determine the docking situation, such as whether nearby structures are suitable for docking, and reflect this information in ship control, etc.

This disclosure has been made to solve the above-mentioned problems, and its main objective is to provide an information processing device that can appropriately determine the docking situation.

The claimed invention is
an acquisition means for acquiring measurement data in a direction toward a berthing location, the measurement data being generated by a measurement device provided on the ship;
an identification means for identifying a detection plane, which is the plane of the docking location detected by the measurement data, and a vertical number of the vertical field of view angle of the measurement device corresponding to the upper end or lower end of the detection plane, based on a normal vector calculated from the measurement data;
a docking status determination means for determining a docking status regarding the docking location based on the result of identification by the identification means;
The information processing device has the following.

The claimed invention also includes:
A computer-implemented control method comprising:
Acquire measurement data in the direction of the docking location generated by a measuring device installed on the ship,
based on the normal vector calculated from the measurement data, a detection plane, which is the plane of the docking location detected by the measurement data, and a vertical number of the vertical field of view angle of the measurement device corresponding to the upper end or lower end of the detection plane are identified;
determining a docking situation regarding docking at a docking location based on the result of identifying the detection surface and the vertical number ;
It is a control method.

The claimed invention also includes:
Acquire measurement data in the direction of the docking location generated by a measuring device installed on the ship,
based on the normal vector calculated from the measurement data, a detection plane, which is the plane of the docking location detected by the measurement data, and a vertical number of the vertical field of view angle of the measurement device corresponding to the upper end or lower end of the detection plane are identified;
The program causes a computer to execute a process for determining the docking status of the docking location based on the result of identifying the detection surface and the vertical number .

FIG. 1 is a schematic configuration diagram of a driving assistance system. FIG. 2 is a block diagram showing a hardware configuration of the information processing device. FIG. 2 is a functional block diagram relating to a docking assistance process. FIG. 10 is a perspective view of a structure showing the calculated normal vectors. This is a diagram showing a docking situation in which both the top and side surfaces of the structure at the docking location are included as the inner field of view and detection surfaces. 1A and 1B are diagrams showing a docking situation in which the inner field of view is the top and side surfaces, while the detection surface is only the side surfaces, respectively. 1A and 1B are diagrams illustrating a docking situation in which the inner field of view and the detection surface are both only the side and top surfaces, respectively. FIG. 1 is a perspective view of a structure showing neighboring points and nearest neighboring points. FIG. 10 is a diagram illustrating an outline of a method for calculating the distance to the opposite shore. FIG. 10 is a diagram showing an outline of a method for calculating a docking speed. FIG. 10 is a diagram illustrating an outline of a method for calculating an approach angle. 10 shows an example of a data structure of reliability information. 10 is an example of a flowchart illustrating an outline of a docking assistance process. FIG. 10 is a diagram showing docking situation B. This is a diagram showing docking situation C. An example in which a pier exists in berthing situation C is shown.

According to a preferred embodiment of the present disclosure, an information processing device includes an acquisition means for acquiring measurement data in the direction toward the docking location generated by a measurement device installed on the ship; an identification means for identifying, based on a normal vector calculated from the measurement data, an inner field of view, which is the surface of the docking location that exists within the field of view of the measurement device, and a detection surface, which is the surface of the docking location detected by the measurement data; and a docking status determination means for determining the docking status of the docking location based on the identification results by the identification means. According to this aspect, the information processing device can accurately determine the docking status based on the detection status of the surface of the docking location by the measurement device.

In one aspect of the information processing device, the information processing device has a normal vector calculation means that calculates the normal vector based on data obtained by removing water surface reflection data from the measurement data using the water surface position as a reference. This aspect allows the information processing device to accurately calculate the normal vector of the docking location based on data obtained by accurately removing erroneous detection data that indicates a position below the water surface position and is generated due to water surface reflection.

In another aspect of the information processing device, the information processing device includes a normal vector calculation means that calculates the normal vector for each measurement point that constitutes the downsampled measurement data. This aspect allows the information processing device to efficiently calculate the required normal vectors while reducing the processing load.

In another aspect of the information processing device, the identification means determines whether the top surface of the docking location is the detection surface based on a vertical normal vector, and determines whether the side surface of the docking location is the detection surface based on a horizontal normal vector. This aspect allows the information processing device to accurately determine whether the top surface and side surface of the docking location are each detection surfaces.

In another aspect of the information processing device, the docking status determination means determines the docking status based on a combination of whether the top surface and the side surface are each within the field of view, and whether the top surface and the side surface are each within the detection surface. This aspect allows the information processing device to appropriately determine the docking status, such as whether the detected object is suitable as a docking location.

In another aspect of the information processing device, the information processing device further includes reliability information generation means for generating reliability information indicating the reliability of the berthing location based on the results of the determination of the berthing situation. This aspect enables the information processing device to suitably generate reliability information useful for controlling the ship during berthing, etc.

In another aspect of the information processing device, the information processing device further includes a shore distance calculation means for calculating the shore distance, which is the distance from the ship to the docking location, based on the normal vector, and an approach angle calculation means for calculating the approach angle of the ship relative to the docking location based on the shore distance, and the reliability information generation means generates the reliability information including a reliability of the shore distance based on a change in the shore distance over time, and a reliability of the approach angle based on a change in the approach angle over time. This aspect allows the information processing device to accurately calculate the shore distance and the approach angle, and to suitably generate reliability information including these reliabilities.

In another aspect of the information processing device, the information processing device further includes a nearby point search means for determining a predetermined number of nearby points at the docking location relative to the ship based on the measurement data, and a nearest point determination means for determining the nearest point at the docking location relative to the ship from the nearby points, wherein the opposite shore distance calculation means calculates the opposite shore distance based on the nearest point and the normal vector, and the reliability information generation means generates the reliability information including the reliability of the nearest point based on the variance of the nearest points. This aspect allows the information processing device to preferably search for the nearest point to the docking location required for calculating the opposite shore distance, and preferably generate reliability information including the reliability of the nearest point.

According to another preferred embodiment of the present disclosure, a computer-implemented control method acquires measurement data in the direction of the docking location generated by a measurement device installed on a ship, identifies an inner field of view, which is the surface of the docking location present in the field of view of the measurement device, and a detection surface, which is the surface of the docking location detected by the measurement data, based on a normal vector calculated from the measurement data, and determines the docking status of the docking location based on the identification of the inner field of view and the detection surface. By executing this control method, the computer can accurately determine the docking status based on the detection status of the surface of the docking location by the measurement device.

According to another preferred embodiment of the present disclosure, there is provided a program that causes a computer to execute a process of acquiring measurement data in the direction of the docking location generated by a measurement device installed on a ship, identifying an inner field of view, which is the surface of the docking location present in the field of view of the measurement device, and a detection surface, which is the surface of the docking location detected by the measurement data, based on a normal vector calculated from the measurement data, and determining the docking status of the docking location based on the identification of the inner field of view and the detection surface. By executing this program, the computer can accurately determine the docking status based on the detection status of the surface of the docking location by the measurement device. Preferably, the program is stored on a storage medium.

Preferred embodiments of the present invention will be described below with reference to the drawings. For convenience, any symbol followed by a "^" or "-" will be represented in this specification as "A^" or " ^A- " (where "A" is any character).

(1) Overview of the driving assistance system
1 shows a schematic configuration of a driving assistance system according to this embodiment. The driving assistance system includes an information processing device 1 that moves together with a ship, which is a moving body, and a sensor group 2 mounted on the ship. Hereinafter, the ship that moves together with the information processing device 1 will also be referred to as the "target ship."

The information processing device 1 is electrically connected to the sensor group 2 and, based on the output of various sensors included in the sensor group 2, provides operational support such as automatic operation control of the target ship on which the information processing device 1 is installed. Operational support also includes docking support such as automatic berthing (docking). Here, "docking" includes docking the target ship to a quay as well as docking the target ship to a structure such as a pier. Furthermore, hereafter, "docking location" is a general term for structures such as quays and piers that are the target for docking. The information processing device 1 may be a navigation device installed on the target ship, or an electronic control device built into the ship.

The sensor group 2 includes various external and internal sensors installed on the target vessel. In this embodiment, the sensor group 2 includes at least a lidar (Light Detection and Ranging, or Laser Illuminated Detection and Ranging) 3.

The LIDAR 3 emits a pulsed laser beam over a predetermined angular range in the horizontal and vertical directions to discretely measure the distance to an object in the external environment and generate three-dimensional point cloud data indicating the object's position. In this case, the LIDAR 3 includes an irradiation unit that emits laser light while changing the irradiation direction, a light receiving unit that receives reflected (scattered) light from the emitted laser light, and an output unit that outputs scan data based on the light receiving signal output by the light receiving unit. The data measured for each direction of laser light irradiation (scanning position) (also referred to as "measurement point" or "measurement point data") is generated based on the irradiation direction corresponding to the laser light received by the light receiving unit and the response delay time of the laser light determined based on the light receiving signal. The LIDAR 3 is not limited to the scanning LIDAR described above, but may also be a flash LIDAR that generates three-dimensional data by irradiating a diffused laser beam within the field of view of a two-dimensional array sensor. The LIDAR 3 is an example of a "measurement device" within the scope of this invention.

(2) Configuration of the information processing device
2 is a block diagram showing an example of the hardware configuration of the information processing device 1. The information processing device 1 mainly includes an interface 11, a memory 12, and a controller 13. These elements are connected to each other via a bus line.

The interface 11 performs interface operations related to the exchange of data between the information processing device 1 and external devices. In this embodiment, the interface 11 acquires output data from each sensor in the sensor group 2 and supplies it to the controller 13. The interface 11 also supplies, for example, signals related to the control of the target vessel generated by the controller 13 to each component of the target vessel that controls the operation of the target vessel. For example, the target vessel may include a drive source such as an engine or electric motor, a screw that generates forward thrust based on the drive force of the drive source, a thruster that generates lateral thrust based on the drive force of the drive source, and a rudder, which is a mechanism for freely determining the direction of travel of the vessel. During automatic operation, such as automatic docking, the interface 11 supplies control signals generated by the controller 13 to each of these components. If the target vessel is equipped with an electronic control device, the interface 11 supplies the control signals generated by the controller 13 to the electronic control device. The interface 11 may be a wireless interface, such as a network adapter, for wireless communication, or a hardware interface for connecting to an external device via a cable or the like. The interface 11 may also perform interface operations with various peripheral devices such as input devices, display devices, and sound output devices.

Memory 12 is composed of various types of volatile and non-volatile memory, such as RAM (Random Access Memory), ROM (Read Only Memory), a hard disk drive, and flash memory. Memory 12 stores programs that allow controller 13 to execute predetermined processes. Note that the programs executed by controller 13 may be stored in storage media other than memory 12.

In addition, the memory 12 stores information necessary for the processing performed by the information processing device 1 in this embodiment. For example, the memory 12 may include map data including information regarding the location of the docking location. In another example, the memory 12 stores information regarding the downsampling size when downsampling is performed on the point cloud data obtained when the lidar 3 performs one scanning cycle.

The controller 13 includes one or more processors, such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), and TPU (Tensor Processing Unit), and controls the entire information processing device 1. In this case, the controller 13 executes programs stored in the memory 12, etc., to perform processing related to operational support for the target ship, etc.

Functionally, the controller 13 also includes a docking location detection unit 15 and a docking parameter calculation unit 16. The docking location detection unit 15 performs processing related to docking location detection based on point cloud data output by the LIDAR 3. The docking parameter calculation unit 16 calculates parameters (also referred to as "docking parameters") required for docking at the docking location. Here, the docking parameters include the distance to the docking location (docking distance), the approach angle to the docking location, and the speed at which the vessel approaches the docking location (docking speed). The docking parameter calculation unit 16 also calculates information (also referred to as "reliability information") indicating the reliability of docking at the docking location based on the processing results of the docking location detection unit 15 and the docking parameters. The controller 13 functions as an "acquisition means," "identification means," "normal vector calculation means," "docking distance calculation means," "approach angle calculation means," "nearby point search means," "nearest neighbor point determination means," "docking situation determination means," "reliability information generation means," and a computer that executes programs.

The processing performed by the controller 13 is not limited to being realized by software programs, but may also be realized by any combination of hardware, firmware, and software. Furthermore, the processing performed by the controller 13 may also be realized using a user-programmable integrated circuit, such as an FPGA (Field-Programmable Gate Array) or a microcomputer. In this case, the program executed by the controller 13 in this embodiment may be realized using this integrated circuit.

(3) Berthing support processing
Next, we will explain the docking assistance process executed by the information processing device 1. In summary, based on point cloud data of the lidar 3 measured in the direction in which the docking location exists, the information processing device 1 identifies a surface present in the field of view of the lidar (i.e., the scanning range of the laser light) and a surface detected by the point cloud data, and determines the reliability of the docking location based on the identification result.

(3-1) Functional Blocks
3 is a functional block diagram of the docking location detection unit 15 and the docking parameter calculation unit 16 related to the docking assistance process. The docking location detection unit 15 functionally includes a normal vector calculation block 20, a field of view/detection plane specification block 21, a normal number specification block 22, a mean/variance calculation block 23, and a docking situation determination block 24. The docking parameter calculation unit 16 functionally includes a neighboring point search block 25, a nearest neighbor determination block 26, a docking distance calculation block 27, an approach angle calculation block 28, a docking speed calculation block 29, and a reliability information generation block 30.

The normal vector calculation block 20 calculates the normal vector of the plane formed by the docking location (also called the "docking surface") based on point cloud data generated by the LIDAR 3 in the direction in which the docking location is located. In this case, the normal vector calculation block 20 calculates the above-mentioned normal vector based on point cloud data generated by the LIDAR 3, for example, whose measurement range includes the docking side of the target ship. Information regarding the measurement range of the LIDAR 3 and the direction in which the docking location is located may be pre-registered, for example, in memory 12.

In this case, the normal vector calculation block 20 preferably downsamples the point cloud data and removes data obtained by the laser light reflecting off the water surface (also called "water surface reflection data").

In this case, the normal vector calculation block 20 first removes data that exists below the water surface position from the point cloud data generated by the LIDAR 3 as water surface reflection data (i.e., false detection data). The normal vector calculation block 20 estimates the water surface position, for example, based on the average height value of the point cloud data generated by the LIDAR 3 when there are no objects other than the water surface in the vicinity. The normal vector calculation block 20 then performs downsampling on the point cloud data after the water surface reflection data has been removed, which is a process of integrating measurement points for each grid space of a predetermined size. The normal vector calculation block 20 then calculates the normal vector for each measurement point indicated by the point cloud data after downsampling, using multiple surrounding measurement points. Downsampling may be performed before removing data reflected by the water surface.

The field of view/detection surface identification block 21 identifies the surface of the docking location that exists within the field of view of the lidar 3 (also referred to as the "inner field of view") and the surface of the docking location detected based on the normal vector calculated by the normal vector calculation block 20 (also referred to as the "detection surface"). In this case, the field of view/detection surface identification block 21 identifies whether the inner field of view and detection surface include the top and/or side surfaces of the docking location. Specific identification methods are described with reference to Figures 5 to 7.

The normal number identification block 22 extracts the vertical normal vectors and the normal vectors perpendicular to them (i.e., the horizontal direction) from the normal vectors calculated by the normal vector calculation block 20, and calculates the number of vertical normal vectors and the number of horizontal normal vectors. Here, the normal number identification block 22 considers the vertical normal vectors to represent normals to measurement points on the top surface of the docking location, and the horizontal normal vectors to represent normals to measurement points on the side of the docking location, and calculates the number of each as an indicator of the reliability of the docking location.

The mean/variance calculation block 23 extracts the vertical normal vectors and the normal vectors perpendicular to them (i.e., the horizontal direction) from the normal vectors calculated by the normal vector calculation block 20, and calculates the mean and variance of the vertical normal vectors and the mean and variance of the horizontal normal vectors.

The docking situation determination block 24 acquires the processing results of the field of view/detection plane determination block 21, the number of normals determination block 22, and the average/variance calculation block 23, which were identified or calculated based on the same point cloud data, as a determination result representing the detection status of the docking location at the time the point cloud data was generated.The docking situation determination block 24 then supplies the processing results of the field of view/detection plane determination block 21, the number of normals determination block 22, and the average/variance calculation block 23 to the reliability information generation block 30 as a determination result of the detection status of the docking location.

The neighboring point search block 25 searches for multiple points with short distances to detect a predetermined number of neighboring points of the target ship's docking location from the measurement points that make up the point cloud data. The nearest neighbor determination block 26 performs processing to determine the nearest point from the predetermined number of neighboring points detected by the neighboring point search block 25.

The shore distance calculation block 27 calculates the shore distance, which corresponds to the shortest distance between the target vessel and the berthing location, based on the nearest point determined by the nearest neighbor determination block 26. The approach angle calculation block 28 calculates the approach angle of the target vessel relative to the berthing location based on the shore distance calculated by the shore distance calculation block 27 or the average value of the normal vector calculated by the mean/variance calculation block 23. The berthing speed calculation block 29 calculates the berthing speed, which is the speed at which the target vessel approaches the berthing location, based on the shore distance calculated by the shore distance calculation block 27.

The reliability information generation block 30 generates reliability information based on the processing results of the docking situation determination block 24, nearby point search block 25, nearest neighbor determination block 26, opposite shore distance calculation block 27, and approach angle calculation block 28. Details of the reliability information will be described later.

(3-2) Operations related to normal vectors
Next, a specific example of the processing of the normal vector calculation block 20, the normal number specification block 22, and the mean/variance calculation block 23 will be described with reference to FIG.

Figure 4(A) shows an example of a hull coordinate system based on the hull of the target ship. As shown in Figure 4(A), the front (forward) direction of the target ship is the "x" coordinate, the lateral direction of the target ship is the "y" coordinate, and the height direction of the target ship is the "z" coordinate. The measurement data measured by the LIDAR 3 in the coordinate system based on the LIDAR 3 is converted into the hull coordinate system shown in Figure 4(A). Note that the process of converting point cloud data in a coordinate system based on a LIDAR installed on a moving body into the coordinate system of the moving body is disclosed, for example, in International Publication WO2019/188745.

Figure 4(B) is a perspective view of the structure 50, which is the docking location, clearly showing measurement points representing measurement positions measured by the lidar 3 and normal vectors calculated based on the measurement points. In Figure 4(B), measurement points are indicated by circles, and normal vectors are indicated by arrows. This shows an example in which both the top and side surfaces of the structure 50 were measured by the lidar 3.

As shown in Figure 4(B), the normal vector calculation block 20 calculates normal vectors for measurement points on the side and top surfaces of the structure 50. Because normal vectors are vectors perpendicular to the target plane or curved surface, they are calculated using multiple measurement points that can be configured as a surface. Therefore, a grid of predetermined length and width or a circle of predetermined radius is set, and calculations are performed using measurement points located within the grid. In this case, the normal vector calculation block 20 may calculate normal vectors for each measurement point, or may calculate normal vectors at predetermined intervals. The normal number determination block 22 then determines that normal vectors whose z-component is greater than a predetermined threshold are normal vectors pointing in the vertical direction. Note that normal vectors are assumed to be unit vectors. Furthermore, normal vectors whose z-component is less than a predetermined threshold are normal vectors pointing in the horizontal direction. The normal number determination block 22 then determines the number of vertical normal vectors (here, five) and the number of horizontal normal vectors (here, four). Furthermore, the mean/variance calculation block 23 calculates the mean and variance of the vertical normal vectors and the mean and variance of the horizontal normal vectors. Note that for edge portions, the measurement points around them are on the top or side, so the directions are oblique.

(3-3) Identifying the inner field of view and detection surface
The process of identifying the inner field of view and the detection surface will be explained below by dividing the process into docking situations A to E, which represent specific docking situations.

(3-3-1) Berthing situation A: Both the side and top surfaces of the berthing location are within the field of view and the detection surface.
5 is a diagram showing a docking situation in which both the top and side surfaces of a structure 50, which is the docking location, are included as the inner field of view and detection surface. In FIG. 5, a dashed line 51 indicates the estimated water surface position. Circles below the dashed line 51 indicate measurement points that have been removed as measurement points below the water surface position. Circles on the top and side surfaces of the structure 50 indicate measurement points corresponding to the normal vectors calculated by the normal vector calculation block 20.

In this case, the field of view/detection surface identification block 21 detects both the top and side surfaces of the structure 50 as detection surfaces based on the normal vectors calculated by the normal vector calculation block 20. For example, the field of view/detection surface identification block 21 determines that the top surface of the structure 50 has been detected if the number of vertical normal vectors is equal to or greater than a predetermined number, and determines that the side surface of the structure 50 has been detected if the number of horizontal normal vectors is equal to or greater than a predetermined number. The predetermined number is stored in advance, for example, in memory 12. The field of view/detection surface identification block 21 may also determine whether the top or side surface of the structure 50 has been detected by setting a threshold value for the average and/or variance of the normal vectors calculated by the average/variance calculation block 23. In this case, for example, the field of view/detection surface identification block 21 determines that the top surface is included as a detection surface if the difference between the average of the vertical normal vectors and the vector representing the vertical direction is within a predetermined threshold value and the variance of the vertical normal vectors is within a threshold value. Similarly, the field of view/detection surface identification block 21 determines that a side surface is included as a detection surface if the difference between the average of the horizontal normal vectors and the vector representing the vertical direction is within a predetermined threshold, and the variance of the horizontal normal vectors is within a threshold.

Furthermore, in the example of Figure 5, the field of view/detection surface identification block 21 determines that both the top and side surfaces of the structure 50 are within the field of view of the lidar 3, since both the top and side surfaces of the structure 50 are detected as detection surfaces.

In docking situation A shown in Figure 5, the inner field of view and detection surface include both the top and side surfaces of structure 50, which is the docking location, so the docking location is accurately detected, and it is estimated that the reliability of the detection of the docking location is high.

(3-3-2) Berthing situation B: Both sides of the berthing location are inside the field of view and only the sides are detected
FIG. 6A is a diagram showing a docking situation in which the inner field of view is the top and side surfaces, while the detection surface is only the side surface.

In the case of the docking situation shown in Figure 6(A), the field of view/detection plane identification block 21 detects the side of the structure 50 as the detection plane based on the normal vectors calculated by the normal vector calculation block 20. For example, the field of view/detection plane identification block 21 detects the side of the structure 50 as the detection plane because there are a predetermined number or more horizontal normal vectors and a predetermined number or less vertical normal vectors. On the other hand, the field of view/detection plane identification block 21 determines that the inner field of view includes both the top and side surfaces because there is a scanning position that can measure above the scanning position that detected the side of the structure 50 (in other words, the highest vertical number (described later in Figure 6(B)) of the data that detected the side of the structure 50 is not the number at the top of the vertical field of view). Figure 6(B) shows an example of the array of data generated by the lidar 3 in one scanning cycle. Each piece of data generated by the lidar 3 is identified by a combination of a vertical number (here, 1 to m) and a horizontal number (here, 1 to n) that correspond to the emission direction of the laser light. In the example of Figure 6(B), the data with vertical number 1 is the number at the top of the vertical field of view. In this case, in the example of Figure 6(A), if the highest vertical number in the data detecting the side of the structure 50 is not 1, the field of view/detection surface identification block 21 determines that the inner field of view includes both the top and side surfaces.

In the docking situation shown in Figure 6(A), although the inner field of view includes both the top and side surfaces, only the side surfaces of the docking location are detected, so it is estimated that the reliability of detecting the docking location is lower than in the case of Figure 5.

(3-3-3) Berthing situation C: Both sides of the berthing location are inside the field of view and only the top surface is the detection surface
FIG. 6C is a diagram showing a docking situation in which the inner surface of the field of view is the top surface and the side surface, while the detection surface is only the top surface.

In the case of the docking situation shown in Figure 6(C), the field of view/detection surface identification block 21 detects the side of the structure 50 as the top surface based on the normal vector calculated by the normal vector calculation block 20. On the other hand, the field of view/detection surface identification block 21 determines that the inner field of view includes both the top surface and the side surface because there is a scanning position that can measure below the scanning position that detected the top surface of the structure 50 (in other words, the lowest vertical number of the data that detected the top surface of the structure 50 is not the number at the bottom of the vertical field of view (number m in Figure 6(B))).

In the docking situation shown in Figure 6(C), although the inner field of view includes both the top and side surfaces, only the top surface of the docking location is detected, so the reliability of detecting the docking location is estimated to be lower than in the cases of Figures 5 and 6(A).

(3-3-4) Berthing situation D: The inner field of view and the detection surface are both side only
FIG. 7A is a diagram showing a docking situation in which the inner field of view and the detection surface are both side surfaces.

In the case of the docking situation shown in Figure 7(A), the field of view/detection plane identification block 21 detects part of the side of the structure 50 as the detection plane based on the normal vector calculated by the normal vector calculation block 20. Furthermore, the field of view/detection plane identification block 21 determines that the inner field of view includes only part of the side because there is no scanning position capable of measuring above the scanning position where the side of the structure 50 was detected (in other words, the highest vertical number in the data detecting the side of the structure 50 is the number at the top of the vertical field of view (number 1 in Figure 6(B))). Furthermore, in the docking situation shown in Figure 7(A), because only part of the side of the docking location was detected, it is estimated that the reliability of the detection of the docking location is lower than in cases such as Figure 5.

(3-3-5) Berthing situation E: The inner field of view and the detection surface are both on the upper surface only
FIG. 7B is a diagram showing a docking situation in which the inner field of view and the detection surface are both only the upper surface.

In the docking situation shown in Figure 7(B), the field of view/detection plane identification block 21 detects only the top surface of the structure 50 as the detection plane based on the normal vector calculated by the normal vector calculation block 20. Furthermore, the field of view/detection plane identification block 21 determines that the inner field of view includes only the top surface because there is no scanning position that can measure below the scanning position that detected the side of the structure 50 (in other words, the lowest vertical number of the data that detected the top surface of the structure 50 is the number at the bottom of the vertical field of view (number m in Figure 6(B))). Furthermore, in the docking situation shown in Figure 7(B), only a portion of the top surface of the structure 50 can be detected, and it is estimated that the reliability of detecting the docking location is lower than in cases such as Figure 5. Furthermore, in this case, the nearest point does not serve as the reference position of the structure 50 when calculating the distance to the opposite bank, so the distance to the opposite bank cannot be calculated accurately.

(3-4) Nearest point search
Next, a specific example of the processing of the neighbor point search block 25 and the nearest neighbor determination block 26 will be described with reference to Fig. 8(A) and Fig. 8(B). Fig. 8(A) is a perspective view of a structure 50, clearly indicating the measurement points of the structure 50 measured by the LIDAR 3 and the nearest points therein. Fig. 8(B) is a perspective view of a structure 50, clearly indicating the measurement points of the structure 50 measured by the LIDAR 3 and the nearest points therein.

The nearby point search block 25 searches for a predetermined number of nearby points based on the distance information in the point cloud data acquired from the LIDAR 3. In the example of Figure 8 (A), the nearby point search block 25 finds six nearby points. In this case, the nearby point search block 25 preferably removes data in the point cloud data that represents a position below the height at which the water surface position can be estimated, as water surface reflection data (i.e., false detection data) obtained by the laser light reflecting off the water surface.

The nearest neighbor determination block 26 determines the nearest neighbor among the neighboring points found by the neighboring point search block 25 as a nearest neighbor candidate and calculates the coordinate difference between the candidate and the other neighboring points (e.g., the average distance between the candidate and the other neighboring points). If the difference is less than a predetermined threshold, the nearest neighbor determination block 26 considers the candidate to be the nearest neighbor. Strong waves or winds near a quay can accidentally capture sea spray or floating objects. Therefore, the threshold is used to determine whether the measurement point is noise or an object other than a docked location, and is stored in advance, for example, in memory 12. On the other hand, if the difference is greater than or equal to the predetermined threshold, the nearest neighbor determination block 26 determines that the candidate is likely to be noise or a measurement point of another object, and does not consider the candidate to be the nearest neighbor. In this case, the nearest neighbor determination block 26 considers the next nearest neighbor after the candidate as a new candidate, calculates the difference, compares it with the threshold, and determines whether the candidate is the nearest neighbor. The variance of the coordinates of multiple neighboring points searched by the neighboring point search block 25 is used to set the reliability of the nearest neighbor search.

(3-5) Calculation of distance to shore, approach angle, and approaching speed
Next, calculation of the approach distance, approach angle, and approach speed will be described. As a representative example, a case where the LIDAR 3 is installed at two locations, one in front of the target ship and one in the rear of the target ship, will be described below.

Figure 9 (A) is a diagram showing an overview of the first calculation method for the distance to the opposite shore. In this example, the target vessel is not parallel to the structure 50 where it docks (there is an angle), so the distance to the nearest point is not the shortest distance.

In the first calculation method, the shore distance calculation block 27 generates an equation for a straight line passing through the nearest point determined based on the point cloud data of the front RIDER 3 and the nearest point determined based on the point cloud data of the rear RIDER 3. This equation is a line parallel to the side of the structure 50 on the target vessel side. The shore distance calculation block 27 then considers the distance to the line indicated by the equation to be the shortest distance from each RIDER 3, and considers these shortest distances to be the shore distance. Note that instead of considering the shortest distance for each RIDER 3 to be the shore distance, the shore distance calculation block 27 may determine the shortest distance among the shortest distances for each RIDER 3 as the shore distance, or may determine the average of these shortest distances as the shore distance.

According to the first calculation method, the shore distance calculation block 27 can appropriately calculate the shore distance even when the target vessel is at an angle to the berthing location.

Figure 9 (B) is a diagram showing an overview of the second calculation method for the distance to the opposite shore. In this example, too, the target vessel is not parallel to the structure 50 where it will be docked (there is an angle), so the distance to the nearest point is not the shortest distance.

In the second calculation method, the shore distance calculation block 27 calculates the dot product of the vector from the rider 3 to the nearest point and the normal vector at that nearest point for each of the front and rear riders 3. The shore distance calculation block 27 then considers the dot product calculation result for each rider 3 to be the shortest distance for each rider 3, and determines the shore distance from these shortest distances. Even with the second calculation method, the shore distance calculation block 27 can preferably calculate the shore distance regardless of the direction of the target vessel relative to the berthing location.

Next, we will explain how to calculate the docking speed. The docking speed calculation block 29 calculates the docking speed as the change over time in the distance to the shore (shortest distance) determined for each of the front rider 3 and the rear rider 3.

10 is a diagram showing an outline of a method for calculating the docking speed, where "d _f " represents the distance to the shore (shortest distance) of the front rider 3, "d _r " represents the distance to the shore (shortest distance) of the rear rider 3, "t" represents the current processing time, "t-1" represents the immediately preceding processing time, and "Δt" represents the time interval from the immediately preceding calculation time of the distance to the shore.

Here, the docking speed calculation block 29 inserts a filter to deal with an increase in noise when dividing the difference in the distance to the shore between the previous time and the current time by the time interval. Specifically, the docking speed calculation block 29 calculates the docking speed " _vf " for the front rider 3 and the docking speed "vr" for the rear rider 3 using the following equations, using a time constant "τ" and a Laplace operator " _s ".

This allows the docking speed calculation block 29 to appropriately calculate the docking speed, taking into account the effects of noise.

Next, we will explain how to calculate the approach angle. In the first calculation method, the approach angle calculation block 28 calculates the approach angle using the function "atan2," which is used to find the arc tangent from two arguments that define the tangent. Specifically, the approach angle calculation block 28 calculates the approach angle by using the function atan2 to calculate the average of the normal vectors of the side of the berthing location.

11(A) is a diagram showing an overview of the first calculation method for the approach angle. " _nf " represents the normal vector of the side of the structure 50, which is the docking location, facing the target ship, calculated based on the point cloud data of the front lidar 3. Also, " _nr " represents the normal vector of the side of the structure 50, which is the docking location, facing the target ship, calculated based on the point cloud data of the rear lidar 3. Also, the components of each normal vector are " _nf = [ _nxf , _nyf , _nzf ] ^T " and " _nr = [ _nxr , _nyr , _nzr ] ^T ".

In this case, the approach angle calculation block 28 calculates the approach angle "Ψ _f " calculated based on the point cloud data of the front rider 3 and the approach angle "Ψ _r " calculated based on the point cloud data of the rear rider 3 based on the following equations.

Furthermore, as a second calculation method for the approach angle, the approach angle calculation block 28 calculates the approach angle based on the angle between the direction of the target ship and the straight line represented by the straight-line equation connecting the nearest points. Figure 11 (B) is a diagram showing an overview of the second calculation method for the approach angle. The straight line connecting the nearest points is the same as the straight line shown in Figure 9 (A) and is parallel to the side of the structure 50, which is the berthing location, facing the target ship. Here, if the directional vector of the straight line is "L = [ _Lx , _Ly , _Lz ] ^T ", the approach angle corresponds to the angle between the straight line connecting the nearest points and the direction of travel, and therefore the approach angle calculation block 28 can suitably calculate the approach angle "Ψ" based on the following equation.

(3-6) Generation of reliability information
The reliability information generation block 30 generates a flag for each element such as the field of view when detecting the docking location, the surface detection of the docking location, the number and variance of normal vectors, etc., and generates a vector of the generated flag as reliability information. Hereinafter, a flag of "1" indicates that the reliability of the corresponding element is high, and a flag of "0" indicates that the reliability of the corresponding element is low.

Figure 12 shows an example of the data structure of reliability information generated by the reliability information generation block 30. As shown in Figure 12, the reliability information has the following items: "Top surface," "Side surface," "Nearby points," "Distance," and "Angle." The "Top surface" item has the sub-items "Viewing angle," "Detection," "Number of normals," and "Variance," while the "Side surface" item has the sub-items "Viewing angle," "Detection," "Number of normals," and "Variance." The "Nearby points" item has the sub-item "Variance," the "Distance" item has the sub-items "Amount of change" and "Rate of change," and the "Angle" item has the sub-item "Amount of change."

Here, the reliability information generation block 30 registers a flag in the "Viewing angle" sub-item of the "Top surface" item that is "1" if the top surface of the docking location is within the viewing angle range, and a flag that is "0" if the top surface is outside the viewing angle. The reliability information generation block 30 also registers a flag in the "Viewing angle" sub-item of the "Top surface" item that is "1" if the top surface of the docking location is the detection surface, and a flag that is "0" if the top surface is not the detection surface. The reliability information generation block 30 also registers a flag in the "Number of normals" sub-item of the "Top surface" item that is "1" if the number of normal vectors to the top surface of the docking location is equal to or greater than a predetermined threshold (e.g., 10), and a flag that is "0" if the number is less than the threshold. Furthermore, the reliability information generation block 30 registers a flag in the "Variance" sub-item of the "Top Surface" item that is set to "1" if the variances of the x, y, and z components of the normal vector relative to the top surface of the docking location are all below a predetermined threshold (e.g., 1.0), and a flag set to "0" if any of the variances is equal to or greater than the threshold. The reliability information generation block 30 also registers flags in each sub-item of the "Side Surface" item, determined according to the same rules as for each sub-item of the "Top Surface" item.

The reliability information generation block 30 also registers a flag in the "Variance" sub-item of the "Nearby Point" item that is set to "1" if the variances of the x, y, and z components of the nearby points searched by the nearby point search block 25 are all less than a threshold (e.g., 1.0), and a flag that is set to "0" if any of the variances is equal to or greater than the threshold. The reliability information generation block 30 also registers a flag in the "Change" sub-item of the "Distance" item that is set to "1" if the change in the opposite bank distance calculated by the opposite bank distance calculation block 27 from one hour before is less than a predetermined threshold (e.g., 1.0 m), and a flag that is set to "0" if the change is equal to or greater than the threshold. The reliability information generation block 30 also registers a flag in the "Change Rate" sub-item of the "Distance" item that is set to "1" if the change rate in the opposite bank distance calculated by the opposite bank distance calculation block 27 from one hour before is less than a predetermined threshold (e.g., ±10%), and a flag that is set to "0" if the change rate is equal to or greater than the threshold. Additionally, in the "Change" sub-item of the "Angle" item, the reliability information generation block 30 registers a flag that is set to "1" if the change in the approach angle calculated by the approach angle calculation block 28 from one time before is less than a predetermined threshold (e.g., 1.0 degrees), and to "0" if the change is equal to or greater than the threshold.

Note that each of the above thresholds is set to a suitable value pre-stored in memory 12, for example. Furthermore, reliability information may be generated for each rider 3.

Reliability information with this data structure makes it possible to grasp the reliability of the calculated distance to berth, berthing speed, and approach angle. Note that a value of "1" for each sub-item of the reliability information indicates the highest reliability. The information processing device 1 then adjusts the output of the drive source when berthing based on this reliability information. For example, the information processing device 1 may determine the upper limit of the target vessel's speed when berthing based on the total value of each sub-item indicated by the reliability information. In this case, the information processing device 1 determines that the smaller the total value, the less reliable the information regarding the berthing location is and the more careful berthing is required, and reduces the upper limit of the target vessel's speed when berthing.

(3-7) Processing flow
13 is an example of a flowchart showing an outline of the docking support process in this embodiment. The information processing device 1 repeatedly executes the process of the flowchart in FIG.

First, the information processing device 1 acquires point cloud data in the direction of the docking location (step S11). In this case, the information processing device 1 acquires point cloud data generated by, for example, a lidar 3 on the target ship whose measurement range includes the docking side. The information processing device 1 may also further downsample the acquired point cloud data and remove data reflected by the water surface.

Next, the docking location detection unit 15 of the information processing device 1 calculates normal vectors based on the point cloud data acquired in step S11 (step S12). Furthermore, in step S12, the docking location detection unit 15 calculates the number of normal vectors, the variance of the normal vectors, etc. Furthermore, the docking location detection unit 15 identifies the inner field of view and the detection plane based on the processing results of step S12 (step S13).

Next, the berthing parameter calculation unit 16 calculates berthing parameters based on the point cloud data acquired in step S11 and the normal vector information calculated in step S12 (step S14). In this case, the berthing parameter calculation unit 16 calculates the neighboring points and nearest points of the berthing location, and then uses the calculation results to further calculate the distance to the berth, approach angle, berthing speed, etc.

Then, the docking parameter calculation unit 16 generates reliability information based on the results of identifying the inner field of view and the detection surface in step S13 and the results of calculating the docking parameters in step S14 (step S15). The information processing device 1 then controls the ship based on the reliability information (step S16). This allows the information processing device 1 to accurately control the ship regarding docking based on reliability that accurately reflects the docking situation.

The information processing device 1 then determines whether the target ship has come alongside (docking) (step S17). In this case, the information processing device 1 determines whether the target ship has come alongside (docking) based on, for example, the output signal of the sensor group 2 or user input via the interface 11. If the information processing device 1 determines that the target ship has come alongside (step S17; Yes), it terminates the processing of the flowchart. On the other hand, if the target ship has not come alongside (step S17; No), the information processing device 1 returns the processing to step S11.

(4) Modified Example
The results of the identification by the field of view/detection surface identification block 21 are not limited to being used for generating reliability information. Alternatively, the information processing device 1 may use the results of the identification by the field of view/detection surface identification block 21 for various purposes other than generating reliability information. Hereinafter, examples of accurately grasping the docking situation and determining whether or not docking is possible based on the results of the identification by the field of view/detection surface identification block 21 will be described for each docking situation (docking situation A to docking situation E) with reference to Figures 14 to 16. The grasping of the docking situation and the determination of whether or not docking is possible based on the results of the identification by the field of view/detection surface identification block 21 are performed, for example, by the docking situation determination block 24.

[Bornelling situation A]
In the case of docking situation A shown in Figure 5, the information processing device 1 can grasp the characteristic of the docking location, that is, both the side and top surfaces of the docking location become the inner field of view and the detection surface, and thus can grasp the docking location's characteristic of having a top surface and a side surface. Therefore, the information processing device 1 can correctly recognize that it is a docking location (pier, quay). Furthermore, since the information processing device 1 knows the height of the docking location, it can also determine whether or not docking (docking) is possible. Furthermore, the information processing device 1 can accurately measure the distance to the docking location (i.e., calculate the distance to the opposite shore).

[Borne-docking situation B]
In the case of docking situation B shown in Fig. 6(A), the information processing device 1 determines that both sides of the docking location are inside the field of view and that only the side is the detection surface. In this case, either a docking location with accumulated water on the top surface or an object with almost no top surface, such as a stake or a fish pen enclosure, has been detected.

Figure 14(A) shows an example in which a structure 50 with a puddle of water 51 formed on its upper surface exists in docking situation B. Figure 14(B) shows an example in which a structure 55 with almost no upper surface exists in docking situation B. In the example of Figure 14(A), a puddle of water 51 has formed on structure 50, but the target vessel can dock. On the other hand, in the example of Figure 14(B), structure 55 has almost no upper surface, making it difficult for the target vessel to dock.

Taking the above into consideration, the information processing device 1 can determine that in docking situation B, the detected structure may not be the docking location.

[Borne-docking situation C]
In the case of docking situation C shown in Figure 6(B), the information processing device 1 determines that both sides of the docking location are within the field of view and only the top surface is the detection surface. In this case, it is possible that a large amount of moisture is attached to the side of the docking location, or that fenders have been installed. In the former case, the distance to the opposite shore can be measured from the point cloud data generated by the lidar 3, but in the latter case, the distance to the fenders cannot be measured.

Figure 15(A) shows an example of a structure 50 with adhering water 56 on its side in docking situation C. Figure 15(B) shows an example of a structure 50 with fenders 57 on its side in docking situation C. In the example of Figure 15(B), the fenders 57 are usually black, so the reflected light does not reach the lidar 3, making it difficult for the lidar 3 to detect the fenders 57 and preventing the distance to the shore (here, the distance to the fenders) from being measured.

Taking the above into consideration, in the case of docking situation C, the information processing device 1 determines that there is a possibility that the docking location (pier, quay) is such that the distance to the shore cannot be calculated accurately. Even in docking situation C, the information processing device 1 can calculate the docking speed, which is the speed relative to the docking location, by measuring the change in the distance to the shore.

Furthermore, in the case of docking situation C shown in Figure 6(B), the information processing device 1 can also determine that a pier with thin sides may be the docking location. In this case, the information processing device 1 can continue measurements using the LIDAR 3 while approaching the docking location, and if the side is detected, can determine that it is a pier. Figures 16(A) and 16(B) show an example in which a pier 58 is present in docking situation C. In the example of Figure 16(A), the information processing device 1 is unable to detect the side of the pier 58 using the LIDAR 3. However, in the example of Figure 16(B), which shows a situation where the pier 58 is approached even closer, the information processing device 1 is able to detect the side of the pier 58 using the LIDAR 3. In this case, the information processing device 1 can determine that the pier 58 is a pier.

[Bornelling situation D]
In the case of docking situation D shown in Figure 7 (A), the information processing device 1 determines that both the inner field of view and the detection surface are only side surfaces. In this case, the information processing device 1 is unable to determine the height of the side surface of the detected docking location. Note that in this case, the detected docking location may be a quay high enough to allow docking, but may also be a quay too high to allow docking. Therefore, in the case of docking situation D, the information processing device 1 can determine that the detected structure may not be a place where docking is possible, even if the distance to the shore can be measured.

[Borne-docking situation E]
In the case of docking situation E shown in Figure 7 (B), the information processing device 1 determines that both the inner field of view and the detected surface are only the upper surface. In this case, the information processing device 1 cannot determine how far the detected upper surface extends toward the front. In this case, it is not possible to determine the distance from the detection data, and the target vessel may be very close to the docking location. In this case, the information processing device 1 controls the operation of the target vessel to move the target vessel away from the detected structure, or outputs driving guidance information for the target vessel to a display unit or the like, urging the target vessel to move away from the detected structure. This allows the information processing device 1 to include both the side and upper surfaces of the docking location in the field of view of the lidar 3, thereby enabling it to appropriately detect and determine the docking location.

As described above, the controller 13 of the information processing device 1 according to this embodiment acquires point cloud data in the direction of the docking location, generated by the LIDAR 3 installed on the target vessel. Then, based on the normal vector calculated from the point cloud data, the controller 13 identifies the inner field of view, which is the surface of the docking location present in the field of view of the LIDAR 3, and the detection surface, which is the surface of the docking location detected by the point cloud data. Then, based on the identification results, the controller 13 determines the docking status regarding the docking location. This allows the information processing device 1 to accurately grasp the docking status.

In the above-described embodiments, the program can be stored using various types of non-transitory computer-readable media and supplied to a computer controller or the like. Non-transitory computer-readable media includes various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R/Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)).

The present invention has been described above with reference to examples, but the present invention is not limited to the above examples. Various modifications to the configuration and details of the present invention that are understandable to those skilled in the art can be made within the scope of the present invention. In other words, the present invention naturally includes various modifications and alterations that a person skilled in the art would be able to make in accordance with the entire disclosure, including the claims, and the technical ideas. Furthermore, the disclosures of the above-cited patent documents and other documents are incorporated herein by reference.

1 Information processing device 2 Sensor group 3 Lidar

Claims (10)

an acquisition means for acquiring measurement data in a direction toward a berthing location, the measurement data being generated by a measurement device provided on the ship;
an identification means for identifying a detection plane, which is the plane of the docking location detected by the measurement data, and a vertical number of the vertical field of view angle of the measurement device corresponding to the upper end or lower end of the detection plane, based on a normal vector calculated from the measurement data;
a docking status determination means for determining a docking status regarding the docking location based on the result of identification by the identification means;
An information processing device having the above. The information processing device of claim 1, further comprising a normal vector calculation means for calculating the normal vector based on data obtained by removing water surface reflection data from the measurement data using the water surface position as a reference. The information processing device of claim 1 or 2, further comprising a normal vector calculation means for calculating the normal vector for each measurement point of the downsampled measurement data. The information processing device according to any one of claims 1 to 3, wherein the identification means determines whether the top surface of the docking location is the detection surface based on a vertical normal vector, and determines whether the side surface of the docking location is the detection surface based on a horizontal normal vector. 5. The information processing device according to claim 1 , further comprising reliability information generating means for generating reliability information representing a reliability of the docking location based on the determination result of the docking situation. a shore distance calculation means for calculating a shore distance, which is a distance from the ship to the berthing location, based on the normal vector;
an approach angle calculation means for calculating an approach angle of the vessel with respect to the berthing location based on the distance to the shore;
and
6. The information processing device according to claim 5, wherein the reliability information generating means generates the reliability information including a reliability of the opposite bank distance based on a time change of the opposite bank distance and a reliability of the approach angle based on a time change of the approach angle . a neighboring point search means for determining a predetermined number of neighboring points at the docking location relative to the vessel based on the measurement data;
a nearest point determining means for determining a nearest point at the docking location relative to the ship from the nearest point;
and
the opposite bank distance calculation means calculates the opposite bank distance based on the nearest point and the normal vector;
The information processing apparatus according to claim 6 , wherein the reliability information generating means generates the reliability information including the reliability of the nearest neighbor point based on a variance of the neighboring points. A computer-implemented control method comprising:
Acquire measurement data in the direction of the docking location generated by a measuring device installed on the ship,
based on the normal vector calculated from the measurement data, a detection plane, which is the plane of the docking location detected by the measurement data, and a vertical number of the vertical field of view angle of the measurement device corresponding to the upper end or lower end of the detection plane are identified;
determining a docking situation regarding docking at a docking location based on the result of identifying the detection surface and the vertical number ;
Control method. Acquire measurement data in the direction of the docking location generated by a measuring device installed on the ship,
based on the normal vector calculated from the measurement data, a detection plane, which is the plane of the docking location detected by the measurement data, and a vertical number of the vertical field of view angle of the measurement device corresponding to the upper end or lower end of the detection plane are identified;
A program that causes a computer to execute a process of determining the docking status of the docking location based on the result of identifying the detection surface and the vertical number . A computer - readable storage medium storing the program according to claim 9 .