JP7572082B2 - LOADING SPACE RECOGNITION DEVICE, SYSTEM, METHOD, AND PROGRAM - Google Patents

Description

[Description of Related Applications]
The present invention is based on the priority claim of Japanese Patent Application: Patent Application No. 2021-030741 (filed on February 26, 2021), the entire contents of which are incorporated herein by reference.
The present invention relates to a loading space recognition device, a system, a method, and a program.

In the logistics industry, cargo loaded in truck containers can fall during transport, causing damage to the cargo and reducing the quality of transport. As a result, technologies have been proposed to generate information for arranging cargo to prevent cargo from collapsing, and to quickly discover if cargo has actually collapsed (see, for example, Patent Documents 1 to 6).

Patent Documents 1 and 2 disclose a shape information generating device as a technology for preventing cargo from falling over, the shape information generating device comprising: a first information acquisition unit that acquires three-dimensional information of a first region of the surface of a plurality of stacked items, obtained by imaging or scanning the plurality of items from a first location; a second information acquisition unit that acquires three-dimensional information of a second region of the surface of the plurality of items, obtained by imaging or scanning the plurality of items from a second location; and a synthesis unit that generates information indicating a three-dimensional shape of at least a portion of the surface of the plurality of items based on the three-dimensional information of the first region acquired by the first information acquisition unit and the three-dimensional information of the second region acquired by the second information acquisition unit, the first location and the second location being located at different positions, and the synthesis unit complements one of the three-dimensional information of the first region and the three-dimensional information of the second region with the other to generate information indicating the three-dimensional shape of at least a portion of the surface of the plurality of items.

Patent Document 3 discloses a loading method for loading multiple objects in a specified packing shape and with a specified weight as a technology for preventing cargo from falling over, the method comprising the steps of measuring the weight and packing shape of each of the loaded objects, calculating the density of each of the loaded objects from the weight and packing shape of each of the loaded objects, and accumulating the density information of each of the loaded objects to calculate the loading position of each of the loaded objects.

Patent Document 4 discloses a vehicle trunk monitoring device that, as technology that allows the driver to check the status of luggage in the trunk at any time (whether luggage has fallen over) on a monitor, comprises surveillance cameras attached to the top surface of the trunk at a midpoint in the vehicle width direction and at two locations that divide the entire length of the trunk into thirds, an information processing device that inputs image data of the interior of the trunk from the surveillance cameras, a monitor that inputs and displays image data of the interior of the trunk from the information processing device, and a communication device that inputs image data of the interior of the trunk from the information processing device and transmits it to a base station, and is configured so that the availability of loading space can be ascertained from images from the surveillance cameras.

Patent Document 5 discloses a cargo shift monitoring system for transport vehicles as technology that enables drivers and a management center to easily become aware of cargo shifts in transport vehicles and quickly respond to the shifts. The system includes an on-board terminal that monitors cargo shifts using a sensor attached inside the bed of the transport vehicle and, when cargo shifts are detected, issues a warning to the driver indicating that cargo shifts have occurred, and a management center that receives a cargo shift monitoring signal indicating that cargo shifts have occurred, sent from the on-board terminal when the on-board terminal issues the warning to the driver, and collects and manages information about cargo shifts in the transport vehicle.

Patent Document 6 discloses a method for detecting cargo collapse, as a technology capable of detecting cargo collapse that occurs during transfer and after the transfer operation is completed, in which an image of a group of goods loaded on a pallet is taken from above before and after each transfer of an individual item, a first image is obtained before the goods are transferred and a second image is obtained after the transfer, the first image is compared with the second image, and the presence or absence of cargo collapse is determined based on the degree of change in the goods area other than the area where the transferred goods were located.

Patent No. 6511681 Patent No. 6577687 JP 2002-29631 A JP 2018-199489 A JP 2005-018472 A JP 2007-179301 A

The following analysis is provided by the present inventors.

However, the technologies for suppressing and preventing cargo shifting described in Patent Documents 1 to 3 do not completely prevent cargo shifting simply by loading the cargo in a way that prevents it from shifting, and since cargo can move in all directions due to vehicle acceleration, braking, vibrations, etc., which can lead to cargo shifting, it is not possible to determine the possibility of cargo shifting due to cargo movement.

In addition, in the technology for checking luggage described in Patent Document 4, the driver checks the status of the luggage in the luggage compartment on a monitor and becomes aware of any luggage shifting, making it difficult from a safety standpoint for the driver to determine the possibility of luggage shifting while driving the vehicle.

In addition, the technology for detecting cargo collapse described in Patent Document 5 determines whether cargo collapse has occurred based on whether or not a sensor receives infrared, ultrasonic, or other signals, so it is not possible to detect cargo movement in an area where there is no change in the signal received by the sensor, and it may not be possible to determine the possibility of cargo collapse occurring due to cargo movement.

Furthermore, in the technology of Patent Document 6 capable of detecting cargo collapse that occurs during transfer and after the transfer operation is completed, edge images are generated from a first image taken before the transfer and a second image taken after the transfer, and the two images are compared to determine whether cargo collapse has occurred based on the degree of change in items other than the transferred item. However, comparison between edge images cannot detect movement of cargo that does not move at the edge position (movement of cargo towards or away from the image capture position), and there is a possibility that it will not be possible to determine the possibility of cargo collapse due to cargo movement.

The main objective of the present invention is to provide a loading space recognition device, system, method, and program that can contribute to determining the possibility of cargo collapse due to cargo movement.

The loading space recognition device according to the first viewpoint includes a luggage overall image estimation unit configured to estimate an overall image of luggage loaded in the loading space based on three-dimensional data obtained by capturing an image of the luggage loading space from a predetermined direction and output the estimation result data as voxelization data, a voxelization unit configured to voxelize the estimation result data and output the voxel data as voxel data, and a determination unit configured to estimate a volumetric fluctuation or movement amount of the luggage by comparing the voxel data at an arbitrary reference time with the voxel data after a predetermined or arbitrary time has elapsed from the reference time, and to determine whether or not there is a possibility of luggage collapse by comparing the estimated volumetric fluctuation or movement amount with a threshold value.
an area designation unit configured to designate a luggage loading area in the loading space by a user's operation,
The luggage overall image estimation unit is configured to estimate an overall image of the luggage loaded in the loading area .

The loading space recognition system relating to the second viewpoint includes a sensor that senses the surface of the luggage in the loading space and outputs three-dimensional image data, and a loading space recognition device relating to the first viewpoint.

A loading space recognition method according to a third perspective is a loading space recognition method that recognizes a loading space for luggage using hardware resources, and includes the steps of: specifying a loading area for luggage in the loading space by a user operation; estimating an overall image of luggage loaded in the loading area for luggage in the loading space based on three-dimensional data obtained by imaging the loading space from a predetermined direction and outputting the estimated result data as estimation result data; voxelizing the estimation result data and outputting it as voxel data; and estimating a volume change or movement amount of the luggage by comparing the voxel data at an arbitrary reference time with the voxel data after a predetermined or arbitrary time has elapsed from the reference time, and determining whether or not there is a possibility of luggage collapse by comparing the estimated volume change or movement amount with a threshold value.

The program relating to the fourth perspective is a program that causes a hardware resource to execute a process for recognizing a luggage loading space, and causes the hardware resource to execute the following processes: a process for specifying a luggage loading area in the loading space by user operation; a process for estimating an overall image of the luggage loaded in the luggage loading area in the loading space based on three-dimensional data captured of the loading space from a predetermined direction and outputting the estimated result data as estimated result data; a process for voxelizing the estimated result data and outputting it as voxel data; and a process for estimating the amount of change in volume or movement of the luggage by comparing the voxel data at an arbitrary reference time with the voxel data after a predetermined or arbitrary time has elapsed from the reference time, and for determining whether or not there is a possibility of the luggage collapsing by comparing the estimated amount of change in volume or movement with a threshold value.

The program can be recorded on a computer-readable storage medium. The storage medium can be a non-transient medium such as a semiconductor memory, a hard disk, a magnetic recording medium, or an optical recording medium. In the present disclosure, the program can also be embodied as a computer program product. The program is input to the computer device from an input device or an external device via a communication interface, stored in a storage device, and drives the processor according to a predetermined step or process, and can display the processing results, including intermediate states as necessary, at each stage via a display device, or can communicate with the outside via the communication interface. For example, a computer device for this purpose typically includes a processor, a storage device, an input device, a communication interface, and a display device as necessary, which are connectable to each other via a bus.

The first to fourth aspects can contribute to determining the possibility of cargo shifting due to luggage movement.

1 is a schematic diagram illustrating an example of a configuration and a usage mode of a loading space recognition system according to a first embodiment. FIG. 1 is a block diagram illustrating a schematic configuration of a loading space recognition device in a loading space recognition system according to a first embodiment. FIG. 4 is a flowchart illustrating an operation of the loading space recognition device in the loading space recognition system according to the first embodiment. 11A to 11C are schematic diagrams showing examples of cases in which there is a volume change when a difference between a reference voxel and a comparison voxel is extracted. 11A and 11B are schematic diagrams showing examples of cases in which there is no volume change and there is a movement amount when a difference between a reference voxel and a comparison voxel is extracted. FIG. 6 is an image diagram showing a transition from the group estimation to the maximum movement amount estimation in Example 2-1 of FIG. 5; FIG. 6 is an image diagram showing a transition from the group estimation to the maximum movement amount estimation in Example 2-2 of FIG. 5. FIG. 6 is an image diagram showing a transition from the group estimation to the maximum movement amount estimation in Example 2-3 of FIG. 5. FIG. 6 is an image diagram showing a transition from the group estimation to the maximum movement amount estimation in Example 2-4 of FIG. 5. 1A to 1C are conceptual diagrams each showing a modified example of the configuration and usage of the loading space recognition system according to the first embodiment. FIG. 11 is a block diagram illustrating a configuration of a loading space recognition device according to a second embodiment. FIG. 2 is a block diagram illustrating a schematic configuration of hardware resources.

The following describes the embodiments with reference to the drawings. In this application, when reference symbols are used, they are intended to aid in understanding and are not intended to limit the invention to the illustrated form. The following embodiments are merely illustrative and do not limit the invention. In addition, the connection lines between blocks in the drawings and the like referred to in the following description include both bidirectional and unidirectional. One-way arrows are used to diagrammatically indicate the flow of the main signal (data) and do not exclude bidirectionality. Furthermore, although not explicitly shown in the circuit diagrams, block diagrams, internal configuration diagrams, connection diagrams, etc. shown in the present disclosure, input ports and output ports exist at the input and output ends of each connection line. The same is true for input/output interfaces. The program is executed via a computer device, which includes, for example, a processor, a storage device, an input device, a communication interface, and a display device as necessary, and the computer device is configured to be able to communicate with devices (including computers) inside or outside the device via the communication interface, regardless of whether it is wired or wireless.

[Embodiment 1]
The loading space recognition system according to the first embodiment will be described with reference to the drawings. Fig. 1 is an image diagram showing an example of the configuration and usage of the loading space recognition system according to the first embodiment. Fig. 2 is a block diagram showing an example of the configuration of the loading space recognition device in the loading space recognition system according to the first embodiment. Here, the loading space recognition system will be described with reference to an example of a container load on a truck.

The loading space recognition system 1 is a system that uses a sensor 10 to recognize the change (volume change, distance change) of an object (baggage 4 in FIG. 1) in a loading space (loading space 5 in a container 3 of a truck 2 in FIG. 1) in which the object is loaded (see FIG. 1). In the loading space recognition system 1, the sensor 10 and the loading space recognition device 200 are connected to each other so that they can communicate (wired communication or wireless communication). The loading space recognition system 1 can be mounted on a truck 2. In the loading space recognition system 1, the loading space recognition device 200 recognizes the change of an object in the loading space 5 based on the photographed data (100 in FIG. 2) photographed by the sensor 10. Here, the photographed data 100 is three-dimensional data photographed by the sensor 10 (data photographed of three-dimensional elements such as point cloud data and depth data, data reconstructed from multiple images into a three-dimensional space, etc.).

The sensor 10 senses and photographs (takes an image) the surface of the luggage 4 in the loading space 5, which is the photographing area (see FIG. 1). The sensor 10 is communicably connected to the loading space recognition device 200. The sensor 10 outputs photographing data (100 in FIG. 2) created from three-dimensional data photographed of the loading space 5 to the loading space recognition device 200. The sensor 10 can be selected according to the photographing conditions such as the photographing distance, angle of view, and size of the container 3 required for detecting the luggage 4, and customer requests. In addition, products sold by various manufacturers can be used for the sensor 10. For example, a stereo camera, a ToF (Time of Flight) camera, a 3D-lidar (Three Dimensions - light detection and ranging), a 2D-lidar (Two Dimensions - light detection and ranging), a LIDAR (Laser Imaging Detection And Ranging), etc. can be used as the sensor 10. The sensor 10 can be attached at a position where it can photograph the loading space 5, which is the photographing area, and can be attached, for example, near the upper part of the baggage entrance side of the container 3. The sensor 10 can photograph the baggage 4 in the loading space 5 from above the baggage entrance side of the container 3. At least one sensor 10 is required in the loading space 5, but there may be multiple sensors. In addition, multiple sensors 10 may be used when the loading space 5 is large and it is difficult to photograph with one sensor. When there are multiple sensors 10 in the loading space 5, the sensors may be of different types or manufacturers. When there are multiple sensors 10 in the loading space 5, the photographed data photographed by each sensor 10 may be synthesized by the loading space recognition device 200.

The loading space recognition device 200 is a device that recognizes the change (volume change, distance change) of the luggage 4 in the loading space 5 in which the luggage 4 is loaded based on the photographed data 100 from the sensor 10 (see Figs. 1 and 2). As the loading space recognition device 200, a device (computer device) having functional parts (e.g., a processor, a storage device, an input device, a communication interface, and a display device) constituting a computer can be used, and for example, a notebook personal computer, a smartphone, a tablet terminal, etc. can be used. The loading space recognition device 200 has a function of determining whether or not there is a possibility of collapse of the luggage 4. The loading space recognition device 200 can also be adopted in a situation such as pallet loading where a certain amount of volume change is expected when a load collapse occurs. The loading space recognition device 200 is realized by a preprocessing unit 210, a package shape recognition unit 220, a determination unit 230, and a user interface unit 240 by executing a predetermined program.

The pre-processing unit 210 is a functional unit that performs pre-processing on the image capture data 100 in order to perform loading space recognition processing (see FIG. 2). The pre-processing unit 210 outputs pre-processed data 101, which is the pre-processed image capture data 100, to the packaging style recognition unit 220 and the user interface unit 240. The pre-processing unit 210 includes a format conversion unit 211 and a noise removal unit 212.

The format conversion unit 211 is a functional unit that converts the format of the shooting data 100 into a common format that can be commonly used in the loading space recognition device 200 as a pre-processing step (see FIG. 2). The format conversion unit 211 outputs the shooting data 100 in the common format to the noise removal unit 212. Note that if the format of the shooting data 100 is originally a common format, the processing of the format conversion unit 211 can be omitted (skip).

The noise removal unit 212 is a functional unit that removes noise (e.g., point clouds unnecessary for loading space recognition) from the photographed data 100 from the format conversion unit 211 as preprocessing (see FIG. 2). The noise removal unit 212 outputs the preprocessed data 101 from which noise in the photographed data 100 has been removed to the luggage overall image estimation unit 222 of the package shape recognition unit 220, and outputs it to the display unit 241 of the user interface unit 240 as necessary. Examples of noise removal methods include smoothing processing, filtering (e.g., moving average filter processing, median filter processing, etc.), and outlier removal processing (e.g., outlier removal processing by chi-square test). Note that if there is almost no noise, the processing of the noise removal unit 212 may be omitted (skip).

The pre-processing data 101 is output to the display unit 241, and the user can view it. In addition, since the shooting data 100 and the pre-processing data 101 are the basis for all processing, it is also possible to use them as confirmation by making them saveable.

The packaging style grasping unit 220 is a functional block that grasps the current packaging style from the pre-processing data 101 (see FIG. 2). The packaging style grasping unit 220 outputs voxel data 102 relating to the grasped current packaging style to the judgment unit 230. The packaging style grasping unit 220 includes an area designation unit 221, a luggage overall image estimation unit 222, and a voxelization unit 223. The packaging style grasping unit 220 measures information about the shape of the loaded luggage or the available space from the pre-processing data 101, and provides quantitative information.

The area designation unit 221 is a functional unit that designates an area (loading area) within the container 3 where luggage 4 can be loaded (see Figure 2). For example, if there is an area within the container 3 where luggage 4 cannot be loaded, the area designation unit 221 excludes that area when designating the area. The area designation unit 221 can allow the user to designate the loading area via the operation unit 242, and can also designate an area automatically in combination with a system that automatically acquires area edges.

The area designation unit 221 may also allow the user to specify, via the operation unit 242, a judgment exclusion area to be excluded from the judgment by the judgment unit 230. This makes it possible to avoid excessive judgment of changes in the packing shape of luggage 4 that cannot maintain a constant shape by not judging the change in the packing shape. The change judgment exclusion area may be specified not only in area units, but also in voxel units or single/individual luggage units (object units).

The luggage overall image estimation unit 222 is a functional unit that estimates an overall image of the luggage 4 loaded in the loading area based on the pre-processing data 101 from the noise removal unit 212 of the pre-processing data 101 and the loading area specified by the area designation unit 221 (see FIG. 2). The luggage overall image estimation unit 222 creates estimation result data related to the overall image of the loaded luggage 4. The luggage overall image estimation unit 222 outputs the created estimation result data to the voxelization unit 223. The luggage overall image estimation unit 222 estimates not only the luggage loaded on the surface or the foreground, but also the luggage loaded in the back part (occlusion part) that is blocked by them and cannot be seen by the sensor 10 (gaps, if necessary). To estimate the luggage loaded in the occluded area, past data accumulated in chronological order may be used for supplementation, or prior knowledge of the size and number of loaded luggage may be used. In addition to a simple algorithm that assumes that a coordinate point indicating the luggage exists on an extension of a straight line connecting the frontmost coordinate point and the sensor, pre-processed data 101 stored in chronological order or machine learning may be used.

An example of a method for collecting teacher data for machine learning is a method of associating the pre-processed data 101 with the actual loading status and recording it in a database. For example, distance sensors are provided at predetermined intervals on the ceiling surface of the container, and the distance between the loaded luggage is measured by the distance sensors. This makes it possible to grasp the luggage loading status, such as whether the luggage directly below the sensor is stacked up to the ceiling, whether it is stacked up to about half the height, or whether there is no luggage at all. By providing multiple sensors on the ceiling surface, it is possible to grasp the luggage loading status throughout the container. In this way, machine learning can be performed by associating the pre-processed data 101 with the actual loading status. The machine learning method is not limited to this, and other methods may be used. In addition, when machine learning is used, a function for estimating information and attributes other than shape, such as "weight" and "no top loading/no bottom loading" may be provided.

The voxelization unit 223 is a functional unit that converts the estimation result data of the luggage overall image estimation unit 222 into voxels (see FIG. 2). The voxelization unit 223 creates voxel data 102 relating to the overall image of the luggage 4 based on the estimation result data of the luggage overall image estimation unit 222. The voxelization unit 223 creates voxel data 102 by treating the portion where the loaded luggage 4 does not exist as empty space. The voxelization unit 223 outputs the created voxel data 102 to the reference determination unit 231 and the difference extraction unit 232 of the determination unit 230. In the voxelization process, it is possible to estimate an occluded area that is not visible from the 3D sensor, or to complement the data accumulated in chronological order, or to use prior knowledge of the size and number of loaded luggage.

Here, the voxel data 102 is data that represents the overall image of the luggage 4 by combining multiple voxels (cubes) of a predetermined size. The voxel data 102 includes information related to the dimensions of each voxel and the position of each face. The voxel data 102 may be stored each time it is created. The voxel data 102 may also include information such as the number of luggage within one voxel, information on which multiple voxels one luggage spans, weight, shape, etc.

The determination unit 230 is a functional unit that estimates the amount of change in the volume or movement of the luggage by comparing the voxel data 102 (reference voxel data) at a certain reference time point with the voxel data 102 (contrast voxel data) at a time point after a predetermined or arbitrary time has elapsed, and determines whether or not there is a possibility of the luggage collapse by comparing the estimated amount of change in the volume or movement with a threshold value (see FIG. 2). This makes it possible to prevent the luggage 4 from falling. In addition, the determination unit 230 may detect the possibility of the luggage collapse not only by comparing the reference voxel data with the contrast voxel data, but also by comparing the contrast voxel data with other contrast voxel data created one time ago. This makes it possible to prevent the possibility of the luggage collapse from being determined when the amount of change between the contrast voxel data per unit time is smaller than the amount of change between the reference voxel data and the contrast voxel data. Furthermore, the truck 2 (structure having a loading space) may be provided with a detection unit 20 for detecting shaking and sound (see FIG. 10), and the determination unit 230 may obtain comparison voxel data from the voxelization unit 223 when the detection unit 20 detects shaking or sound of a certain level or more, and detect the possibility of cargo collapse. Furthermore, to complement and verify the accuracy of the movement estimation, information obtained from another system, such as a large number of heavy cargoes or a large number of light cargoes, may be used. The determination unit 230 includes a reference determination unit 231, a difference extraction unit 232, a fluctuation volume estimation unit 233, a lump extraction unit 234, a combination estimation unit 235, a movement amount estimation unit 236, and a cargo collapse determination unit 237.

The reference determination unit 231 is a functional unit that determines the voxel data 102 at a certain reference point (reference time) from the voxelization unit 223 as a change reference point (see Figure 2). The reference determination unit 231 stores the voxel data 102 at the reference point (instructed time) from the voxelization unit 223 in response to an instruction from the operation unit 242, and holds it as reference data for positional fluctuations. The reference determination unit 231 outputs the voxel data 102 at the reference point to the difference extraction unit 232.

The difference extraction unit 232 is a functional unit that extracts voxel differences (e.g., differences in depth positions) at corresponding positions on a surface when the luggage 4 is viewed from a predetermined position (e.g., the rear of the truck 2, the position of the sensor 10) by comparing the voxel data 102 (reference voxel data) at an arbitrary reference time with the voxel data 102 (contrast voxel data) at a time when a predetermined or arbitrary time has elapsed from the reference time (see FIG. 2). In the difference extraction process, the difference extraction unit 232 obtains the reference voxel data from the reference determination unit 231 and obtains the contrast voxel data from the voxelization unit 223. In the difference extraction process, for example, when the position of the surface of the voxel when the luggage 4 is viewed from the rear of the truck 2 changes to the front side of the truck 2 in the depth direction (for example, when the luggage 4 at the position of the target voxel moves to the left, right, or downward and a luggage 4 one level further back appears, when the luggage 4 at the position of the target voxel moves to the rear, and when the occlusion portion decreases), a difference is extracted to represent a "decrease" in volume, and when the position of the surface of the voxel changes to the rear side of the truck 2 (for example, when the luggage 4 at the position of the target voxel moves to the front, when another luggage 4 moves in front of the luggage 4 at the position of the target voxel, and when the occlusion portion increases), a difference is extracted to represent an "increase" in volume. In addition, in the difference extraction process, for example, a difference can be extracted to represent the degree of decrease or increase in volume in stages according to the magnitude of the distance by which the position of the surface of the voxel has changed forward or backward of the truck 2. The difference extraction unit 232 outputs the difference data extracted by the difference extraction process to the fluctuation volume amount estimation unit 233 and the lump extraction unit 234 .

The volume change estimation unit 233 is a functional unit that estimates the volume change (volume change) of the overall image of the luggage 4 based on the difference data from the difference extraction unit 232 (see FIG. 2). The volume change can be expressed, for example, by the sum of the differences in the depth direction positions of the voxels at corresponding positions on the surface of the luggage 4 viewed from the rear of the truck 2. If a certain volume increases at a certain position and the same volume decreases at another position, the volume change is none (0) (see movement examples 2-1 to 2-4 in FIG. 5). If a certain volume increases at a certain position and a volume decrease is greater than the increased volume at another position, the volume change is a negative value. If a certain volume increases at a certain position and a volume decrease is less than the increased volume at another position, the volume change is a positive value (see movement examples 1-1 to 1-3 in FIG. 4). If a certain volume increases at a certain position and there is no decrease in the volume at another position, the volume change is a positive value. Furthermore, when a certain volume amount decreases at a certain position and there is no increase in the volume amount at another position, the fluctuation volume amount becomes a negative value. The fluctuation volume amount estimating unit 233 outputs the estimated fluctuation volume amount estimation data to the cargo collapse determining unit 237.

The clump extraction unit 234 is a functional unit that extracts clumps of voxels with no increase or decrease in volume that exist between voxels with an increased volume (volume-increased voxels) and voxels with a decreased volume (volume-decreased voxels) at the same horizontal position based on the difference data from the difference extraction unit 232 (see FIG. 2). It can be assumed that the clump extraction unit 234 does not involve a change in the volume of the luggage 4. If the clump extraction unit 234 is able to extract a clump, it outputs difference data including the extracted clump data to the combination estimation unit 235. If the clump extraction unit 234 is unable to extract a clump, it outputs difference data from the difference extraction unit 232 to the combination estimation unit 235.

The combination estimation unit 235 is a functional unit that estimates the combination of volume-increasing voxels and volume-decreasing voxels based on the difference data from the clump extraction unit 234 (including clump data if clumps can be extracted) (see FIG. 2). The combination estimation unit 235 can assume that the volume of the luggage 4 does not change. In the combination estimation process, for example, when multiple combinations are possible, a volume-increasing voxel and a volume-decreasing voxel that are in the same horizontal position can be preferentially combined. Also, when multiple combinations are possible, a volume-increasing voxel and a volume-decreasing voxel that are the furthest apart from each other can be preferentially combined. Also, when multiple combinations are possible, a volume-increasing voxel and a volume-decreasing voxel that are the furthest apart from each other can be preferentially combined. Also, when multiple combinations are possible, a volume-increasing voxel and a volume-decreasing voxel can be preferentially combined so that the sum of the distances is maximized. Furthermore, a volume-decreasing voxel can be prevented from being combined with a volume-increasing voxel in the next level above, since the luggage 4 cannot move upwards when it falls. A volume-increasing voxel can be prevented from being combined with a volume-decreasing voxel in the next level below, since the luggage 4 cannot move upwards when it falls. When the combination estimation unit 235 is able to estimate a combination, it outputs difference data including the extracted combination data to the movement amount estimation unit 236. When the combination estimation unit 235 is unable to estimate a combination, it outputs difference data from the lump extraction unit 234 to the movement amount estimation unit 236.

The movement amount estimation unit 236 is a functional unit that estimates the amount of movement of the luggage 4 that has occurred from a reference time to a time when a predetermined or arbitrary time has passed, based on the difference data (which may include the lump data and the combination data) from the combination estimation unit 235 (see FIG. 2). The movement amount estimation unit 236 can assume that the volume of the luggage 4 does not change. When the difference data includes the lump data and the combination data, the movement amount estimation unit 236 calculates the distance from the volume decrease voxel related to the combination data to the volume increase voxel, calculates the length of the voxel with no increase or decrease in volume related to the lump data, and estimates the value obtained by subtracting the calculated length from the calculated distance as the amount of movement of the luggage 4. When the difference data does not include the lump data and includes the combination data, the movement amount estimation unit 236 calculates the distance from the volume decrease voxel related to the combination data to the volume increase voxel, and estimates the calculated distance value as the amount of movement of the luggage 4. In addition, when the difference data does not include combination data (whether or not there is lump data), the movement of the luggage 4 involves a change in volume, so the movement amount estimation unit 236 does not estimate the movement amount, and the estimation of the change in volume amount by the change volume amount estimation unit 233 can be performed preferentially. In addition, in the estimation of the movement amount, correction may be performed so that different weights are assigned to horizontal movement and vertical movement (dropping), for example, the horizontal movement may be corrected to be smaller than the estimated movement amount so that no warning is issued even if the movement is large, and when a vertical or diagonal movement occurs, the movement amount may be corrected to be larger than the estimated movement amount so that even a small movement is warned. When the difference data includes multiple combination data, the movement amount estimation unit 236 estimates the movement amount for each combination data.

In addition, the movement amount estimation unit 236 selects the longest movement amount from the estimated movement amounts, and estimates the selected movement amount as the longest movement amount. Note that, if there is only one estimated movement amount, that movement amount is estimated as the longest movement amount. The movement amount estimation unit 236 outputs the estimated longest movement amount estimation data to the cargo collapse determination unit 237.

Examples of the operation of the cluster extraction unit 234, combination estimation unit 235, and movement amount estimation unit 236 will be described later.

The load collapse determination unit 237 is a functional unit that determines whether or not there is a possibility of load collapse of the luggage 4 by comparing the fluctuation volume amount estimation data from the fluctuation volume amount estimation unit 233 or the longest movement amount estimation data from the movement amount estimation unit 236 with a preset threshold (a first threshold for the fluctuation volume amount or a second threshold for the longest movement amount) (see FIG. 2). The load collapse determination unit 237 determines that there is a possibility of load collapse of the luggage 4 when the fluctuation volume amount estimation data is greater than the first threshold (or may be equal to or greater than the first threshold). The load collapse determination unit 237 determines that there is no possibility of load collapse of the luggage 4 when the fluctuation volume amount estimation data is equal to or less than the first threshold (or may be less than the first threshold). The load collapse determination unit 237 determines that there is a possibility of load collapse of the luggage 4 when the longest movement amount is greater than the second threshold (or may be equal to or greater than the second threshold). The load collapse determination unit 237 determines that there is no possibility of load collapse of the luggage 4 when the longest movement amount is equal to or less than the second threshold (it may be less than the second threshold). The load collapse determination unit 237 is free to use either the volume change amount or the longest movement amount to determine whether or not there is a possibility of load collapse of the luggage 4, but considering the data processing load, it is possible to make the determination by using the volume change amount, which imposes a relatively small load. When it is determined that there is a possibility of load collapse of the luggage 4, the load collapse determination unit 237 outputs warning output instruction information to the warning output unit 243 to warn the user of the occurrence of an abnormality.

The user interface unit 240 is a functional unit equipped with a user interface (a function for exchanging information with the user) (see FIG. 2). The user interface unit 240 acts as an interface between the user and the loading space recognition device 200, allowing the user to operate each process and check the results of each process. The user interface unit 240 is equipped with a display unit 241, an operation unit 242, and a warning output unit 243.

The display unit 241 is a functional unit that displays the preprocessed data 101 from the noise removal unit 212, etc. (see FIG. 2). As the display unit 241, for example, a liquid crystal display, an organic EL (Electroluminescence) display, AR (Augmented Reality) glasses, etc. may be used.

The operation unit 242 is a functional unit that issues area designation and determination instructions to the area designation unit 221 and the reference determination unit 231 based on user operations (see FIG. 2). As the operation unit 242, for example, a touch panel, a mouse, a camera and software that recognizes gestures and eye movements, etc. can be used.

The warning output unit 243 is a functional unit that outputs a warning to a user based on warning output instruction information from the cargo collapse determination unit 237 (see FIG. 2). For example, the warning output unit 243 may be a display that displays characters or images related to the warning, a speaker that outputs an alarm sound, a lamp that turns on an alarm, or a communication unit that transmits warning output instruction information to another system.

Next, the operation of the loading space recognition device in the loading space recognition system according to embodiment 1 will be described with reference to the drawings. Fig. 3 is a flow chart that illustrates the operation of the loading space recognition device in the loading space recognition system according to embodiment 1. Please refer to Figs. 1 and 2 and their explanations for the components and details of the loading space recognition device.

First, the format conversion unit 211 of the pre-processing unit 210 acquires reference photographic data 100 (reference photographic data; three-dimensional data) from the sensor 10, which photographs luggage 4 in the loading space 5, which is the photographing area (step A1).

Next, the format conversion unit 211 of the pre-processing unit 210 converts the format of the reference shooting data 100 into a common format (step A2).

Next, the noise removal unit 212 of the pre-processing unit 210 removes noise from the reference shooting data 100 converted into the common format to create reference pre-processed data 101 (step A3).

Next, the luggage overall image estimation unit 222 of the package shape grasping unit 220 estimates an overall image of the luggage 4 already loaded on the loading area based on the reference pre-processing data 101 and the loading area specified by the area designation unit 221 (step A4).

Next, the voxelization unit 223 of the package shape recognition unit 220 creates reference voxel data 102 (reference voxel data) relating to the overall image of the luggage 4 based on the estimation result data estimated in step A4 (step A5).

Next, the reference determination unit 231 of the determination unit 230 stores the reference voxel data created by the voxelization unit 223 as a reference time value for the cargo collapse determination process by the user operating the operation unit 242 (step A6). The user can operate the operation unit 242, for example, after loading of the cargo 4 into the container 3 is completed and before delivery is started.

Next, the format conversion unit 211 of the pre-processing unit 210 acquires comparison photography data 100 (comparison photography data; three-dimensional data) from the sensor 10, which is an image of the luggage 4 in the loading space 5, which is the photography area, at a time when a predetermined or arbitrary time has elapsed since acquiring the reference photography data 100 (reference photography data) (step A7).

Next, the format conversion unit 211 of the pre-processing unit 210 converts the format of the comparison shooting data 100 into a common format (step A8).

Next, the noise removal unit 212 of the pre-processing unit 210 removes noise from the comparison shooting data 100 converted into a common format to create comparison pre-processed data 101 (step A9).

Next, the luggage overall image estimation unit 222 of the package posture grasping unit 220 estimates an overall image of the luggage 4 already loaded on the loading area based on the pre-processing data 101 for comparison and the loading area specified by the area designation unit 221 (step A10).

Next, the voxelization unit 223 of the package shape recognition unit 220 creates comparison voxel data 102 (contrast voxel data) relating to the overall image of the luggage 4 based on the estimation result data estimated in step A10 (step A11).

Next, the difference extraction unit 232 of the judgment unit 230 compares the reference voxel data stored in the reference determination unit 231 with the comparison voxel data created by the voxelization unit 223 to extract voxel differences (e.g., differences in depth positions) at corresponding positions on the surface when the luggage 4 is viewed from a specified position (e.g., the rear of the truck 2, the position of the sensor 10) (step A12).

Next, the fluctuation volume estimation unit 233 of the judgment unit 230 estimates the changed volume (fluctuation volume) of the overall image of the luggage 4 based on the difference data from the difference extraction unit 232 (step A13).

Next, the load collapse determination unit 237 of the determination unit 230 determines whether the fluctuation volume estimation data from the fluctuation volume estimation unit 233 is greater than a first threshold value for the fluctuation volume amount that has been set in advance (step A14). If the fluctuation volume estimation is greater than the first threshold value (YES in step A14), it determines that there is a possibility of a load collapse of the luggage 4, outputs warning output instruction information to the warning output unit 243, and proceeds to step A20.

If the estimated volume change is equal to or less than the first threshold (NO in step A14), the chunk extraction unit 234 of the determination unit 230 extracts chunks of voxels with no increase or decrease in volume that are located between a volume increase voxel and a volume decrease voxel at the same horizontal position based on the difference data from the difference extraction unit 232 (step A15). If a chunk cannot be extracted, the process is skipped.

Next, the combination estimation unit 235 of the judgment unit 230 estimates combinations of volume-increasing voxels and volume-decreasing voxels based on the difference data from the cluster extraction unit 234 (including cluster data if clusters are extracted) (step A16).

Next, the movement amount estimation unit 236 of the determination unit 230 estimates the amount of movement of the luggage 4 that has occurred between the reference time and a predetermined or arbitrary time has elapsed based on the difference data (which may include the lump data and the combination data) from the combination estimation unit 235 (step A17).

Here, in estimating the amount of movement, if the difference data includes block data and combination data, the distance from the volume-decreasing voxel in the combination data to the volume-increasing voxel is calculated, the length of the voxel in the block data with no increase or decrease in volume is calculated, and the value obtained by subtracting the calculated length from the calculated distance is estimated as the amount of movement of the luggage 4. In addition, in estimating the amount of movement, if the difference data does not include block data but includes combination data, the distance from the volume-decreasing voxel in the combination data to the volume-increasing voxel is calculated, and the calculated distance value is estimated as the amount of movement of the luggage 4. In addition, in estimating the amount of movement, if the difference data includes multiple combination data, the amount of movement is estimated for each combination data.

Next, the movement amount estimation unit 236 of the judgment unit 230 selects the longest movement amount from the estimated movement amounts and estimates the selected movement amount as the longest movement amount (step A18).

Next, the load collapse determination unit 237 of the determination unit 230 determines whether the longest movement amount estimation data from the movement amount estimation unit 236 is greater than a second threshold value for the longest movement amount that has been set in advance (step A19). If the longest movement amount estimation data is greater than the second threshold value (YES in step A19), it determines that there is a possibility of load collapse of the luggage 4, outputs warning output instruction information to the warning output unit 243, and proceeds to step A20. If the longest movement amount estimation data is equal to or less than the second threshold value (NO in step A19), it determines that there is no possibility of load collapse of the luggage 4, ends one cycle, and repeats steps A7 to A20 until the user gives an instruction to end the cycle.

If the estimated volume fluctuation is greater than the first threshold (YES in step A14), or if the longest movement amount estimated data is greater than the second threshold (YES in step A19), the warning output unit 243 of the user interface unit 240 outputs a warning to the user based on the warning output instruction information from the cargo collapse determination unit 237 (step A20). After that, one cycle ends, and steps A7 to A20 are repeated until the user gives an instruction to end the cycle.

Next, the operation of the difference extraction unit, the volume fluctuation estimation unit, and the cargo collapse determination unit of the loading space recognition device in the loading space recognition system according to embodiment 1 will be described with reference to several examples and drawings. Figure 4 is an image diagram that shows several examples in which there is a volume fluctuation when the difference between a reference voxel and a comparison voxel is extracted.

If the reference voxel data when viewing the entire image of the luggage (4 in Figure 1) from the rear of the truck (2 in Figure 1) is in a state similar to the reference voxel data in Figure 4, then when the reference voxel data in Figure 4 changes to the contrast voxel data related to movement example 1-1, the difference extraction data extracted by the difference extraction unit (232 in Figure 2) will become like the difference extraction data related to movement example 1-1 in Figure 4. In the difference extraction data related to movement example 1-1, the volume of only the four voxels in the upper left increases by one level, and there is no increase or decrease in the other voxels. In other words, because there is only a one-level increase in volume for four voxels, the total volumes of the reference voxel data and the contrast voxel data are different. In this case, it is considered that the occlusion area (gaps or spaces) has increased behind the voxel whose volume has increased, so the volume fluctuation estimation unit (233 in Figure 2) estimates the volume fluctuation by assuming that the distance from the surface position of the reference voxel data to the surface position of the increased volume in the comparison voxel data has moved, and the load collapse determination unit (237 in Figure 2) can determine the threshold value of the volume fluctuation.

When the reference voxel data in Figure 4 changes to the comparison voxel data for movement example 1-2, the difference extraction data extracted by the difference extraction unit (232 in Figure 2) becomes the difference extraction data for movement example 1-2 in Figure 4. In the difference extraction data for movement example 1-2, the volume of four voxels decreases by one step, the volume of another eight voxels increases by one step, and there is no increase or decrease in the other voxels. In other words, there are increases and decreases in volume, but even if the increases and decreases in volume are offset, there is still four more voxels with an increase in volume of one step, so the total volume differs between the reference voxel data and the comparison voxel data. In this case, the luggage has moved overall, so it is impossible to calculate the exact amount of movement. In this case, the luggage 4 moves between four increases and four decreases in volume, and the remaining four increases in volume correspond to a movement in the distance from the surface position of the reference voxel data to the increased surface position of the comparison voxel data. The volume fluctuation estimation unit (233 in Figure 2) estimates the volume fluctuation, and the cargo collapse determination unit (237 in Figure 2) determines the threshold value of the volume fluctuation.

When the reference voxel data in Figure 4 changes to the comparison voxel data for Movement Example 1-3, the difference extraction data extracted by the difference extraction unit (232 in Figure 2) becomes the difference extraction data for Movement Example 1-3 in Figure 4. In the difference extraction data for Movement Example 1-3, the volume of eight voxels decreases by one level, the volume of another 12 voxels increases by one level, the volume of another four voxels increases by two levels, and there is no increase or decrease in the other voxels. In other words, there are increases and decreases in volume, but even when the increases and decreases in volume are offset, there are four more voxels with an increase in volume of one level, and four more voxels with an increase in volume of two levels, so the total volumes of the reference voxel data and the comparison voxel data are different. In this case, luggage 4 moves between an increase of 8 units in one step and a decrease of 8 units in one step, and the distance from the surface position of the reference voxel data to the increased surface position of the comparison voxel data moves by the remaining increase of 4 units in one step and an increase of 4 units in two steps. The amount of change in volume is estimated by the change in volume estimation unit (233 in Figure 2), and the load collapse determination unit (237 in Figure 2) determines the threshold value of the change in volume.

Next, the operation of the difference extraction unit, the lump extraction unit, the combination estimation unit, the movement amount estimation unit, and the load collapse determination unit of the loading space recognition device in the loading space recognition system according to the first embodiment will be described using several examples and drawings. FIG. 5 is an image diagram that shows several examples in which there is no volume fluctuation and there is a movement amount when the difference between the reference voxel and the comparison voxel is extracted. FIG. 6 is an image diagram that shows a transition from the lump estimation to the longest movement amount estimation in Example 2-1 of FIG. 5. FIG. 7 is an image diagram that shows a transition from the lump estimation to the longest movement amount estimation in Example 2-2 of FIG. 5. FIG. 8 is an image diagram that shows a transition from the lump estimation to the longest movement amount estimation in Example 2-3 of FIG. 5. FIG. 9 is an image diagram that shows a transition from the lump estimation to the longest movement amount estimation in Example 2-4 of FIG. 5.

When the reference voxel data when viewing the overall image of the luggage (4 in Figure 1) from the rear side of the truck (2 in Figure 1) is in a state similar to the reference voxel data in Figure 5, if the reference voxel data in Figure 5 changes to the contrast voxel data related to movement example 2-1, the difference extraction data extracted by the difference extraction unit (232 in Figure 2) will become like the difference extraction data related to movement example 2-1 in Figure 5. In the difference extraction data related to movement example 2-1, the volume of six voxels each increases by one step, the volume of another six voxels each decreases by one step, and there is no increase or decrease in the other voxels. In other words, there are increases and decreases in volume, and the increases and decreases in volume are offset by the same number and number of steps, so the total volume of the reference voxel data and the contrast voxel data is the same. In this case, the cluster extraction unit (234 in FIG. 2) extracts two clusters of voxels with no increase or decrease in volume that are located between a volume-increasing voxel and a volume-decreasing voxel at the same horizontal position as shown in FIG. 6A. The combination estimation unit (235 in FIG. 2) estimates two combinations of clusters of volume-increasing voxels and clusters of volume-decreasing voxels that are linked together as shown in FIG. 6B. The movement estimation unit (236 in FIG. 2) estimates the movement of the volume-increasing voxels from the volume-decreasing voxels related to the combination as shown in FIG. 6C. The distance to the cell (see the two arrows in Figure 6 (C)) is calculated, the length of the voxel (not shown) with no increase or decrease in volume related to the block data is calculated, and the value obtained by subtracting the calculated length from the distance calculated as in Figure 6 (D) is estimated as the amount of movement of the luggage 4 (see the two arrows in Figure 6 (D)). The movement amount estimation unit (236 in Figure 2) estimates the longest movement amount among the movement amounts as the maximum movement amount (see the circled arrow in Figure 6 (D)), and the cargo collapse determination unit (237 in Figure 2) determines the threshold value of the maximum movement amount.

When the reference voxel data in Figure 5 changes to the contrast voxel data for Movement Example 2-2, the difference extraction data extracted by the difference extraction unit (232 in Figure 2) becomes the difference extraction data for Movement Example 2-2 in Figure 5. In the difference extraction data for Movement Example 2-2, the volume of eight voxels increases by one step, the volume of another eight voxels decreases by one step, and there is no increase or decrease in the other voxels. In other words, there are increases and decreases in volume, and the increases and decreases in volume are offset by the same number and number of steps, so the total volumes of the reference voxel data and the contrast voxel data are the same. In this case, the cluster extraction unit (234 in FIG. 2) extracts one cluster of voxels with no increase or decrease in volume that exists between a volume-increasing voxel and a volume-decreasing voxel at the same horizontal position as in FIG. 7A, the combination estimation unit (235 in FIG. 2) estimates two combinations of a cluster of volume-increasing voxels and a cluster of volume-decreasing voxels that are linked together as in FIG. 7B, and the movement estimation unit (236 in FIG. 2) estimates the distance from the volume-decreasing voxel to the volume-increasing voxel related to the combination as in FIG. 7C (see the two arrows in FIG. 7C). Then, the length of a voxel (not shown; one) with no increase or decrease in volume related to the block of data is calculated, and the value obtained by subtracting the calculated length from the distance calculated as in FIG. 7(D) is estimated as the amount of movement of the luggage 4 (see the two arrows in FIG. 7(D); one with deduction and the other without deduction). A movement amount estimation unit (236 in FIG. 2) estimates the longest movement amount among the movement amounts as the longest movement amount (see the arrow surrounded by a circle in FIG. 7(D); in this case, there are two, but either one is acceptable), and a cargo collapse determination unit (237 in FIG. 2) determines the threshold value of the longest movement amount.

When the reference voxel data in Figure 5 changes to the contrast voxel data for Movement Example 2-3, the difference extraction data extracted by the difference extraction unit (232 in Figure 2) becomes the difference extraction data for Movement Example 2-3 in Figure 5. In the difference extraction data for Movement Example 2-3, the volume of eight voxels increases by one step, the volume of another eight voxels decreases by one step, and there is no increase or decrease in the other voxels. In other words, there are increases and decreases in volume, and the increases and decreases in volume are offset by the same number and number of steps, so the total volume of the reference voxel data and the contrast voxel data is the same. In this case, the cluster extraction unit (234 in FIG. 2) cannot extract clusters of voxels that are located between a volume increase voxel and a volume decrease voxel and have no change in volume, as shown in FIG. 8A, in the same horizontal position, so it skips them. The combination estimation unit (235 in FIG. 2) estimates two combinations of clusters of volume increase voxels and clusters of volume decrease voxels that are linked together, as shown in FIG. 8B, and the movement estimation unit (236 in FIG. 2) The distance from the volume-decreasing voxel to the volume-increasing voxel related to the combination is calculated as shown in Figure 8(C) (see the two arrows in Figure 8(C)), and the calculated distance is estimated as the movement amount of the luggage 4 as shown in Figure 8(D) (see the two arrows in Figure 8(D)).The movement amount estimation unit (236 in Figure 2) estimates the longest movement amount among the movement amounts as the maximum movement amount (see the circled arrow in Figure 8(D)), and the cargo collapse determination unit (237 in Figure 2) determines the threshold value of the maximum movement amount.

When the reference voxel data in Figure 5 changes to the contrast voxel data for Movement Example 2-4, the difference extraction data extracted by the difference extraction unit (232 in Figure 2) becomes the difference extraction data for Movement Example 2-4 in Figure 5. In the difference extraction data for Movement Example 2-4, the volume of 12 voxels increases by one step, the volume of another 12 voxels decreases by one step, and there is no increase or decrease in the other voxels. In other words, there are increases and decreases in volume, and the increases and decreases in volume are offset by the same number and number of steps, so the total volume of the reference voxel data and the contrast voxel data is the same. In this case, the cluster extraction unit (234 in FIG. 2) cannot extract clusters of voxels that are located between a volume increase voxel and a volume decrease voxel and have no increase or decrease in volume, as shown in FIG. 9A, and so skips them. The combination estimation unit (235 in FIG. 2) estimates three combinations of clusters of volume increase voxels and clusters of volume decrease voxels that are linked together, as shown in FIG. 9B, and the movement estimation unit (236 in FIG. 2) The distance from the volume-decreasing voxel to the volume-increasing voxel related to the combination is calculated as shown in Figure 9(C) (see the three arrows in Figure 9(C)), and the calculated distance is estimated as the movement amount of the luggage 4 as shown in Figure 9(D) (see the three arrows in Figure 9(D)).The movement amount estimation unit (236 in Figure 2) estimates the longest movement amount among the movement amounts as the maximum movement amount (see the circled arrow in Figure 9(D)), and the cargo collapse determination unit (237 in Figure 2) determines the threshold value of the maximum movement amount.

Please refer to the detailed explanation in Figure 2 for the criteria for the lump extraction process by the lump extraction unit 234, the criteria for the combination estimation process by the combination estimation unit 235, and the criteria for the movement amount estimation process by the movement amount estimation unit 236 in the above movement examples 1-1 to 1-3 and 2-1 to 2-4.

According to embodiment 1, the difference between voxels at corresponding positions between the reference voxel data and the comparison voxel data is extracted to estimate the volumetric change or the maximum movement amount of the overall image of the luggage 4, and a threshold judgment is performed, which can contribute to determining the possibility of cargo collapse due to the movement of the luggage.

Furthermore, according to embodiment 1, even if an occlusion portion hidden by luggage 4 occurs, the occlusion portion is estimated and the shape of the luggage 4 is grasped, so that it is possible to take into account changes in the luggage 4 in the occlusion portion.

Furthermore, according to the first embodiment, by retaining reference voxel data of the state of the package at a specific time point, such as before the truck 2 starts moving, and comparing it with the contrast voxel data of the state of the package after a predetermined or arbitrary time has passed from the reference time, it is possible to detect the possibility of the package collapsing, so that even if there is package that has become unstable due to loading and unloading work and is in danger of falling, it is possible to detect this and notify the driver. This makes it possible to detect the possibility of the package collapsing of the package 4 at an early stage and prevent damage to the package 4, thereby contributing to preventing a decline in transport quality.

[Embodiment 2]
The loading space recognition device according to the second embodiment will be described with reference to the drawings. Fig. 11 is a block diagram showing a schematic configuration of the loading space recognition device according to the second embodiment.

The loading space recognition device 200 is a device that recognizes changes in the luggage (volume changes, distance changes) in the loading space in which the luggage is loaded based on the photographic data. The loading space recognition device 200 includes a luggage overall image estimation unit 222, a voxelization unit 223, and a determination unit 230.

The luggage overall image estimation unit 222 is configured to estimate an overall image of the luggage loaded in the loading space based on three-dimensional data captured from a specified direction of the luggage loading space, and output the estimated result data.

The voxelization unit 223 is configured to voxelize the estimation result data and output it as voxel data.

The determination unit 230 is configured to estimate the amount of change in volume or movement of the luggage by comparing voxel data at an arbitrary reference time with voxel data after a predetermined or arbitrary time has elapsed from the reference time, and to determine whether or not there is a possibility of the luggage collapsing by comparing the estimated amount of change in volume or movement with a threshold value.

According to embodiment 2, the difference between voxels at corresponding positions between voxel data at the reference time and voxel data other than the reference time is extracted, the amount of change in volume or the maximum amount of movement of the overall image of the luggage is estimated, and a threshold judgment is performed, which can contribute to determining the possibility of cargo collapse due to the movement of luggage.

The loading space recognition device according to the first and second embodiments can be configured by so-called hardware resources (information processing device, computer), and can use a device having the configuration shown in Fig. 12. For example, the hardware resource 1000 includes a processor 1001, a memory 1002, a network interface 1003, etc., which are connected to each other by an internal bus 1004.

Note that the configuration shown in FIG. 12 is not intended to limit the hardware configuration of the hardware resource 1000. The hardware resource 1000 may include hardware (e.g., an input/output interface) that is not shown. Furthermore, the number of units such as the processor 1001 included in the device is not intended to be limited to the example shown in FIG. 12, and for example, multiple processors 1001 may be included in the hardware resource 1000. The processor 1001 may be, for example, a CPU (Central Processing Unit), an MPU (Micro Processor Unit), a GPU (Graphics Processing Unit), etc.

The memory 1002 may be, for example, a RAM (Random Access Memory), a ROM (Read Only Memory), a HDD (Hard Disk Drive), a SSD (Solid State Drive), etc.

The network interface 1003 may be, for example, a LAN (Local Area Network) card, a network adapter, a network interface card, etc.

The functions of the hardware resource 1000 are realized by the above-mentioned processing module. The processing module is realized, for example, by the processor 1001 executing a program stored in the memory 1002. The program can be downloaded via a network or updated using a storage medium on which the program is stored. Furthermore, the processing module may be realized by a semiconductor chip. In other words, it is sufficient that the functions performed by the processing module are realized by the execution of software in some kind of hardware.

Some or all of the above embodiments may be described as follows, but are not limited to:

[Appendix 1]
a luggage overall image estimation unit configured to estimate an overall image of the luggage loaded in the loading space based on three-dimensional data obtained by capturing an image of the luggage loading space from a predetermined direction, and output the estimated result data;
a voxelization unit configured to voxelize the estimation result data and output it as voxel data;
a determination unit configured to estimate a volumetric fluctuation or movement amount of a cargo by comparing the voxel data at an arbitrary reference time with the voxel data after a predetermined or arbitrary time has elapsed from the reference time, and to determine whether or not there is a possibility of cargo collapse by comparing the estimated volumetric fluctuation or movement amount with a threshold value;
A loading space recognition device comprising:
[Appendix 2]
The determination unit is
a reference determination unit configured to determine the voxel data at the reference time from the voxelization unit as a change reference point;
a difference extraction unit configured to extract a difference between voxels at corresponding positions on a surface of the baggage viewed from a predetermined position by comparing the voxel data at the reference time with the voxel data at a time point when a predetermined or arbitrary time has elapsed from the reference time, and output the difference data;
a volume fluctuation estimation unit configured to estimate a volume fluctuation of an entire image of the package based on the difference data and output the volume fluctuation estimation data;
a cargo collapse determination unit configured to determine whether or not there is a possibility of cargo collapse by comparing the fluctuation volume amount estimation data with a threshold value for the fluctuation volume amount as the threshold value;
2. The loading space recognition device of claim 1, comprising:
[Appendix 3]
The determination unit is
a chunk extraction unit configured to extract chunks of voxels with no increase or decrease in volume that exist between a volume-increasing voxel whose volume has increased and a volume-decreasing voxel whose volume has decreased, which are located at the same horizontal position, based on the difference data, and output the chunk data;
a combination estimation unit configured to estimate a combination of the volume-increased voxels and the volume-decreased voxels based on the difference data and output the combination data as combination data;
a movement amount estimation unit configured to estimate at least one movement amount of luggage that has occurred from the reference time to a time when a predetermined or arbitrary time has elapsed based on the difference data, the collection data, and the combination data, select the longest movement amount from the estimated movement amounts, estimate the selected movement amount as a longest movement amount, and output it as longest movement amount estimated data;
Further equipped with
The cargo collapse determination unit is configured to determine whether or not there is a possibility of cargo collapse by comparing the fluctuation volume amount estimation data with a threshold value for the fluctuation volume amount as the threshold value, and when it is determined that there is a possibility of cargo collapse, to determine whether or not there is a possibility of cargo collapse by comparing the maximum movement amount estimation data with a threshold value for the maximum movement amount as the threshold value.
[Appendix 4]
The determination unit is
a reference determination unit configured to determine the voxel data at the reference time from the voxelization unit as a change reference point;
a difference extraction unit configured to extract a difference between voxels at corresponding positions on a surface of the baggage viewed from a predetermined position by comparing the voxel data at the reference time with the voxel data at a time point when a predetermined or arbitrary time has elapsed from the reference time, and output the difference data;
a chunk extraction unit configured to extract chunks of voxels with no increase or decrease in volume that exist between a volume-increasing voxel whose volume has increased and a volume-decreasing voxel whose volume has decreased, which are located at the same horizontal position, based on the difference data, and output the chunk data;
a combination estimation unit configured to estimate a combination of the volume-increased voxels and the volume-decreased voxels based on the difference data and output the combination data as combination data;
a movement amount estimation unit configured to estimate at least one movement amount of luggage that has occurred from the reference time to a time when a predetermined or arbitrary time has elapsed based on the difference data, the collection data, and the combination data, select the longest movement amount from the estimated movement amounts, estimate the selected movement amount as a longest movement amount, and output it as longest movement amount estimated data;
a cargo collapse determination unit configured to determine whether or not there is a possibility of cargo collapse by comparing the longest movement amount estimation data with a threshold value for the longest movement amount as the threshold value;
2. The loading space recognition device of claim 1, comprising:
[Appendix 5]
The vehicle further includes an area designation unit configured to designate a luggage loading area in the loading space by a user's operation,
The luggage overall image estimation unit is configured to estimate an overall image of luggage loaded in the loading area.
5. The loading space recognition device according to claim 1 .
[Appendix 6]
the region designation unit is configured to designate, in response to a user's operation, a determination exclusion region to be excluded from determination by the determination unit;
the baggage overall image estimation unit is configured to estimate an overall image of the baggage loaded in the loading area by excluding the baggage loaded in the judgment exclusion area;
6. The loading space recognition device according to claim 5.
[Appendix 7]
A detection unit is attached to the structure having the loading space and detects vibration or sound,
The determination unit is configured to acquire the voxel data from the voxelization unit when the detection unit detects a certain level of shaking or sound, and estimate the amount of change in volume or movement of the luggage by comparing the acquired voxel data with the voxel data at the reference time.
7. The loading space recognition device according to any one of claims 1 to 6.
[Appendix 8]
the movement amount estimation unit is configured to correct the movement amount to be smaller than the estimated movement amount when the movement direction of the luggage is a horizontal direction, or to correct the movement amount to be larger than the estimated movement amount when the movement direction of the luggage is a vertical direction or a diagonal direction, and to select the longest movement amount from the corrected movement amounts.
5. The loading space recognition device according to claim 3 or 4.
[Appendix 9]
The determination unit is further configured to determine whether or not there is a possibility of cargo collapse by comparing an amount of change between the voxel data other than the reference time and the other voxel data immediately before the reference time and an amount of change between the voxel data at the reference time and the voxel data at the other than the reference time.
9. The loading space recognition device according to any one of claims 1 to 8.
[Appendix 10]
The determination unit is configured to output warning output instruction information when it determines that there is a possibility of collapse of luggage,
The loading space recognition device further includes a warning output unit configured to output a warning based on the warning output instruction information.
10. The loading space recognition device according to any one of claims 1 to 9.
[Appendix 11]
A sensor that senses the surface of the luggage in the loading space and outputs three-dimensional data obtained by capturing an image of the surface;
A loading space recognition device according to any one of appendix 1 to 10,
Equipped with a loading space recognition system.
[Appendix 12]
A method for recognizing a loading space of a load using hardware resources, comprising:
a step of estimating an overall image of the luggage loaded in the loading space based on three-dimensional data obtained by capturing an image of the loading space from a predetermined direction, and outputting the estimated result data;
A step of converting the estimation result data into voxel data and outputting the voxel data;
a step of estimating a volumetric change or movement amount of the luggage by comparing the voxel data at an arbitrary reference time with the voxel data after a predetermined or arbitrary time has elapsed from the reference time, and judging whether or not there is a possibility of the luggage collapse by comparing the estimated volumetric change or movement amount with a threshold value;
A method for recognizing a load space, comprising:
[Appendix 13]
A program for causing a hardware resource to execute a process for recognizing a loading space of a luggage,
A process of estimating an overall image of the luggage loaded in the loading space based on three-dimensional data obtained by capturing an image of the loading space from a predetermined direction, and outputting the estimated result data;
A process of converting the estimation result data into voxel data and outputting the voxel data;
A process of estimating the amount of change in volume or movement of the luggage by comparing the voxel data at an arbitrary reference time with the voxel data after a predetermined or arbitrary time has elapsed from the reference time, and determining whether or not there is a possibility of the luggage collapse by comparing the estimated amount of change in volume or movement with a threshold value;
A program for causing the hardware resource to execute the above.

The disclosures of the above patent documents are incorporated herein by reference and may be used as the basis or part of the present invention as necessary. Within the framework of the entire disclosure of the present invention (including the claims and drawings), and further based on the basic technical ideas, modifications and adjustments of the embodiments and examples are possible. Furthermore, within the framework of the entire disclosure of the present invention, various disclosed elements (including each element of each claim, each element of each embodiment or example, each element of each drawing, etc.) may be combined or selected (or not selected as necessary). In other words, the present invention naturally includes various modifications and corrections that a person skilled in the art would be able to make in accordance with the entire disclosure, including the claims and drawings, and the technical ideas. Furthermore, with regard to the numerical values and numerical ranges described in this application, any intermediate value, lower numerical value, and small range are deemed to be described even if not specified. Furthermore, the disclosures of the above cited documents may be used in part or in whole in combination with the descriptions in this document as part of the disclosure of the present invention in accordance with the spirit of the present invention as necessary, and are deemed to be included (belong) to the disclosures of this application.

1 Loading space recognition system 2 Truck 3 Container 4 Baggage 5 Loading space 10 Sensor 20 Detection unit 100 Photographed data 101 Preprocessed data 102 Voxel data 200 Loading space recognition device 210 Preprocessing unit 211 Format conversion unit 212 Noise removal unit 220 Package shape recognition unit 221 Area designation unit 222 Package overall image estimation unit 223 Voxelization unit 230 Determination unit 231 Reference determination unit 232 Difference extraction unit 233 Fluctuation volume amount estimation unit 234 Lumping extraction unit 235 Combination estimation unit 236 Movement amount estimation unit 237 Load collapse determination unit 240 User interface unit 241 Display unit 242 Operation unit 243 Warning output unit 1000 Hardware resources 1001 Processor 1002 Memory 1003 Network interface 1004 Internal bus

Claims (9)

a luggage overall image estimation unit configured to estimate an overall image of the luggage loaded in the loading space based on three-dimensional data obtained by capturing an image of the luggage loading space from a predetermined direction, and output the estimated result data;
a voxelization unit configured to voxelize the estimation result data and output it as voxel data;
a determination unit configured to estimate a volumetric fluctuation or movement amount of a cargo by comparing the voxel data at an arbitrary reference time with the voxel data after a predetermined or arbitrary time has elapsed from the reference time, and to determine whether or not there is a possibility of cargo collapse by comparing the estimated volumetric fluctuation or movement amount with a threshold value;
an area designation unit configured to designate a luggage loading area in the loading space by a user's operation,
The luggage overall image estimation unit is configured to estimate an overall image of luggage loaded in the loading area . The determination unit is
a reference determination unit configured to determine the voxel data at the reference time from the voxelization unit as a change reference point;
a difference extraction unit configured to compare the voxel data at the reference time with the voxel data at a time point when a predetermined or arbitrary time has elapsed from the reference time, thereby extracting a difference between voxels at corresponding positions on a surface of the baggage viewed from a predetermined position, and outputting the difference data;
a volume fluctuation estimation unit configured to estimate a volume fluctuation of an entire image of the package based on the difference data and output the volume fluctuation estimation data;
a cargo collapse determination unit configured to determine whether or not there is a possibility of cargo collapse by comparing the fluctuation volume amount estimation data with a threshold value for the fluctuation volume amount as the threshold value;
The load space recognition device according to claim 1 . The determination unit is
a chunk extraction unit configured to extract chunks of voxels with no increase or decrease in volume that exist between a volume-increasing voxel whose volume has increased and a volume-decreasing voxel whose volume has decreased, which are located at the same horizontal position, based on the difference data, and output the chunk data;
a combination estimation unit configured to estimate a combination of the volume-increased voxels and the volume-decreased voxels based on the difference data and output the combination data as combination data;
a movement amount estimation unit configured to estimate at least one movement amount of luggage that has occurred from the reference time to a time when a predetermined or arbitrary time has elapsed based on the difference data, the collection data, and the combination data, select the longest movement amount from the estimated movement amounts, estimate the selected movement amount as a longest movement amount, and output it as longest movement amount estimated data;
Further equipped with
The loading space recognition device according to claim 2, wherein the cargo collapse determination unit is configured to determine whether or not there is a possibility of cargo collapse by comparing the fluctuation volume amount estimation data with a threshold value for the fluctuation volume amount as the threshold value, and when it is determined that there is a possibility of cargo collapse, to determine whether or not there is a possibility of cargo collapse by comparing the maximum movement amount estimation data with a threshold value for the maximum movement amount as the threshold value. The determination unit is
a reference determination unit configured to determine the voxel data at the reference time from the voxelization unit as a change reference point;
a difference extraction unit configured to compare the voxel data at the reference time with the voxel data at a time point when a predetermined or arbitrary time has elapsed from the reference time, thereby extracting a difference between voxels at corresponding positions on a surface of the baggage viewed from a predetermined position, and outputting the difference data;
a chunk extraction unit configured to extract chunks of voxels with no increase or decrease in volume that exist between a volume-increasing voxel whose volume has increased and a volume-decreasing voxel whose volume has decreased, which are located at the same horizontal position, based on the difference data, and output the chunk data;
a combination estimation unit configured to estimate a combination of the volume-increased voxels and the volume-decreased voxels based on the difference data and output the combination data as combination data;
a movement amount estimation unit configured to estimate at least one movement amount of luggage that has occurred from the reference time to a time when a predetermined or arbitrary time has elapsed based on the difference data, the collection data, and the combination data, select the longest movement amount from the estimated movement amounts, estimate the selected movement amount as a longest movement amount, and output it as longest movement amount estimated data;
a cargo collapse determination unit configured to determine whether or not there is a possibility of cargo collapse by comparing the longest movement amount estimation data with a threshold value for the longest movement amount as the threshold value;
The load space recognition device according to claim 1 . the movement amount estimation unit is configured to correct the movement amount to be smaller than the estimated movement amount when the movement direction of the luggage is a horizontal direction, or to correct the movement amount to be larger than the estimated movement amount when the movement direction of the luggage is a vertical direction or a diagonal direction, and to select the longest movement amount from the corrected movement amounts.
The loading space recognition device according to claim 3 or 4. The determination unit is further configured to determine whether or not there is a possibility of cargo collapse by comparing an amount of change between the voxel data other than the reference time and the other voxel data immediately before the reference time and an amount of change between the voxel data at the reference time and the voxel data at the other than the reference time.
The loading space recognition device according to any one of claims 1 to 5 . A sensor that senses the surface of the luggage in the loading space and outputs three-dimensional data obtained by capturing an image of the surface;
The loading space recognition device according to any one of claims 1 to 6 ,
Equipped with a loading space recognition system. A method for recognizing a loading space of a load using hardware resources, comprising:
A step of designating a luggage loading area in the loading space by a user operation;
a step of estimating an overall image of the luggage loaded in the luggage loading area in the loading space based on three-dimensional data obtained by capturing an image of the loading space from a predetermined direction, and outputting the overall image as estimation result data;
A step of converting the estimation result data into voxel data and outputting the voxel data;
a step of estimating a volumetric change or movement amount of the luggage by comparing the voxel data at an arbitrary reference time with the voxel data after a predetermined or arbitrary time has elapsed from the reference time, and determining whether or not there is a possibility of the luggage collapsing by comparing the estimated volumetric change or movement amount with a threshold value. A program for causing a hardware resource to execute a process for recognizing a loading space of a luggage,
A process of designating a cargo loading area in the loading space by a user operation;
a process of estimating an overall image of the luggage loaded in the luggage loading area in the loading space based on three-dimensional data obtained by capturing an image of the loading space from a predetermined direction, and outputting the estimated result data;
A process of converting the estimation result data into voxel data and outputting the voxel data;
A process of estimating the amount of change in volume or movement of the luggage by comparing the voxel data at an arbitrary reference time with the voxel data after a predetermined or arbitrary time has elapsed from the reference time, and determining whether or not there is a possibility of the luggage collapse by comparing the estimated amount of change in volume or movement with a threshold value;
A program for causing the hardware resource to execute the above.

Images