JP7847362B2 - Volume estimation device and volume estimation method - Google Patents

Description

This invention relates to a volume estimation device and a volume estimation method for estimating the volume of waste.

In recent years, from the perspective of efficient resource utilization, there has been a growing demand to reuse waste as recyclable resources. Mixed waste, such as construction waste, which contains different types of waste, can be transformed into valuable resources by sorting the waste by type and crushing it to facilitate reuse. Mixed waste undergoes sorting and crushing at the treatment plant. The amount of mixed waste brought to the treatment plant is estimated.

Patent Document 1 discloses a conventional volume estimation system for estimating the volume of waste. This volume estimation system acquires depth image data, including depth information for each pixel, from a depth camera that captures the three-dimensional space on the truck bed where the waste is loaded. The volume estimation system obtains the distance from the depth camera to the region corresponding to a pixel in the empty state of the depth image data, based on the depth information of the pixels in the empty state of the depth image data (when no waste is loaded on the truck bed), as the empty state distance. The volume estimation system also obtains the distance from the depth camera to the region corresponding to a pixel in the operating state of the depth image data (when waste is loaded on the truck bed), based on the depth information of the pixels in the operating state, as the operating state distance. Finally, the volume estimation system estimates the volume of waste on the truck bed based on the empty state distance and the operating state distance.

Japanese Patent Publication No. 2019-196927

A waste generator entrusts the disposal of mixed waste to a waste treatment company. At this time, the waste generator creates an industrial waste management manifest (hereinafter referred to as "manifest") that records the types and quantities of waste contained in the mixed waste. The waste treatment company then transports the entrusted mixed waste to the treatment site and verifies that the types and quantities of waste contained in the mixed waste match those recorded in the manifest.

The volume estimation system described above can estimate the total volume of mixed waste loaded onto a truck when it is transported to a waste disposal site. However, the volume estimation system cannot estimate the volume of each type of waste when the mixed waste contains multiple types of waste. Therefore, as shown in Figure 8, workers at the waste disposal site had to spread the mixed waste on the floor and visually estimate the types and volumes of the various wastes contained in the mixed waste, and verify whether it matched the manifest. This verification process was time-consuming, and the accuracy of the verification sometimes depended on the worker's experience.

Therefore, the present invention aims to provide a volume estimation device and volume estimation method that can stably estimate multiple types and volumes of waste with high accuracy.

To achieve the above objective, a volume estimation device according to one aspect of the present invention is characterized by comprising: an identification unit that identifies a plurality of waste materials based on image data of the plurality of waste materials captured by a photography device; a type estimation unit that estimates the type of each of the plurality of waste materials based on the image data; a volume estimation unit that estimates the volume of each of the plurality of waste materials based on the image data; a summation unit that calculates the total volume for each type of waste material; and an output unit that outputs the summation value.

According to this invention, the type and volume of waste can be estimated by image recognition without visual inspection by an operator, and the total volume for each type can be calculated. Therefore, the types and volumes of multiple types of waste can be estimated with high accuracy and stability.

In the present invention, it is preferable that the imaging device includes a depth camera positioned above the surface on which the plurality of wastes are placed, the image data includes depth image data captured by the depth camera, each containing depth information, and the volume estimation unit estimates the volume of each of the plurality of wastes based on the depth image data. This allows for the estimation of the volume of each of the plurality of wastes with higher accuracy.

In this invention, it is preferable that the imaging device includes a mobile camera that can move above the surface on which the plurality of wastes are placed, and that the volume estimation unit estimates the volume of each of the plurality of wastes based on the image data captured by the mobile camera at a first position and the image data captured by the mobile camera at a second position different from the first position. In this way, the volume of each of the plurality of wastes can be estimated based on parallax. By using multiple image data captured by the mobile camera at multiple different positions, the volume of each of the plurality of wastes can be estimated with higher accuracy.

In this invention, it is preferable that the imaging device includes a mobile camera that can move above the surface on which the plurality of wastes are placed, and that the type estimation unit estimates the type of each of the plurality of wastes based on the image data captured by the mobile camera. This allows, for example, the mobile camera to be moved upwards to avoid interfering with the work when multiple wastes are being brought in, and to be moved closer to the multiple wastes for imaging. The mobile camera may also be moved laterally. Therefore, work efficiency can be improved, and the type of each of the multiple wastes can be estimated with higher accuracy.

In this invention, it is preferable that the moving camera is a camera mounted on a drone capable of flying above the aforementioned mounting surface. This eliminates the need to install, for example, a crane for moving the camera, allowing the camera to be moved with a simple configuration.

In this invention, it is preferable that the output unit outputs notification information indicating that the plurality of wastes, whose types have been estimated, contain waste of a type other than those predetermined (hereinafter referred to as "unspecified type"). By doing so, for example, by pre-specifying the types of waste expected to be included in the plurality of wastes based on the manifest, it is possible to notify the system if an unexpected type of waste (unspecified type) is mixed in with the plurality of wastes.

In this invention, it is preferable that the output unit outputs notification information indicating that a specified type of waste is included in the plurality of wastes whose types have been estimated. By doing so, for example, by pre-specifying a type of waste requiring special attention, it is possible to notify that a specified type of waste is mixed in with the plurality of wastes.

In this invention, it is preferable that the output unit outputs notification information indicating that the waste materials exceeding a predetermined volume limit are included in the plurality of waste materials whose volumes have been estimated. By doing so, for example, by setting a volume limit for waste materials that can be processed in a subsequent process, it is possible to notify that the plurality of waste materials contain waste materials that cannot be processed in a subsequent process.

To achieve the above objective, another aspect of the present invention provides a volume estimation method characterized in that a computer identifies a plurality of waste materials based on image data of the waste materials captured by a camera, estimates the type of each of the plurality of waste materials based on the image data, estimates the volume of each of the plurality of waste materials based on the image data, calculates the total volume for each type of waste material, and outputs the total volume.

According to this invention, the type and volume of waste can be estimated by image recognition without visual inspection by an operator, and the total volume for each type can be calculated. Therefore, the types and volumes of multiple types of waste can be estimated with high accuracy and stability.

According to this invention, the types and volumes of multiple types of waste can be estimated with high accuracy and stability.

This figure illustrates the schematic configuration of a volume estimation system having a volume estimation device according to one embodiment of the present invention. This diagram illustrates the functional blocks of the volume estimation system shown in Figure 1. This diagram schematically illustrates the three-dimensional space captured by a depth camera. This flowchart shows an example of the processing (volume estimation process) in the control device of the volume estimation device shown in Figure 1. This figure shows an example of multiple waste materials captured by the camera of the volume estimation system shown in Figure 1. This figure shows an example of the results of estimating the type and volume of each of the multiple waste materials contained in mixed waste. The figure shows an example of the results of calculating the total volume for each type of waste contained in mixed waste. This figure shows an example of volume estimation work performed by a worker.

The following describes a volume estimation system having a volume estimation device according to an embodiment of the present invention, with reference to Figures 1 to 7.

Figure 1 is a diagram illustrating the schematic configuration of a volume estimation system having a volume estimation device according to one embodiment of the present invention. Figure 2 is a diagram illustrating the functional blocks of the volume estimation system in Figure 1. Figure 3 is a diagram schematically showing the three-dimensional space captured by the depth camera. Figure 4 is a flowchart showing an example of processing (volume estimation processing) in the control device of the volume estimation device in Figure 1. Figure 5 is a diagram showing an example of multiple waste materials captured by the camera of the volume estimation system in Figure 1. Figure 6 is a diagram showing an example of the results of estimating the type and volume of each of the multiple waste materials contained in mixed waste. Figure 7 is a diagram showing an example of the results of calculating the total volume value for each type of waste material contained in mixed waste.

Volume estimation system 1 calculates the total volume of each type of waste contained in mixed waste placed on a horizontal floor F of a waste disposal site. Floor F is the surface on which the multiple wastes are placed. Examples of waste types included in the mixed waste include wood scraps, metal scraps, waste plastics, glass/concrete/ceramic waste, and rubble.

As shown in Figure 1, the volume estimation system 1 includes a camera 20 as an imaging device and a control device 40 as a volume estimation device.

Camera 20 is fixedly positioned above the floor F. Camera 20 is directed towards the shooting area S of the floor F. In Figures 1 and 5, the shooting area S of camera 20 is shown by a dashed line.

Camera 20 includes a digital still camera. Camera 20 captures the shooting area S and generates still image data.

Furthermore, camera 20 includes a depth camera. Camera 20 captures images of the shooting area S and generates depth image data. Specifically, camera 20 detects ultrasonic waves or laser light reflected from the floor F or the surface of waste W placed on the floor F, acquires information related to the distance from camera 20 to the reflective surface (referred to as "depth information"), and generates depth image data including depth information for each pixel.

The image data generated by camera 20 is composed of numerous pixels arranged vertically and horizontally, and more specifically, is a collection of data corresponding to numerous pixels. In this embodiment, the still image data is, for example, image data relating to a color image in which each pixel is represented in 24-bit color. The depth image data is, for example, image data relating to a grayscale image in which each pixel is represented by shades of gray corresponding to depth. The depth image data is represented as 256 shades of gray for each pixel, such that the color becomes lighter as the distance decreases and darker as the distance increases. The depth image data may also be represented by colors corresponding to depth for each pixel. The depth image data only needs to contain depth information for each pixel, and its viewing mode is arbitrary.

Figure 3 schematically shows the three-dimensional space captured by the depth camera. The depth camera is located at the vertex (position P0) of the pyramidal three-dimensional space and captures this three-dimensional space to generate depth image data. When two planar shooting areas S1 and S2 are captured in the three-dimensional space, the area of shooting area S1 at position P1, which is a distance L1 from position P0, is larger than the area of the planar shooting area S2 at position P2, which is a distance L2 (L2 < L1) from position P0. Therefore, the area s1 corresponding to the pixels in the depth image data generated when shooting area S1 is captured is larger than the area s2 corresponding to the pixels in the depth image data generated when shooting area S2 is captured. In other words, the area of the area corresponding to the pixels containing depth information indicating a large distance from the depth camera is larger than the area of the area corresponding to the pixels containing depth information indicating a small distance from the depth camera. From this, it can be seen that the area corresponding to the pixels in the depth image data has an area corresponding to the distance from the depth camera. The area of each region may be calculated based on the field of view of the depth camera, or it may be obtained using a table that associates depth information (distance from the depth camera) with the area of each region.

The volume of waste W is estimated as follows: The distance from the depth camera to the floor F is measured in advance. Pixels inside the outline of waste W are extracted from depth image data of waste W placed on the floor F. Based on the depth information of each extracted pixel, the area of the region corresponding to each pixel and the distance (height) to the floor F are obtained. The height is obtained by subtracting the distance indicated in the depth information from the distance from the depth camera to the floor F. The volume is calculated by multiplying the obtained area for the region corresponding to each pixel by the height, and the sum of these calculated volumes is taken as the volume of waste W.

The control device 40 controls the operation of the entire volume estimation system 1. The control device 40 has a computer. The computer may be a personal computer, a workstation, or a large-scale computer such as a server. The computer has a central processing unit (CPU) and a program storage device consisting of memory and a hard disk drive that stores various programs executed by the CPU. A tablet terminal 50 is connected to the control device 40 via a wireless communication line. The tablet terminal 50 is used as a display device for displaying information output by the volume estimation system 1 and as an input device for receiving operations on the volume estimation system 1.

The control device 40 may be connected to a display device such as a liquid crystal display or an organic EL display, as well as an input device such as a keyboard and mouse, instead of the tablet terminal 50. Alternatively, the control unit (not shown) of the computer in the tablet terminal 50 may be used as the control device 40.

Furthermore, a storage device 60, such as an external hard disk drive, is connected to the control device 40. The control device 40 stores the results of various processes in the storage device 60.

The storage device 60 also stores training data used for learning to estimate the type of waste W. This training data includes combinations of pre-prepared images of waste W (e.g., still image data of wood chips) and the type of waste W contained in those images (e.g., "wood chips"). The generation of this training data is disclosed in Japanese Patent Application Publication No. 2017-109161, filed by the present applicant, and a detailed explanation is omitted here. The control device 40, the storage device 60, and the tablet terminal 50 may be connected, for example, via the Internet.

The computer of the control device 40 functions as the identification unit 41, type estimation unit 42, volume estimation unit 43, total unit 44, and output unit 45 shown in Figure 2, by executing the program (volume estimation program) stored in the program storage device.

The identification unit 41 acquires image data (still image data and depth image data) output from the camera 20. Based on the image data, the identification unit 41 identifies multiple waste materials W placed in the shooting area S on the floor F and acquires the position and shape of each of the waste materials W. The identification unit 41 processes the image data, such as by cropping the portion corresponding to each of the waste materials W, and outputs it to the type estimation unit 42 and the volume estimation unit 43. Alternatively, the identification unit 41 may output the image data to the type estimation unit 42 and the volume estimation unit 43 without processing it.

The type estimation unit 42 estimates the type of each of the multiple waste materials W identified by the identification unit 41. Specifically, the type estimation unit 42 acquires image data from the identification unit 41 and performs pattern matching processing on this image data using recognition pattern data prepared for each type, thereby estimating the type of each of the multiple waste materials W. Of course, the type may also be estimated by other processing methods. The recognition pattern data used for estimation is updated by machine learning using training data stored in the storage device 60.

The type estimation unit 42 reads training data from the storage device 60 at predetermined timings, such as periodically, during periods when the volume estimation system 1 is not performing volume estimation processing, or when new training data is added to the storage device 60, and updates the recognition pattern data by performing machine learning. In this embodiment, the type estimation unit 42 is configured to perform machine learning using a convolutional neural network with an optimization method based on backpropagation. Of course, machine learning may be performed by other methods.

The volume estimation unit 43 estimates the volume of each of the multiple waste items W identified by the identification unit 41. Specifically, the volume estimation unit 43 acquires image data from the identification unit 41 and, based on the depth information of each pixel contained within the contour of the waste item W, acquires the area of the region corresponding to each pixel and the distance (height) to the floor F. The volume estimation unit 43 calculates the volume by multiplying the acquired area for the region corresponding to each pixel by the height, and the sum of the calculated volumes is taken as the volume of the waste item W.

The summing unit 44 calculates the total volume estimated by the volume estimation unit 43 for each type of waste W identified by the identification unit 41, based on the type estimated by the type estimation unit 42.

The output unit 45 outputs the total volume of each type of waste W, calculated by the summing unit 44, to the tablet terminal 50.

Next, an example of the processing (volume estimation processing) according to the present invention, which is performed in the computer of the control device 40 of the volume estimation system 1 of the above-described embodiment, will be explained with reference to the flowchart shown in Figure 4.

Multiple waste materials W contained in the mixed waste are placed in the imaging area S on the floor F so that they do not overlap. Then, the worker inputs an operation into the tablet terminal 50 requesting the control device 40 to perform volume estimation processing. The control device 40 performs volume estimation processing in response to the request from the tablet terminal 50.

The control device 40 controls the camera 20 to capture images of the shooting area S (S110).

Next, the control device 40 analyzes the image data output from the camera 20 to identify multiple waste materials W contained in the image data, and then determines the location and shape of each of the identified waste materials W (S120).

Specifically, when image data of multiple waste materials W (W11 to W34) as illustrated in Figure 5 is output from the camera 20, the control device 40 analyzes the image data to identify the waste materials W11 to W34. As shown in Figure 6, the control device 40 assigns identification numbers to the waste materials W11 to W34 and stores them in the storage device 60. Then, the control device 40 analyzes the image data to determine the location and shape of the waste materials W11 to W34.

Next, the control device 40 estimates the type of each of the identified waste materials W based on the image data (S130).

Specifically, the control device 40 estimates the type of waste by performing pattern matching on each of the waste materials W11 to W34 included in the still image data. Then, as shown in Figure 6, the control device 40 stores the estimated type of each waste material W11 to W34 in the storage device 60, associating it with the identification number of the waste material W11 to W34.

Next, the control device 40 estimates the volume of each of the identified waste materials W based on image data (S140).

Specifically, the control device 40 extracts pixels located inside the contours of each of the waste materials W11 to W34 included in the depth image data, and calculates the volume using the depth information of each extracted pixel for each of the waste materials W11 to W34. Then, as shown in Figure 6, the control device 40 stores the estimated volume for each of the waste materials W11 to W34 in the storage device 60, associating it with the identification number of the waste material W11 to W34.

Next, the control device 40 calculates the total volume for each type of the identified waste material W (S150).

Specifically, the control device 40 calculates the total volume of waste materials W11-W14, W16, W21, W22, W25, W28, W30, W31, W33, and W34 that are estimated to be "wood waste" stored in the storage device 60. It also calculates the total volume of waste materials W15, W17, W18, W20, W24, W27, and W29 that are estimated to be "metal scrap," and the total volume of waste materials W19, W23, W26, and W32 that are estimated to be "waste plastics." Then, as shown in Figure 7, the control device 40 stores the total volume calculated for each type in the storage device 60.

Next, the control device 40 outputs the total volume of each type of waste W to the tablet terminal 50 (S160). The tablet terminal 50 then displays the types of waste W included in the mixed waste and their total volumes. The worker refers to the display on the tablet terminal 50 to confirm whether the types of waste W and their total volumes match the manifest.

As described above, the volume estimation system 1 includes a camera 20 that captures images of the floor F's shooting area S, and a control device 40. The control device 40 includes an identification unit 41, a type estimation unit 42, a volume estimation unit 43, a totalization unit 44, and an output unit 45. The identification unit 41 identifies multiple waste materials W based on image data of multiple waste materials W captured by the camera 20. The type estimation unit 42 estimates the type of each of the multiple waste materials W based on the image data. The volume estimation unit 43 estimates the volume of each of the multiple waste materials W based on the image data. The totalization unit 44 calculates the total volume for each type of waste material W. The output unit 45 then outputs the calculated total value to the tablet terminal 50. The tablet terminal 50 displays the total volume for each type of waste material W.

As a result, the control device 40 can estimate the type and volume of waste W through image recognition without visual inspection by an operator, and calculate the total volume for each type. Therefore, the control device 40 can stably estimate the types and volumes of multiple types of waste W with high accuracy.

Furthermore, camera 20 includes a depth camera positioned above the floor F on which multiple waste materials W are placed. The image data of the multiple waste materials W captured by camera 20 includes depth image data, which contains depth information for each pixel, captured by the depth camera. The volume estimation unit 43 estimates the volume of each of the multiple waste materials W based on the depth image data. In this way, the volume of each of the multiple waste materials W can be estimated with higher accuracy.

Furthermore, in the volume estimation system 1, the camera 20 may include a mobile camera that can move above the floor F, and the volume estimation unit 43 may estimate the volume of each of the multiple waste materials W based on image data captured by the mobile camera at a first position and image data captured by the mobile camera at a second position different from the first position. The second position is, for example, a position moved horizontally from the first position. In this way, the control device 40 can estimate the volume of each of the multiple waste materials W based on parallax. By using multiple image data captured by the mobile camera at two or more different positions, the control device 40 can estimate the volume of each of the multiple waste materials W with higher accuracy.

Furthermore, in the volume estimation system 1, the camera 20 may include a mobile camera that can move above the floor F, and the type estimation unit 42 may estimate the type of each of the multiple waste materials W based on image data captured by the mobile camera. For example, the control device 40 may move the mobile camera upward when the multiple waste materials W are placed on the floor F, and move the mobile camera downward to approach the multiple waste materials W and capture the images. Alternatively, the control device 40 may move the mobile camera laterally. This improves work efficiency and allows for more accurate estimation of the type of each of the multiple waste materials W.

The mobile camera may be a camera mounted on a drone capable of flying over floor F. This eliminates the need to install a crane or other equipment to move the camera, allowing for a simpler camera movement configuration.

Furthermore, in the volume estimation system 1, when inputting an operation to request the execution of volume estimation processing to the tablet terminal 50, it may be possible to input the types of waste W that are expected to be included in the multiple waste W (for example, wood scraps, metal scraps, and waste plastics; hereinafter referred to as "expected types"). Alternatively, the expected types may be stored in the storage device 60. The output unit 45 may then output notification information to the tablet terminal 50 indicating that the multiple waste W whose types have been estimated by the type estimation unit 42 include waste W of a type other than the expected types. In this way, for example, by pre-specifying the types of waste W that are expected to be included in the multiple waste W based on the manifest, the tablet terminal 50 can notify that the multiple waste W contain waste W of an unexpected type (unspecified type).

Furthermore, in the volume estimation system 1, the storage device 60 may store the types of waste W that require attention (for example, polyvinyl chloride, hereinafter referred to as "designated types"). The output unit 45 may output notification information to the tablet terminal 50 indicating that the designated type of waste W is included in the multiple waste W whose types have been estimated by the type estimation unit 42. By doing so, by pre-specifying the types of waste W that require attention, the tablet terminal 50 can notify that the designated type of waste W is mixed in with the multiple waste W.

Furthermore, in the volume estimation system 1, the storage device 60 may store the upper limit of the volume of waste W that can be processed in a subsequent process. The output unit 45 may also output notification information to the tablet terminal 50 indicating that, when the multiple waste W whose volumes have been estimated by the volume estimation unit 43 contain waste W exceeding the upper limit, the multiple waste W contain waste W exceeding the upper limit. By doing so, by pre-setting the upper limit of the volume of waste W that can be processed in a subsequent process, the tablet terminal 50 can notify that the multiple waste W contain waste W that cannot be processed in a subsequent process.

The volume estimation system 1 described above estimates the types and volumes of multiple waste materials W contained in mixed waste, but it is not limited to this. The volume estimation system 1 may also estimate the types and volumes of multiple waste materials W that are expected to contain only one type of waste material W.

Although embodiments of the present invention have been described above, the present invention is not limited to the configurations of these embodiments. Additions, deletions, design modifications, and combinations of features of the embodiments, as appropriate by those skilled in the art, are also included within the scope of the present invention, as long as they do not contradict the spirit of the invention.

1...Volume estimation system 20...Camera 40...Control device 41...Identification unit 42...Type estimation unit 43...Volume estimation unit 44...Totaling unit 45...Output unit 50...Tablet terminal 60...Storage device

Claims (9)

A volume estimation device for estimating the volume of multiple waste materials placed on the floor, which include at least one of wood, metal, plastic, glass, concrete, ceramics, and rubble,
An identification unit that distinguishes each of the multiple wastes as a separate entity based on image data of the multiple wastes captured by the imaging device,
A type estimation unit that estimates the type of each of the aforementioned plurality of wastes based on the image data,
A volume estimation unit that estimates the volume of each of the aforementioned plurality of wastes based on the image data,
A summing unit that calculates the total volume for each type of waste mentioned above,
A volume estimation device characterized by having an output unit that outputs the total value so that the total value is displayed on a display device for each type . The aforementioned imaging device includes a depth camera positioned above the floor on which the plurality of wastes are placed,
The image data includes depth image data captured by the depth camera, which includes depth information for each pixel.
The volume estimation device according to claim 1, wherein the volume estimation unit estimates the volume of each of the plurality of wastes based on the depth image data. The aforementioned imaging device includes a mobile camera that can move above the floor on which the plurality of wastes are placed,
The volume estimation device according to claim 1, wherein the volume estimation unit estimates the volume of each of the plurality of wastes based on the image data captured by the mobile camera at a first position and the image data captured by the mobile camera at a second position different from the first position. The aforementioned imaging device includes a mobile camera that can move above the floor on which the plurality of wastes are placed,
The volume estimation device according to any one of claims 1 to 3, wherein the type estimation unit estimates the type of each of the plurality of wastes based on the image data captured by the mobile camera. The volume estimation device according to claim 3 or 4, wherein the mobile camera is a camera mounted on a drone capable of flying above the floor . The volume estimation device according to any one of claims 1 to 5, wherein the output unit outputs notification information indicating that the plurality of wastes whose types have been estimated include waste of a type other than those predetermined (hereinafter referred to as "unspecified type"). The volume estimation device according to any one of claims 1 to 6, wherein the output unit outputs notification information indicating that the plurality of wastes include waste of a predetermined type (hereinafter referred to as "designated type") when the plurality of wastes whose types have been estimated include waste of the designated type. The volume estimation device according to any one of claims 1 to 7, wherein the output unit outputs notification information indicating that the plurality of wastes whose volumes have been estimated contain waste exceeding a predetermined volume limit. A volume estimation method for estimating the volume of multiple wastes placed on the floor, which include at least one of wood, metal, plastic, glass, concrete, ceramics, and rubble,
Computers
Based on the image data of the multiple wastes captured by the imaging device , each of the multiple wastes is distinguished as a separate entity .
For each of the aforementioned multiple wastes, the type is estimated based on the image data.
For each of the aforementioned plurality of waste materials, the volume is estimated based on the image data.
For each of the aforementioned types of waste, the total volume is calculated.
A volume estimation method characterized by outputting the total value so that the total value is displayed on a display device for each type .