JP7708017B2 - Image Processing Device - Google Patents

Description

The present invention relates to an image processing device that synthesizes multiple images.

To improve logistics efficiency, there is a need to automatically calculate the loading rate of luggage loaded on the bed of a truck or other vehicle. The loading rate can be predicted from a database, but errors occur due to gaps during loading. In addition, since there are many luggage, it is difficult to manage them using RFID. Therefore, it is desirable to automatically calculate the loading rate from captured images, but when the distance between the photographer and the bed is short, it is not possible to capture an image of the entire bed, so it is necessary to combine and synthesize the captured images.

Regarding the above image synthesis technology, Patent Document 1 discloses a technology that detects overlap positions (corresponding points) between the images to be synthesized, finds the points in one image where the edge is stronger than a predetermined value and most similar to the other image, and synthesizes them.

Japanese Patent Application Publication No. 9-91407

When trying to use the technology of Patent Document 1 to calculate the loading rate of luggage loaded on the bed of a truck, for example, if multiple luggage of the same type or similar dimensions are loaded on the bed, there will be multiple similarities between the images, making it impossible to accurately detect matching points between the images and making it impossible to synthesize the images.

The present invention has been made to solve the above problems, and aims to provide an image processing device that can accurately synthesize images even when multiple images obtained by capturing images of multiple packages contain multiple packages of the same type or similar dimensions.

To solve the above problems, the image processing device of the present invention has the following configuration. The effects achieved by each configuration will also be described.

The image processing device according to the present invention is an image processing device that synthesizes multiple images obtained by capturing images of multiple pieces of luggage, and includes an image acquisition unit that acquires multiple images, a luggage detection unit that detects the type and position coordinates of each of the multiple pieces of luggage from the multiple images, a luggage dimension calculation unit that calculates the dimensions of the luggage from the position coordinates of the luggage, a luggage determination unit that determines whether or not the multiple images contain luggage that is common to both images based on the detected luggage type and dimensions, and an image synthesis unit that synthesizes the multiple images by superimposing parts of the luggage that are common to both images when the luggage determination unit determines that the multiple images contain luggage that is common to both images.

In accordance with the present invention, instead of detecting overlapping positions between images as in conventional technology, the present invention determines whether or not there is a common piece of luggage between images by recognizing the degree of overlap between images based on the type of luggage and its position coordinates, making it possible to accurately synthesize images even when multiple images contain multiple pieces of luggage of the same type or similar dimensions.

In a more preferred image processing device, when the luggage determination unit determines that a common piece of luggage is included among the multiple images, it further determines whether other luggage adjacent to the common piece of luggage is common among the multiple images.

This configuration makes it possible to confirm that there are multiple pieces of luggage that are common between the images, and allows for a more accurate calculation of the commonality between the images.

In a more preferred image processing device, the luggage detection unit detects the type and position coordinates for each row of luggage in which multiple pieces of luggage are stacked vertically, and the luggage determination unit determines for each row of luggage whether the luggage is common to the multiple images, and if it determines that a piece of luggage is common to the multiple images, it further determines whether other luggage adjacent to the top or bottom of the common luggage is common to the multiple images.

Luggage loaded onto the bed of a truck or other motor vehicle is often stacked vertically in rows aligned across the width of the bed, and by adopting the above configuration, luggage determination is performed according to certain rules, providing convenience to users of the image processing device.

In a more preferred image processing device, the luggage determination unit determines that luggage whose type matches between multiple images and whose dimensional differences are within a predetermined threshold are the same luggage.

With this configuration, when obtaining multiple images, even if the distance or angle between the imaging device and the subject differs slightly between the images, it is possible to correct the difference and accurately synthesize the images.

In a more preferred image processing device, the luggage detection unit uses images of multiple pieces of luggage as training data and machine-learns a method for extracting the luggage contained in the images and a method for identifying the type of luggage, and the luggage detection unit detects the type and position coordinates of each of the multiple pieces of luggage from the multiple images acquired by the image acquisition unit.

This configuration makes it possible to detect luggage by utilizing an AI model that has been trained on a large number of packages in advance using, for example, a neural network, and is expected to significantly improve detection accuracy.

In a more preferred image processing device, the multiple pieces of luggage are multiple pieces of luggage loaded on the bed of a motor vehicle, and the multiple images are multiple images of the bed divided in the front-rear direction of the motor vehicle.

Trucks used in logistics are often loaded with a large number of items of the same type and similar dimensions on their loading platforms, and therefore the present invention can be suitably applied to such configurations.

According to the present invention, accurate image synthesis is possible even when there are multiple pieces of luggage of the same type or similar dimensions among the multiple images obtained by capturing multiple pieces of luggage.

1 is a block diagram showing a hardware configuration of an image processing apparatus according to an embodiment of the present invention; FIG. 2 is a block diagram showing a functional configuration of the image processing device. FIG. 13 is a diagram showing a process of detecting the type and position coordinates of each piece of luggage from a plurality of images obtained by an image processing device. 11 is a table showing detected package information sorted based on location coordinates. 13A and 13B are diagrams illustrating a process of determining whether or not a common package is included in a plurality of images, and a process of performing image synthesis. 4 is a flowchart showing a process executed by the image processing device.

The following describes an embodiment of the present invention with reference to the drawings. In this embodiment, the bed of a truck (motor vehicle) carrying multiple packages is the subject of imaging, and the multiple images obtained as a result of imaging are images of the bed of the truck divided in the front-rear direction of the truck.

Figure 1 is a block diagram showing the hardware configuration of an image processing device 1 according to one embodiment of the present invention. The image processing device 1 has a ROM (Read Only Memory) 11, which is a storage device capable of reading data, a RAM (Random Access Memory) 12, which is capable of both writing and reading data, a CPU (Central Processing Unit) 13, which calculates parameters required for vehicle control based on the data, and a communication module 14, which is an interface for sending and receiving data.

The image processing device 1 is connected to an imaging device 3 and an external server 4 via a communication path 2. The communication path 2 may be configured from a wireless network such as Wi-Fi (registered trademark). The image processing device 1 receives a plurality of images obtained by capturing an image of an object from the imaging device 3. In addition, the image processing device 1 transmits and receives various information including calculation results and training data for machine learning, which will be described later, between the image processing device 1 and the external server 4. The external server 4 may be installed in a control center as a computer with storage, or may be implemented as a virtual machine on the cloud.

The image processing device 1 receives data via the communication path 2. In response to the reception, the received data is expanded in the RAM 12. The CPU 13 then performs calculations based on the data written in the RAM 12 and the data read from the ROM 11. A program is also pre-installed in the image processing device 1, and the CPU 13 executes the program to perform the processing described below. Below, the details of each part will be described, omitting such well-known techniques in the computer field as appropriate.

Next, the functional configuration of the image processing device 1 will be described with reference to the functional block diagram of FIG. 2. As shown in FIG. 2, the image processing device 1 includes an image acquisition unit 15, a baggage detection unit 16, a baggage size calculation unit 17, a baggage determination unit 18, an image synthesis unit 19, and an output unit 20. In addition, a baggage information database (DB) 21 is stored in the memory area of the image processing device 1.

The functional block diagram shown in FIG. 2 is an example, and the functional units and names are not limited to this. For example, the image acquisition unit 15 and the output unit 20 may be collectively treated as a data transmission/reception unit.

The image acquisition unit 15 acquires multiple images captured by the imaging device 3 via the communication path 2. The luggage detection unit 16 analyzes the multiple images acquired by the image acquisition unit 15 for each image and detects the type and position coordinates of the luggage contained in the image. The luggage dimension calculation unit 17 calculates the dimensions of each luggage from the position coordinates of each luggage detected by the luggage detection unit 16. The luggage determination unit 18 determines whether or not a common luggage is included among the multiple images based on the type of each luggage detected by the luggage detection unit 16 and the dimensions of each luggage calculated by the luggage dimension calculation unit 17. The image synthesis unit 19 synthesizes multiple images that are determined by the luggage determination unit 18 to include a common luggage. The output unit 20 outputs the synthesized image data, luggage information included in the images, results calculated by each functional unit, etc. to the external server 4.

The luggage information DB 21 stores various data related to luggage obtained by machine learning in advance using images of multiple luggage as training data to detect the dimensions of the luggage contained in the images and the type of luggage. In this embodiment, the luggage detection unit 16 configures an AI model (learning model) as described below. The AI model is a model that uses image data of luggage that can be the subject of an image as training data to machine learn a method for extracting luggage contained in an image (recognizing the luggage and detecting the coordinates of the luggage on the image) and a process for identifying the type of the extracted luggage. Specifically, the training data is used to machine learn the features of the luggage so that features such as the dimensions or shape of each luggage can be extracted. This makes it possible to detect luggage contained in an image with high accuracy.

Next, the process of detecting the type and location coordinates of each piece of luggage from multiple images obtained by the image processing device will be described with reference to FIG. 3. Note that in this embodiment, luggage is common to two images, and the process of synthesizing the two images to generate a single image will be described, but it goes without saying that the number of images to be synthesized and the number of images to be generated are not limited to this embodiment.

The image acquisition unit 15 acquires the first image shown in Fig. 3(a) and the second image shown in Fig. 3(b) from the imaging device 3. As described above, the image acquired in this embodiment is a part of the bed of a truck on which a plurality of luggage items are loaded, and the first image shown in Fig. 3(a) is an image obtained by imaging a state in which a row of luggage items (L ₁ , L ₂ , L ₃ . . . ) consisting of a plurality of luggage items 100a stacked vertically is loaded from the front of the bed to the rear, and the second image shown in Fig. 3(b) is an image obtained by imaging a state in which a row of luggage items (L _k , L _k+1 , L _k+2 . . . ) of an arbitrary range is loaded from the middle of the bed to the rear. Therefore, the second image shown in Fig. 3(b) is an image obtained by imaging a part of the bed that is further rearward than the first image shown in Fig. 3(a), and therefore, when the first image and the second image are combined, a part including the right end of the first image and a part including the left end of the second image are superimposed as a common part and combined.

In this embodiment, the cargo 100a is defined as consisting of a skid portion 102a for inserting the forks of a forklift to transport it, and a load portion 101a that is placed on the skid portion 102a. However, it goes without saying that there are an infinite number of objects that can be defined as cargo, not limited to cargo that can be loaded onto the bed of a truck, and the load portion 101a and the skid portion 102a may be distinguished and each defined as cargo. In this way, the definition of cargo within the scope of application of this invention can be determined arbitrarily.

The image acquisition unit 15 transmits the acquired image data to the luggage detection unit 16. As described above, the luggage detection unit 16 constitutes an AI model for luggage detection, and receives image data from the image acquisition unit 15 as input, and outputs the number, type, and position coordinates of luggage contained in the image. Specifically, the learned AI model is used to extract the characteristics of objects contained in the image on a pixel-by-pixel basis, and automatically processes color, brightness, distortion, noise, range distinctions, etc. to visualize and classify what is in the image, perform an "image classification process" that automatically recognizes the subject and detects the range in which the object exists in the image, an "image detection process" that automatically recognizes the subject and detects the range in which the object exists in the image, and an "image segmentation process" that clearly distinguishes the boundary line of the subject to identify what is within the image area.

For example, taking the skid section 102a in FIG. 3(a) as an example, an AI model (learning model) is constructed in the luggage detection unit 16, which is trained in the extraction process of luggage contained in an image using features such as the "ratio of vertical and horizontal dimensions," "position of holes for fork insertion and area ratio to the whole," "brightness, color, surface unevenness, transparency, etc. of the constituent materials" as training data of the luggage image. When an image to be detected for luggage is input to the constructed AI model, the AI model extracts the features of the luggage from the image, and based on the extracted features, it can detect that the object marked with the symbol 102a is a skid section. This allows the luggage detection unit 16 (AI model) to detect the position coordinates of the luggage 100a in the image by detecting the skid section included in the image, and to identify the type of luggage 100a. The same applies to the load section 101a placed on the skid section 102a, and of course objects other than these can also be detected with high accuracy by providing training data in advance and training the model.

The luggage detection unit 16 detects the type of each piece of luggage as a result of the image analysis described above. Then, as shown in FIG. 3(b), a luggage segmentation that distinguishes each piece of luggage is assigned to each piece of luggage (luggage segmentation 200a for luggage 100a, luggage segmentation 200b for luggage 100b). During the above series of processes, the image is subjected to pixel analysis, and the position coordinates for each piece of luggage segmentation are also calculated. The origin of the position coordinates is, for example, the top left point of each image.

By the above process, the luggage detection unit 16 detects the type and location coordinates of each piece of luggage and transmits the detection results to the luggage dimension calculation unit 17. The luggage dimension calculation unit 17 performs the following process based on the received detection results.

Figure 4(a) is a list of luggage information detected by the luggage detection unit 16, including the luggage ID of the detected luggage, the type of luggage (label), the detection accuracy (accuracy), and the coordinates of the four sides (left, top, right, bottom). In Figure 4(a), the data is listed according to the ID that the luggage detection unit 16 assigns when it recognizes each piece of luggage, for example, and it is difficult to handle the data as is. Therefore, the luggage dimension calculation unit 17 first organizes the data in Figure 4(a) to make it easier to handle.

Specifically, for example, the coordinates of the left side of each piece of luggage are sorted. Then, as shown in FIG. 4(b), luggage with the same coordinates on the left side are displayed from the top of the chart. From this example, it can be seen that the luggage with id:21, 19, 3, and 1 are in the row of luggage at the left end of the captured image, and that the skid with id:1 is placed directly on the bottom surface of the loading platform, on top of which is the poli with id:19, on top of which is the skid with id:3, on top of which is the poli with id:21 (note that the coordinate origin is the top left of the image).

In this way, the luggage dimension calculation unit 17 sorts the luggage information and organizes it so that it can be handled as data for each luggage row. Then, for each luggage row, the dimensions (height and width) of each luggage are calculated by finding the difference in position coordinates. The luggage dimension information calculated in this way is further added to the luggage information in Figure 4 (b) and transmitted to the luggage determination unit 18.

In addition, referring to the series of data with the left side coordinate "911" in the table of FIG. 4(b), the right side coordinate of five of the six detected pieces of luggage is "1024", and the remaining one is also "1021". The luggage width calculated from this coordinate difference is also extremely small compared to the other pieces of luggage, at "113" and "110". In this way, when the calculated luggage width is extremely small compared to the width of the other pieces of luggage, the luggage row can be excluded from the judgment by the luggage judgment unit 18 as it is cut off from the image. The exclusion criteria in this case may be preset, for example, to 50% or less of the average width of the detected luggage, or may be changed in real time depending on the situation.

Next, the process in which the luggage determination unit 18, which has received luggage information from the luggage dimension calculation unit 17, determines whether or not a common piece of luggage is included among the multiple images, and the process in which the image synthesis unit 19 synthesizes the images when the luggage determination unit 18 determines that a common piece of luggage is included among the multiple images, will be described with reference to FIG. 5.

First, the luggage determination unit 18 recognizes from the luggage information received from the luggage dimension calculation unit 17 that the first image shown in Fig. 5(a) is an image obtained by capturing an image of a portion of the platform to the left of the second image shown in Fig. 5(b), and arranges the first image on the left and the second image on the right. In addition, as a result of processing by the luggage dimension calculation unit 17, the luggage sequence to be determined in the first image is set to _L1 , _L2 , and _L3 .

Here, the baggage determination unit 18 in this embodiment is set to recognize all the bags included in each baggage row as a baggage row segmentation (the baggage included in baggage row _L3 is recognized as baggage row segmentation 300a). Then, the baggage determination unit 18 executes a process of determining whether or not there is a baggage common to the images for each baggage row segmentation.

First, the luggage determination unit 18 extracts from the luggage information the type and dimensions of the luggage 110a-140a included in the rightmost luggage row segmentation 300a of the luggage row to be determined in the first image. Then, it starts to determine whether the luggage 110a placed on the topmost level of this luggage row segmentation 300a is included in the second image. Hereinafter, the type and dimensions of the luggage 110a are referred to as reference luggage information.

Based on the above reference luggage information, the type and dimensions of the luggage 100b placed on the topmost level of the leftmost luggage row segmentation in the luggage row to be judged in the second image are detected. If either the type or dimensions of the detected luggage do not match the reference luggage information, the type and dimensions of the luggage placed one level below the judged luggage 100b are detected and a similar judgment is made. If the luggage placed on the bottom level of the luggage row segmentation has been judged, the judgment is continued by moving to the luggage placed on the topmost level of the luggage row segmentation one row to the right of the luggage row segmentation. Note that when comparing the luggage dimensions between images, each image may be calibrated to correct deviations due to differences in the distance and optical axis angle between the imaging device and the imaging target area when each image is captured. A certain threshold value may also be set at which it can be determined that the dimensions match.

In this embodiment, as a result of repeating the above process, the luggage determination unit 18 determines that the type and dimensions of the uppermost luggage 110b in the second leftmost luggage row segmentation 300b in the luggage row to be determined in the second image match the reference luggage information. In other words, it is determined that the leftmost luggage row in the second image is the same as the luggage row _L2 in the first image.

The luggage determination unit 18 then refers to the first image again, this time extracting luggage information for luggage 120a that is placed second from the top of luggage row segmentation 300a, and determining whether this matches the type and dimensions of luggage 120b that is placed second from the top of luggage row segmentation 300b in the second image.

Through the above processing, the luggage determination unit 18 determines that two vertically adjacent luggage (luggages 110a and 120a in the first image and luggage 110b and 120 in the second image) exist in a common luggage row segmentation (300a in the first image and 300b in the second image) between the first image and the second image, and transmits the determination result to the image synthesis unit 19.

When the image synthesis unit 19 receives the judgment result from the luggage judgment unit 18, it superimposes the two images so that the luggage row segmentation 300a of the first image and the luggage row segmentation 300b of the second image overlap, generating a composite image as shown in FIG. 5(c).

In the above, when two adjacent pieces of luggage included in the luggage row segmentation match, the luggage row segmentation is determined to match. However, the number of pieces of luggage to be compared is not limited to this, and the above determination process may be performed similarly for the pieces of luggage 130a, 140a in the first image and the pieces of luggage 130b, 140b in the second image. The more pieces of luggage to be compared, the more accurate the image determination becomes.

In the above explanation, in order to simplify the explanation, it is assumed that the first image is a part including the front of the loading platform, and the second image is further back on the loading platform than the first image. However, the luggage determination unit 18 cannot necessarily grasp the positional relationship between the received image data, and therefore the positional relationship may be the opposite of that between the first image and the second image described above. Therefore, if the result of performing the determination process on all luggage included in the luggage row segmentation with the first image located on the left is that it is not determined that the luggage included in the second image matches the luggage included in the luggage row segmentation located on the left side, the possibility that the second image is an image of a part of the loading platform further forward than the first image is taken into consideration, and the luggage included in the luggage row segmentation one row to the left in the first image is newly set as the determination target, and the determination process continues.

Finally, the output unit 20 outputs the judgment result to the luggage information DB 21 and/or the external server 4, and updates the teacher data, etc.

The series of determination processes described above will be explained using the flowchart in FIG. 6. The process starts when the image acquisition unit 15 acquires multiple images from the imaging device 3.

In step S601, the luggage detection unit 16 detects the type and location coordinates of luggage contained in each of the multiple images acquired from the image acquisition unit 15 using the AI model described above.

In step S602, the luggage detection unit 16 calculates the average width of each type of luggage detected.

In step S603, the luggage dimension calculation unit 17 sorts the luggage information as described with reference to FIG. 4 and organizes the information for each luggage row. At this time, the luggage row to be excluded from the judgment target by the luggage judgment unit 18 may be selected using the average width for each luggage type calculated in step S602.

In step S604, the luggage determination unit 18 extracts luggage in the luggage row segmentation at the right end in the first image (the image capturing the most forward part of the loading platform among the multiple images acquired) and calculates luggage information, as described with reference to FIG. 5.

In steps S605-S606, the luggage determination unit 18 detects luggage information of the luggage to be determined in the second image (an image captured from the rear of the loading platform relative to the first image), and determines, based on the luggage information received from the luggage dimension calculation unit 17, whether or not there is luggage in the second image that is common to the luggage in the luggage row segmentation extracted in step S604.

If it is determined in step S606 that there is no luggage in common between the first image and the second image, the luggage information in the luggage row segmentation one row to the left in the first image is calculated, and similarly, the determination of whether there is any luggage in common in the second image is continued.

If it is determined in step S606 that there is a piece of luggage in common between the first image and the second image, the process proceeds to step S607. In step S607, the luggage determination unit 18 further determines whether there is a piece of luggage in common between the first image and the second image that is one level below the piece of luggage determined to be in common in step S606.

If it is determined in step S607 that the images of the luggage one level below match, then in step S608 the images are combined by the image combination unit 19 which receives the result. If it is determined that the images of the luggage one level below do not match, then it is determined that the luggage determined to be common in step S606 happened to exist in both images, and the processing by the luggage determination unit 18 proceeds to step S609. The process thereafter is as described above.

Once image synthesis is performed by the image synthesis unit 19, the results of the process are output by the output unit 20 to the luggage information DB 21 and/or the external server 4, and the process ends.

As described above, according to the present invention, the determination of whether or not there is luggage common between images is performed based on the type of luggage and dimensions based on the position coordinates, making it possible to improve the accuracy of the determination even when there are many similar luggage. Moreover, when it is determined that there is luggage common between images, it is possible to further improve the accuracy of the determination by determining whether there is commonality in luggage adjacent to that luggage. Furthermore, when detecting the type and dimensions of luggage, the luggage detection unit can be an AI model generated by machine learning the type and dimensions of luggage using images obtained by previously capturing multiple luggage as training data, making it possible to greatly improve the accuracy of luggage detection.

Although the embodiments of the present invention have been described in detail above, the present invention is not limited to the above embodiments, and various modifications can be made without departing from the spirit of the present invention described in the claims. The present invention allows the configuration of one embodiment to be added to the configuration of another embodiment, the configuration of one embodiment to be replaced with another embodiment, or part of the configuration of one embodiment to be deleted.

1 Image processing device, 15 Image acquisition unit, 16 Baggage detection unit, 17 Baggage size calculation unit, 18 Baggage determination unit, 19 Image synthesis unit

Claims (5)

An image processing device that captures images of a plurality of packages and synthesizes the images,
an image acquisition unit for acquiring the plurality of images;
a baggage detection unit that detects the type and location coordinates of each of the plurality of bags from the plurality of images;
a luggage dimension calculation unit that calculates the dimensions of the luggage from the position coordinates of the luggage;
a baggage determination unit that determines whether or not a common baggage is included among the plurality of images based on the detected baggage type and size;
an image synthesis unit that synthesizes the plurality of images by superimposing a portion of the common package when the package determination unit determines that the plurality of images contain a common package;
Equipped with
When it is determined that a common package is included among the plurality of images, the package determination unit further determines whether or not another package adjacent to the common package is common among the plurality of images.
13. An image processing device comprising: 2. The image processing device according to claim 1 ,
the luggage detection unit detects the type and position coordinates for each row of luggage in which the luggage is stacked in a vertical direction;
the luggage determination unit determines, for each of the luggage rows, whether or not the luggage is a common luggage among the plurality of images, and when it is determined that a common luggage is included among the plurality of images, it further determines whether or not another luggage adjacent to the top or bottom of the common luggage is common among the plurality of images.
13. An image processing device comprising: 2. The image processing device according to claim 1,
the baggage determination unit determines, among the plurality of images, baggage whose type matches and whose dimensional difference is within a predetermined threshold value as a common baggage;
13. An image processing device comprising: 2. The image processing device according to claim 1,
the luggage detection unit uses images of the plurality of luggage as training data and performs machine learning to learn a method for extracting luggage included in the images and a method for identifying the type of the luggage;
The baggage detection unit detects the type and position coordinates of each of the plurality of bags of luggage from the plurality of images acquired by the image acquisition unit.
13. An image processing device comprising: 2. The image processing device according to claim 1,
The plurality of packages are a plurality of packages loaded on a loading platform of a motor vehicle,
2. An image processing device according to claim 1, wherein the plurality of images are images obtained by dividing the loading platform in the front-rear direction of the motor vehicle.