JP7758331B2 - Image processing device, image processing method and program - Google Patents

Description

This disclosure relates to an image processing device, an image processing method, and a program.

At production sites, products are transported on a conveyor belt. Various systems have been proposed that track products on a conveyor belt and mark them with a projector or perform tasks on them with a robot. For example, Patent Document 1 describes a robot system that detects the position of an object being transported by a conveyor and has a robot perform tasks on the object based on the detected position. The robot system disclosed in Patent Document 1 captures an image of an item being transported by a conveyor and detects the position of the item through image processing.

JP 2017-47511 A

Using the technology disclosed in Patent Document 1, the position of a target object can be detected and a robot can perform work on that object. However, when detecting the object's position using image processing, there is a risk that part of the robot will appear in the image. This appearance can become noise, which can reduce the accuracy of detecting the position of the target object using the image.

One example of the objective of the present disclosure is to provide an image processing device, an image processing method, and a program that can accurately estimate the position of an object based on image data.

In order to achieve the above object, an image processing device according to one aspect of the present disclosure includes:
a first position estimation unit that estimates a position of the object in a first area based on image data of the first area set upstream of a path along which the object moves;
a speed estimation unit that estimates a speed of the object on the path;
a second position estimation unit that, when the object satisfies a set condition, estimates a position of the object in a second area set downstream of the route at a predetermined time based on the detected position of the object and the estimated speed of the object; and
The present invention is characterized in that it is provided with:

In order to achieve the above object, an image processing method according to one aspect of the present disclosure includes:
estimating a position of the object in a first area based on image data of the first area set upstream of a path along which the object moves;
estimating the velocity of the object along the path;
If the object satisfies a set condition, estimating a position of the object in a second area set downstream of the route at a predetermined time based on the detected position of the object and the estimated speed of the object;
The present invention is characterized in that it is provided with:

Furthermore, in order to achieve the above object, a program according to one aspect of the present disclosure includes:
On the computer,
estimating a position of the object in a first area based on image data of the first area set upstream of a path along which the object moves;
estimating the velocity of the object along the path;
If the object satisfies a set condition, estimating a position of the object in a second area set downstream of the route at a predetermined time based on the detected position of the object and the estimated speed of the object;
The present invention is characterized in that the following is executed.

As described above, the technology disclosed herein makes it possible to accurately estimate the position of an object based on image data.

FIG. 1 is a diagram illustrating a system in which an image processing apparatus according to an embodiment is used. FIG. 2 is a block diagram showing the configuration of the image processing apparatus according to the embodiment. FIG. 3 is a diagram illustrating a system for marking an object with a light-projecting marker, in which the image processing device according to the embodiment is used. FIG. 4 is a block diagram showing a specific configuration of the image processing device. FIG. 5 is a diagram for explaining the conversion from the coordinates in the imaging coordinate system to the coordinates in the three-dimensional coordinate system. FIG. 6 is a diagram for explaining the conversion from the coordinates of the three-dimensional coordinate system to the coordinates of the projection imaging coordinate system. FIG. 7 is a flowchart showing the operation of the image processing apparatus when performing calibration. FIG. 8 is a flowchart showing the operation of the image processing device during position estimation. FIG. 9 is a block diagram illustrating an example of a computer that realizes the image processing apparatus according to the embodiment.

First, we will provide an overview to facilitate understanding of the embodiments described below.

Figure 1 is a diagram illustrating a system that uses an image processing device 10 according to this embodiment. The image processing device 10 is used, for example, at a product production site to estimate the position of an object 101, such as a product being transported on a belt conveyor 100.

An imaging area 105 is set on the upstream side of the belt conveyor 100, and a processing area 106 is set on the downstream side. The imaging area 105 is the area imaged by the imaging device 21. The processing area 106 is the area where optical or physical processing is performed on the object 101.

When the object 101 is imaged in the imaging area 105, it is inspected based on the image data to determine whether it is a defective product. Objects 101 determined to be defective undergo optical or physical processing in the processing area 106. Optical processing is, for example, marking the object 101 with a light-projecting marker. Physical processing is, for example, removing the object 101 with a robot arm. The position of the object 101 in the processing area 106 is estimated by the image processing device 10.

The image processing device 10 is a device that detects the position of the object 101 in the imaging area 105 and estimates its position in the processing area 106 from that position. The image processing device 10 acquires image data of the imaging area 105 in which the object 101 is located, using the imaging device 21. The image processing device 10 estimates the position of the object 101 in the imaging area 105 and the movement speed of the object 101 from that image data. The image processing device 10 estimates the position of the object 101 in the processing area 106 at a given time from the estimated position and movement speed.

As such, in this embodiment, the imaging area 105 and the processing area 106 are set at different positions on the belt conveyor 100. If the imaging area 105 and the processing area 106 overlap, a light-projecting marker or robot arm may appear in the image data captured by the imaging area 105. If a light-projecting marker or the like appears in the image data, this may become noise and affect the position detection of the object 101, which is performed based on the image data, and the results of the inspection for defective products.

The image processing device 10 according to this embodiment therefore enables separation of the imaging area 105 and the processing area 106, preventing unnecessary objects from appearing when capturing an image of the imaging area 105. As a result, the image processing device 10 achieves highly accurate position detection of the object 101 and inspection of defective products.

[Device configuration]
FIG. 2 is a block diagram showing the configuration of the image processing apparatus 10 according to this embodiment.

The image processing device 10 includes a first position estimation unit 1, a speed estimation unit 2, and a second position estimation unit 3.

The first position estimation unit 1 estimates the position of the object 101 in a first area set upstream of the belt conveyor 100, which is the path along which the object 101 moves, from image data of the first area. The first area is the imaging area 105 described in Figure 1. The image data is data obtained by capturing an image using the imaging device 21.

The speed estimation unit 2 estimates the speed of the object 101 on the belt conveyor 100.

When the object 101 satisfies the set conditions, the second position estimation unit 3 estimates the position of the object 101 in a second area set downstream of the belt conveyor 100 at a predetermined time based on the detected position of the object 101 and the estimated speed of the object 101. The second area is the processing area 106 described in Figure 1. An example of a case in which the object 101 satisfies the set conditions is when the object 101 is inspected based on image data and determined to be a defective product. The inspection of the object 101 may be performed by the image processing device 10 or by a separate device.

The image processing device 10 configured as described above estimates the position of the object 101 on the downstream side of the belt conveyor 100 from the position of the object 101 on the upstream side and the speed of the object 101. As a result, by using the image processing device 10 of this embodiment at a product production site, it is possible to detect the position and inspect the object 101 on the upstream side of the belt conveyor 100, and to perform optical or physical processing on the object 101 on the downstream side of the belt conveyor 100.

In other words, as explained in FIG. 1, the image processing device 10 enables separation of the imaging area 105 and the processing area 106. As a result, no unnecessary objects are captured in the image data obtained by capturing an image of the imaging area 105. Therefore, the image processing device 10 enables highly accurate detection of the position of the object 101 and inspection of defective products using image data.

The specific contents of the image processing device 10 will be described in more detail below. In the following, the image processing device 10 will be described as being used in a system that inspects an object 101 on the upstream side of a belt conveyor 100 and, depending on the inspection results, marks the object 101 with a light-projecting marker on the downstream side. First, a system that marks an object 101 with a light-projecting marker will be described.

Figure 3 is a diagram illustrating a system for marking an object 101 with a light-projecting marker, in which the image processing device 10 according to this embodiment is used.

The belt conveyor 100 transports an object 101 in the direction of the arrow in the figure. Multiple markers 100A are provided on each side of the width of the belt conveyor 100. The multiple markers 100A face each other in the width direction and are arranged at equal intervals along the belt extension direction (conveyance direction). The positional relationship between each of the multiple markers 100A, for example, the distance between the markers, is known. Note that the multiple markers 100A are fixed and do not move even if the belt portion of the belt conveyor 100 moves, but they move when the belt conveyor 100 itself moves.

An imaging device 21 is provided upstream of the belt conveyor 100. A projection device 22 and an imaging device 23 are provided downstream of the belt conveyor 100. The imaging device 21, projection device 22, and imaging device 23 are connected to the image processing device 10.

The imaging device 21 captures an image of the upstream side of the belt conveyor 100. The area captured by the imaging device 21 is the imaging area 105 described in FIG. 1. The imaging area 105 is set to include the marker 100A. The imaging device 21 may be, for example, a camera, an infrared camera, an ultrasonic camera, or an X-ray camera.

The projection device 22 acquires the position of the object 101 downstream of the belt conveyor 100, estimated by the image processing device 10, and marks the object 101 at that position with a light-projecting marker 102. The projection area 107 onto which the projection device 22 projects the light-projecting marker 102 is the processing area 106 described in Figure 1. The projection device 22 may be, for example, a projector.

The imaging device 23 is configured to capture an imaging area 108 set downstream of the belt conveyor 100 so as to overlap with the projection area 107 of the projection device 22. The imaging device 23 captures the projection pattern projected by the projection device 22 when calibrating the projection device 22. The imaging device 23 is, for example, a camera, an infrared camera, an ultrasonic camera, an X-ray camera, etc.

The imaging area 108 is set to include the marker 100A. As described above, the positional relationship between each of the markers 100A is known. Therefore, the positional relationship between the imaging area 105, which is set to include the marker 100A, and the imaging area 108 is clear.

Figure 4 is a block diagram showing the specific configuration of the image processing device 10.

In addition to the first position estimation unit 1, velocity estimation unit 2, and second position estimation unit 3 described above, the image processing device 10 further includes a processing execution unit 4 and a calibration unit 5.

Before the image processing device 10 estimates the position of the object 101, the calibration unit 5 performs calibration to estimate the parameters (internal parameters and external parameters) of the image capture device 21 and the image capture device 23. Since the calibration methods for the image capture device 21 and the image capture device 23 are the same, only the calibration method for the image capture device 21 will be described below.

The internal parameters are parameters that indicate, for example, the lens focal length of the imaging device 21 and the lens distortion coefficient. The calibration unit 5 estimates the internal parameters using, for example, the method by Z. Zhang described in "A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330-1334, 2000." This method involves capturing images of a board with a printed grid pattern from multiple directions using the imaging device 21, and estimating parameters based on feature points in the images. The estimated parameters can be used during imaging.

The extrinsic parameters are parameters that indicate the position and angle (extrinsic parameters) of the imaging device 21 relative to the belt conveyor 100. These extrinsic parameters are parameters that convert a three-dimensional coordinate system (second coordinate system) set on the belt conveyor 100 into a coordinate system (first coordinate system) of image data obtained from the imaging device 21. More specifically, the calibration unit 5 calculates the extrinsic parameters by solving a PnP (Perspective-n-Point) problem using image data obtained by imaging the imaging area 105 with the imaging device 21. An algorithm for solving the PnP problem is described, for example, in "V. Lepetit, F. Moreno-Noguer, and P. Fua, "EPnP: An Accurate O(n) Solution to the PnP problem," International Journal of Computer Vision, vol. 81, no. 2, pp. 155-166, 2009."

The calibration unit 5 also performs calibration between the projection device 22 and the imaging device 23. The coordinate system of the projection area 107 of the projection device 22 (hereinafter referred to as the projection coordinate system) and the coordinate system of the imaging area 108 of the imaging device 23 (hereinafter referred to as the projection imaging coordinate system) are different coordinate systems. This is because the projection device 22 and the imaging device 23 cannot be installed in the same position. The calibration unit 5 estimates the geometric transformation between the projection imaging coordinate system and the projection coordinate system from image data obtained by capturing the projection pattern projected by the projection device 22 with the imaging device 23. The projection pattern is a checkerboard pattern in which multiple markers are arranged in a grid.

This transformation uses, for example, a perspective projection transformation. If the transformation parameter for transforming the projection imaging coordinate system into the projection coordinate system is H, the coordinates of the projection coordinate system are (x', y'), and the coordinates of the projection imaging coordinate system are (x, y), then (x', y') = H(x, y).

The calibration unit 5 determines the correspondence between the coordinates ( _xm , _ym ) of the corners of each marker in the projection pattern in the image data captured by the imaging device 23 and the coordinates ( _x'm , _y'm ) of the corners of each marker in the projection pattern projected by the projection device 22. Then, the calibration unit 5 estimates a transformation parameter H that minimizes the positional deviation between the coordinates of the two corners by the least squares method using the following equation (1).

Σ _i ||x' _m ⁱ −H(x _m ⁱ ) || …(1)
Here, _xm = ( _xm , _ym ), _x'm = ( _x'm , _y'm ), and i is the number of each marker in the projection pattern.

The first position estimation unit 1 estimates the position of the object 101 in the imaging area 105 based on image data acquired from the imaging device 21. More specifically, the first position estimation unit 1 estimates the coordinates of the object 101 in the coordinate system in the image data (hereinafter referred to as the imaging coordinate system). The first position estimation unit 1 then converts these coordinates into coordinates in a three-dimensional coordinate system set on the belt conveyor 100, and estimates the position of the object 101 on the belt conveyor 100. When estimating the position of the object 101 from the image data, the first position estimation unit 1 corrects lens distortion in the image data using a lens distortion coefficient, which is an internal parameter.

Figure 5 is a diagram illustrating the conversion from coordinates in the imaging coordinate system to coordinates in a three-dimensional coordinate system. In this example, the three-dimensional coordinate system has the belt extension direction as the X axis, the belt width direction as the Y axis, and the height direction as the Z axis.

The first position estimation unit 1 detects the coordinates of the object 101 in the image coordinate system. The first position estimation unit 1 converts the detected coordinates in the image coordinate system into coordinates in a three-dimensional coordinate system. Note that the coordinates of the object 101 in the converted three-dimensional coordinate system will exist on the X-Y plane where Z = 0.

The relationship between the coordinates (u, v) of the object 101 in the two-dimensional imaging coordinate system and the coordinates (X, Y, Z) in the three-dimensional coordinate system can be expressed, for example, as in equation (2) below, using a pinhole camera model.

s[u v 1] ^T = A[R t][X Y 0 1] ^T = A[r ₀ r ₁ t][X Y 1] ^T
[X Y 1] ^T = s(A[r ₀ r ₁ t]) ^-1 [u v 1] ^T … (2)
Here, R is the rotation matrix [r ₀ , r ₁ , r ₂ ], t is the translation vector, A is the camera matrix (external parameter), and s is a constant, which can be obtained from the constraint that the third dimension on the right-hand side of equation (2) above is 1.

The first position estimation unit 1 uses the above conversion formula to convert the estimated coordinates of the object 101 in the imaging coordinate system into coordinates in a three-dimensional coordinate system. This allows the position of the object 101 on the upstream side of the belt conveyor 100 to be estimated.

The speed estimation unit 2 estimates the moving speed of the object 101 from image data acquired from the imaging device 21. The moving speed of the object 101 estimated by the speed estimation unit 2 means the speed used in calculations when the second position estimation unit 3 estimates the position of the object 101 (hereinafter referred to as the calculation speed), and is not the actual moving speed of the object 101. The speed estimation unit 2 may acquire the speed of the belt conveyor 100, for example, from a speed sensor, and estimate the calculation speed from the acquired speed, or may use the acquired speed as the calculation speed. The speed estimation unit 2 may also use the average value of the speeds of the multiple objects 101 as the calculation speed.

The second position estimation unit 3 estimates the coordinates of the object 101 in the three-dimensional coordinate system at a predetermined time (the time of transmission to the linked device, i.e., the time of executing the marking process) based on the coordinates of the object 101 in the three-dimensional coordinate system estimated by the first position estimation unit 1 and the calculation speed estimated by the speed estimation unit 2. As described above, the positional relationship (distance) between the imaging area 105 and the imaging area 108 is known. The second position estimation unit 3 estimates the coordinates of the object 101 within the imaging area 108 in the three-dimensional coordinate system from that distance and the position and speed of the object 101.

Note that the object 101 on the belt conveyor 100 moves in a straight line.

Figure 6 is a diagram illustrating the conversion from coordinates in a three-dimensional coordinate system to coordinates in a projection imaging coordinate system.

The second position estimation unit 3 converts the coordinates of the estimated three-dimensional coordinate system into the projection imaging coordinate system using the following equation (4): s is a constant as above, and is obtained from the constraint that the third dimension on the right side of the following equation (4) is 1.
s[u v 1] ^T = (A[r ₀ r ₁ t])[X Y 1] ^T … (4)

The second position estimation unit 3 uses the lens distortion coefficient, an internal parameter, to add the lens distortion of the imaging device 23 to the coordinates in the projection imaging coordinate system estimated by equation (4) above. Specific methods for adding lens distortion are described, for example, at URL: https://docs.opencv.org/4.5.2/d9/d0c/group__calib3d.html.

The second position estimation unit 3 converts the coordinates (u, v) in the projection imaging coordinate system into coordinates (x, y) in the projection coordinate system based on the calibration results between the projection device 22 and the imaging device 23 performed by the calibration unit 5.

At a predetermined time, the processing execution unit 4 causes the projection device 22 to mark the object 101 located at a position in the imaging area 108 estimated by the second position estimation unit 3 using the light-projecting marker 102 (see Figure 3).

Next, the operation of the image processing device in this embodiment will be described using Figure 7. Figures 7 and 8 are flow diagrams showing the operation of the image processing device 10. In the following description, Figures 1 to 6 will be referred to as appropriate. Furthermore, in this embodiment, an image processing method is implemented by operating the image processing device. Therefore, the description of the image processing method in this embodiment will be replaced by the description of the operation of the image processing device 10 below.

Figure 7 is a flow diagram showing the operation of the image processing device 10 when performing calibration.

The calibration unit 5 performs calibration to estimate the internal parameters of the imaging device 21 and the imaging device 23 (S1). Next, the calibration unit 5 performs calibration to estimate the external parameters of the imaging device 21 and the imaging device 23 (S2). The calibration unit 5 performs calibration between the projection device 22 and the imaging device 23 (S3).

Note that the execution order of S1 to S3 in Figure 7 can be changed as appropriate.

Figure 8 is a flow diagram showing the operation of the image processing device 10 during position estimation.

The image processing device 10 acquires image data from the imaging device 21 (S1). The first position estimation unit 1 estimates the position of the object 101 in the imaging area 105, i.e., the coordinates of the object 101 in the imaging coordinate system, based on the acquired image data (S12). At this time, the first position estimation unit 1 corrects lens distortion in the image data and then estimates the coordinates of the object 101 in the imaging coordinate system.

Next, the first position estimation unit 1 converts the estimated coordinates in the imaging coordinate system into coordinates in a three-dimensional coordinate system (S13). This conversion is performed using equation (2) above.

The speed estimation unit 2 estimates the moving speed of the object 101 from the acquired image data (S14). Here, the speed estimated by the speed estimation unit 2 is the calculation speed described above. In other words, the speed estimation unit 2 calculates the conveyor speed of the belt conveyor 100 from the speeds of the multiple objects 101, and estimates this conveyor speed as the calculation speed.

The second position estimation unit 3 estimates the coordinates of the object 101 in the imaging area 108 in three-dimensional coordinates from the coordinates in the three-dimensional coordinate system converted in S13 and the velocity estimated in S14 (S15). The second position estimation unit 3 converts the estimated three-dimensional coordinates into coordinates in the projection imaging coordinate system (S16).

The second position estimation unit 3 converts the coordinates in the projection imaging coordinate system into coordinates in the projection coordinate system (S17). The second position estimation unit 3 performs the conversion in S17 based on the calibration results between the projection device 22 and the imaging device 23 performed by the calibration unit 5.

The processing execution unit 4 uses the projection device 22 to perform marking with the light-projecting marker 102 (see Figure 3) on the object 101 located at the coordinates in the projection coordinate system obtained in S17 (S18).

As described above, the image processing device 10 according to this embodiment enables separation of the imaging area 105 and the processing area 106, preventing unnecessary objects from appearing in the image when capturing an image of the imaging area 105. As a result, the image processing device 10 achieves highly accurate position detection of the object 101 and inspection of defective products.

In this embodiment, optical processing is performed by the projection device 22 in the processing area 106, but physical processing, such as removing the target object 101 using a robot arm, may also be performed.

The program in this embodiment may be any program that causes a computer to execute steps S1 to S3 shown in FIG. 7 and steps S11 to S18 shown in FIG. 8. By installing and executing this program on a computer, the image processing device and image processing method of this embodiment can be realized. In this case, the computer's processor functions as a first position estimation unit 1, a speed estimation unit 2, a second position estimation unit 3, a processing execution unit 4, and a calibration unit 5, and performs processing.

The program in this embodiment may also be executed by a computer system constructed with multiple computers. In this case, for example, each computer may function as one of the first position estimation unit 1, speed estimation unit 2, second position estimation unit 3, processing execution unit 4, and calibration unit 5.

Here, a computer that implements the image processing device by executing the program of this embodiment will be described using Figure 9. Figure 9 is a block diagram showing an example of a computer that implements the image processing device 10 of this embodiment.

As shown in FIG. 9, the computer 110 includes a CPU 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader/writer 116, and a communication interface 117. These components are connected to each other via a bus 121 so that they can communicate data with each other. Note that the computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to or instead of the CPU 111.

The CPU 111 performs various calculations by expanding the program (code) of this embodiment, which is stored in the storage device 113, into the main memory 112 and executing it in a predetermined order. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory). The program of this embodiment is provided in a state stored on a computer-readable recording medium 120. The program of this embodiment may also be distributed over the Internet connected via the communication interface 117.

Specific examples of the storage device 113 include a hard disk drive and semiconductor storage devices such as flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and mouse. The display controller 115 is connected to the display device 119 and controls the display on the display device 119.

The data reader/writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads programs from the recording medium 120, and writes the processing results of the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and other computers.

Specific examples of the recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), magnetic recording media such as flexible disks, and optical recording media such as CD-ROMs (Compact Disk Read Only Memory).

Note that the image processing device 10 in this embodiment can also be realized by using hardware corresponding to each part, rather than a computer on which a program is installed. Furthermore, the image processing device 10 may be partially realized by a program and the remaining part by hardware.

Some or all of the above-described embodiments can be expressed by (Appendix 1) to (Appendix 30) described below, but are not limited to the following descriptions.

(Appendix 1)
a first position estimation unit that estimates a position of the object in a first area based on image data of the first area set upstream of a path along which the object moves;
a speed estimation unit that estimates a speed of the object on the path;
a second position estimation unit that, when the object satisfies a set condition, estimates a position of the object in a second area set downstream of the route at a predetermined time based on the detected position of the object and the estimated speed of the object; and
An image processing device comprising:

(Appendix 2)
2. The image processing device according to claim 1,
The first position estimation unit
Estimating coordinates of the object in a first coordinate system in the image data, and converting the coordinates of the object in the first coordinate system into coordinates in a second coordinate system set on the path;
The second position estimation unit
estimating the coordinates of the object in the second coordinate system at the predetermined time based on the transformed coordinates in the second coordinate system and the velocity of the object;
Image processing device.

(Appendix 3)
3. The image processing device according to claim 2,
The speed estimation unit
estimating a velocity of the object based on coordinates in the second coordinate system obtained by converting coordinates of the object in the first coordinate system;
Image processing device.

(Appendix 4)
10. The image processing device according to claim 2, wherein:
a first calibration unit that estimates parameters of an imaging device that captures the first region;
An image processing device comprising:

(Appendix 5)
10. The image processing device according to claim 2, wherein:
The first calibration unit
Estimating parameters for converting coordinates in the coordinate system of the imaging device and coordinates in the second coordinate system;
Image processing device.

(Appendix 6)
6. The image processing device according to claim 1,
a processing execution unit that executes optical or physical processing on the object located at the estimated position in the second area at the predetermined time;
The image processing device further comprises:

(Appendix 7)
7. The image processing device according to claim 6,
the processing execution unit executes optical marking on the object located at the estimated position in the second area at the predetermined time using a projection device.
Image processing device.

(Appendix 8)
8. The image processing device according to claim 6,
a second calibration unit that estimates parameters of an imaging device that captures the second region;
An image processing device comprising:

(Appendix 9)
9. The image processing device according to claim 8,
The second calibration unit
Estimating parameters for converting coordinates in the second coordinate system and coordinates in the coordinate system of the imaging device;
Image processing device.

(Appendix 10)
10. The image processing device according to claim 9,
The second calibration unit
Estimating parameters for converting coordinates in the coordinate system of the imaging device and coordinates in the coordinate system of the projection device;
Image processing device.

(Appendix 11)
a step of estimating a position of the object in a first area based on image data of the first area set upstream of a path along which the object moves;
estimating a velocity of the object along the path;
If the object satisfies a set condition, estimating a position of the object in a second area set downstream of the route at a predetermined time based on the detected position of the object and the estimated speed of the object;
An image processing method comprising:

(Appendix 12)
12. The image processing method according to claim 11,
In the step of estimating the position of the object in the first region,
Estimating coordinates of the object in a first coordinate system in the image data, and converting the coordinates of the object in the first coordinate system into coordinates in a second coordinate system set on the path;
In the step of estimating a position in the second region,
estimating the coordinates of the object in the second coordinate system at the predetermined time based on the transformed coordinates in the second coordinate system and the velocity of the object;
Image processing methods.

(Appendix 13)
13. The image processing method according to claim 12, further comprising:
In the step of estimating the velocity of the object,
estimating a velocity of the object based on coordinates in the second coordinate system obtained by converting coordinates of the object in the first coordinate system;
Image processing methods.

(Appendix 14)
14. The image processing method according to claim 12, further comprising:
estimating parameters of an imaging device that images the first region;
An image processing method comprising:

(Appendix 15)
14. The image processing method according to claim 12, further comprising:
In the step of estimating parameters of an imaging device that captures an image of the first region,
Estimating parameters for converting coordinates in the coordinate system of the imaging device and coordinates in the second coordinate system;
Image processing methods.

(Appendix 16)
16. An image processing method according to any one of Supplementary Note 11 to Supplementary Note 15, comprising:
performing optical or physical processing on the object at the estimated position in the second region at the predetermined time;
The image processing method further comprises:

(Appendix 17)
17. The image processing method according to claim 16, further comprising:
In the step of performing the processing, optical marking is performed on the object located at the estimated position in the second area at the predetermined time by a projection device.
Image processing methods.

(Appendix 18)
18. The image processing method according to claim 16, further comprising:
estimating parameters of an imaging device that images the second region;
An image processing method comprising:

(Appendix 19)
19. The image processing method of claim 18, further comprising:
In the step of estimating parameters of an imaging device that images the second region, parameters for converting coordinates in the second coordinate system and coordinates in a coordinate system of the imaging device are estimated.
Image processing methods.

(Appendix 20)
20. The image processing method of claim 19, further comprising:
In the step of estimating parameters of an imaging device that captures the second area, parameters for converting coordinates in a coordinate system of the imaging device and coordinates in a coordinate system of the projection device are estimated.
Image processing methods.

(Appendix 21)
On the computer,
a step of estimating a position of the object in a first area based on image data of the first area set upstream of a path along which the object moves;
estimating a velocity of the object along the path;
If the object satisfies a set condition, estimating a position of the object in a second area set downstream of the route at a predetermined time based on the detected position of the object and the estimated speed of the object;
A program that executes.

(Appendix 22)
22. The program of claim 21,
In the step of estimating the position of the object in the first region,
Estimating coordinates of the object in a first coordinate system in the image data, and converting the coordinates of the object in the first coordinate system into coordinates in a second coordinate system set on the path;
In the step of estimating a position in the second region,
estimating the coordinates of the object in the second coordinate system at the predetermined time based on the transformed coordinates in the second coordinate system and the velocity of the object;
program.

(Appendix 23)
23. The program of claim 22,
In the step of estimating the velocity of the object,
estimating a velocity of the object based on coordinates in the second coordinate system obtained by converting coordinates of the object in the first coordinate system;
program.

(Appendix 24)
24. The program according to claim 22 or 23,
a step of causing the computer to estimate parameters of an imaging device that images the first region;
A program that executes.

(Appendix 25)
24. The program according to claim 22 or 23,
In the step of estimating parameters of an imaging device that captures an image of the first region,
Estimating parameters for converting coordinates in the coordinate system of the imaging device and coordinates in the second coordinate system;
program.

(Appendix 26)
26. The program according to any one of Supplementary Note 21 to Supplementary Note 25,
The computer,
performing optical or physical processing on the object at the estimated position in the second region at the predetermined time;
A program that executes.

(Appendix 27)
27. The program of claim 26,
In the step of performing the processing, optical marking is performed on the object located at the estimated position in the second area at the predetermined time by a projection device.
program.

(Appendix 28)
28. The program according to claim 26 or 27,
The computer,
estimating parameters of an imaging device that images the second region;
A program that executes.

(Appendix 29)
39. The program of claim 38,
In the step of estimating parameters of an imaging device that images the second region, parameters for converting coordinates in the second coordinate system and coordinates in a coordinate system of the imaging device are estimated.
program.

(Appendix 30)
29. The program of claim 28,
In the step of estimating parameters of an imaging device that captures the second area, parameters for converting coordinates in a coordinate system of the imaging device and coordinates in a coordinate system of the projection device are estimated.
program.

The image processing device disclosed herein can be used in a system that estimates the position of a transported object along a path and performs optical or physical processing on the object.

1: First position estimation unit 2: Speed estimation unit 3: Second position estimation unit 4: Processing execution unit 5: Calibration unit 10: Image processing device 21: Imaging device 22: Projection device 23: Imaging device 100: Belt conveyor 100A: Marker 101: Object 102: Light-projecting marker 105: Imaging area 106: Processing area 107: Projection area 108: Imaging area 110: Computer 111: CPU
112: Main memory 113: Storage device 114: Input interface 115: Display controller 116: Writer 117: Communication interface 118: Input device 119: Display device 120: Recording medium 121: Bus

Claims (6)

a first position estimation unit that estimates a position of the object in a first area based on image data of the first area set upstream of a path along which the object moves;
a speed estimation unit that estimates a speed of the object on the path;
a second position estimation unit that, when the object satisfies a set condition, estimates a position of the object in a second area set downstream of the route at a predetermined time based on the detected position of the object and the estimated speed of the object; and
a processing execution unit that executes marking with a light-projecting marker on the object located at the estimated position in the second area at the predetermined time using a projection device whose second area overlaps with a projection area;
a calibration unit that estimates a geometric transformation between a coordinate system of the imaging device and a projection coordinate system of the projection device based on image data obtained by capturing an image of the second area using an imaging device that captures an image of the projection pattern projected by the projection device;
Equipped with
the second position estimation unit transforms the position of the object in the second region into the projected coordinate system using the estimated geometric transformation;
the processing execution unit executes marking on the object using the transformed position of the object in the second area.
Image processing device. 2. The image processing device according to claim 1,
The first position estimation unit
Estimating coordinates of the object in a first coordinate system in the image data, and converting the coordinates of the object in the first coordinate system into coordinates in a second coordinate system set on the path;
The second position estimation unit
estimating the coordinates of the object in the second coordinate system at the predetermined time based on the transformed coordinates in the second coordinate system and the velocity of the object;
Image processing device. 3. The image processing device according to claim 2,
The speed estimation unit
estimating a velocity of the object based on coordinates in the second coordinate system obtained by converting coordinates of the object in the first coordinate system;
Image processing device. 2. The image processing device according to claim 1 ,
a processing execution unit that executes optical or physical processing on the object located at the estimated position in the second area at the predetermined time;
The image processing device further comprises: a step of estimating a position of the object in a first area based on image data of the first area set upstream of a path along which the object moves;
estimating a velocity of the object along the path;
If the object satisfies a set condition, estimating a position of the object in a second area set downstream of the route at a predetermined time based on the detected position of the object and the estimated speed of the object;
At the predetermined time, marking the object located at the estimated position in the second area with a light-projecting marker using a projection device whose second area overlaps with a projection area;
a step of estimating a geometric transformation between a coordinate system of the imaging device and a projection coordinate system of the projection device based on image data obtained by capturing an image of the second area using an imaging device that captures an image of the projection pattern projected by the projection device;
Equipped with
In estimating the position in the second region, the position of the object in the second region is transformed into the projected coordinate system using the estimated geometric transformation;
performing marking on the object using the transformed position of the object in the second region;
Image processing methods. On the computer,
a step of estimating a position of the object in a first area based on image data of the first area set upstream of a path along which the object moves;
estimating a velocity of the object along the path;
If the object satisfies a set condition, estimating a position of the object in a second area set downstream of the route at a predetermined time based on the detected position of the object and the estimated speed of the object;
At the predetermined time, marking the object located at the estimated position in the second area with a light-projecting marker using a projection device whose second area overlaps with a projection area;
a step of estimating a geometric transformation between a coordinate system of the imaging device and a projection coordinate system of the projection device based on image data obtained by capturing an image of the second area using an imaging device that captures an image of the projection pattern projected by the projection device;
Execute
In estimating the position in the second region, the position of the object in the second region is transformed into the projected coordinate system using the estimated geometric transformation;
performing marking on the object using the transformed position of the object in the second region;
program.