JP7583249B2 - Industrial camera calibration operation support device - Google Patents

Description

The present invention relates to a support device that assists users in performing calibration operations on industrial cameras.

Conventionally, there is a calibration operation support device that detects the orientation of a calibration pattern in captured image data obtained by capturing an image of the calibration pattern, and rotates and displays the captured image so that the detected orientation is a predetermined orientation (see Patent Document 1). According to Patent Document 1, since the calibration pattern can always be displayed in a constant orientation, the movement of the hand moving the camera matches the movement of the displayed calibration pattern, making it easy to perform calibration operations.

JP 2008-135996 A

Incidentally, to perform accurate camera calibration, it is necessary to capture the calibration pattern from various angles using as much of the captured image as possible. In Patent Document 1, although the movement of the hand moving the camera matches the movement of the displayed calibration pattern, this does not provide any reference for the user to decide what kind of captured image should be captured.

The present invention was made to solve these problems, and its main purpose is to help a user determine what kind of captured images should be captured in a support device that supports the user in calibrating an industrial camera.

The first means for solving the above problem is to
1. A calibration operation support device for an industrial camera, which is applied to a system including a camera that captures an image of a preset predetermined area, a display device that displays a predetermined image that is an image of the predetermined area being captured by the camera, and a processing device that calibrates the camera based on a plurality of captured images of a calibration pattern captured by the camera,
The position of the calibration pattern in the plurality of captured images captured as images to be used for the calibration is superimposed on the predetermined image and displayed by the display device.

According to the above configuration, the camera captures an image of a predetermined area. The display device displays a predetermined image, which is an image of the predetermined area being captured by the camera. The processing device calibrates the camera based on multiple captured images of the calibration pattern captured by the camera.

Here, the calibration operation support device for industrial cameras displays the positions of the calibration patterns in the multiple captured images captured as images to be used for calibration on a display device, superimposed on a specified image. This allows the user to know in which position of the specified area the captured images in which the calibration pattern has been placed have already been captured. This makes it easy for the user to place the calibration pattern in a position where it has not yet been placed and capture the captured images, thereby helping the user determine what kind of captured images should be captured.

The second means is
1. A calibration operation support device for an industrial camera, which is applied to a system including a camera that captures an image of a preset predetermined area, a display device that displays a predetermined image that is an image of the predetermined area being captured by the camera, and a processing device that calibrates the camera based on a plurality of captured images of a calibration pattern captured by the camera,
The calibration pattern in the plurality of captured images captured as images to be used for the calibration has an effect of improving the accuracy of the calibration in the specified region, and the distribution of this effect is superimposed on the specified image and displayed on the display device.

According to the above configuration, the calibration operation support device for industrial cameras displays, on a display device, the distribution of the effect of the calibration patterns in the multiple captured images captured as images used for calibration on improving the accuracy of the calibration in a specified area (hereinafter referred to as the "accuracy improvement effect"), superimposed on the specified image. This allows the user to understand to what extent the accuracy improvement effect of the calibration patterns in the multiple captured images is being exerted on which parts of the specified area. This makes it easier for the user to place the calibration patterns in positions where the accuracy improvement effect is insufficient and capture the captured images, thereby assisting the user in determining what kind of captured images should be captured.

In a third method, the calibration pattern includes a plurality of feature points, and the distribution of the effect is a distribution of total values at each position, the total value being a predetermined value that becomes smaller the further away from each feature point, with the position of each feature point as a vertex. With this configuration, the distribution of the accuracy improvement effect in a specified region can be easily represented using a predetermined value that is largest at the position of each feature point and becomes smaller the further away from each feature point. Furthermore, because the distribution of the accuracy improvement effect is represented as a distribution of total values being a predetermined value at each position, it is possible to prevent the process of determining the distribution of the accuracy improvement effect from becoming complicated even when the number of captured images captured is large.

Specifically, as in the fourth means, a configuration can be adopted in which the predetermined value is a value based on each normal distribution with the position of each feature point as its apex. With this configuration, the accuracy improvement effect of each feature point in the predetermined region can be reflected even in positions distant from each feature point. Therefore, it becomes easier to quantitatively grasp to what extent the accuracy improvement effect extends to which part of the predetermined region.

In the fifth method, the predetermined value is a value greater than 0 throughout the entirety of the predetermined region. This configuration makes it possible to prevent the occurrence of parts of the predetermined region where the accuracy improvement effect is zero. This makes it easier to quantitatively grasp the extent to which the accuracy improvement effect extends to which parts of the predetermined region.

In the sixth method, the position of the calibration pattern in the specified region where the sum of the respective total values in the region corresponding to the calibration pattern is the smallest is determined as the position where the calibration pattern should be placed in the specified region, and is displayed by the display device superimposed on the specified image.

According to the above configuration, the position of the calibration pattern where the sum of the total values obtained by adding up the predetermined values at each position within the area corresponding to the calibration pattern is smallest, that is, the position of the calibration pattern where the effect of improving accuracy of the calibration pattern in the already captured captured image is smallest, can be determined as the position at which the calibration pattern should be placed in the predetermined area. Then, the position at which the calibration pattern should be placed in the predetermined area is displayed by the display device, superimposed on the predetermined image. Therefore, when photographing the calibration pattern with a camera, the user can easily understand where the calibration pattern should be placed.

If camera calibration has not been performed, the calibration operation support device cannot accurately calculate the position of each feature point of the calibration pattern in three-dimensional space based on the captured image. As a result, the calibration operation support device cannot accurately grasp the distribution of the accuracy improvement effect in a specified area in three-dimensional space.

In this regard, in the seventh means, the specified area is regarded as a two-dimensional plane, the position of each of the feature points is regarded as a position on the two-dimensional plane, and the distribution of the effect is superimposed on the specified image and displayed by the display device. With this configuration, even if the camera calibration has not been performed, the calibration operation support device can grasp and display the distribution of the accuracy improvement effect in the specified area as a two-dimensional plane. Therefore, it becomes easy for the user to determine what kind of captured image should be imported when performing the calibration.

As described above, if the camera has not been calibrated, the calibration operation support device cannot accurately grasp the distribution of the accuracy improvement effect in a specified area in three-dimensional space. In addition, it is difficult to display the exact position where the calibration pattern should be placed in the specified area in three-dimensional space by superimposing it on the specified image.

In this regard, based on the seventh means, the eighth means causes the display device to display the position where the calibration pattern should be placed in the specified area as the two-dimensional plane, superimposed on the specified image. With this configuration, even if the camera calibration has not been performed, the calibration operation support device can grasp the distribution of the accuracy improvement effect in the specified area as a two-dimensional plane, and can therefore display the position where the calibration pattern should be placed, superimposed on the specified image.

After performing camera calibration, the camera parameters obtained by the calibration can be used to accurately calculate the position of each feature point of the calibration pattern in three-dimensional space based on the captured image.

In the ninth aspect, after the calibration is performed, the specified area is regarded as a three-dimensional space using the camera parameters acquired by the calibration, the positions of the feature points are regarded as positions within the three-dimensional space, and the distribution of the effect is superimposed on the specified image and displayed by the display device. With this configuration, the calibration operation support device for an industrial camera can accurately grasp and display the distribution of the accuracy improvement effect in a specified area as a three-dimensional space. Therefore, it becomes easy for a user to place a calibration pattern in a position where the accuracy improvement effect is insufficient in the specified area as a three-dimensional space and capture a captured image.

In a tenth aspect, the position at which the calibration pattern should be placed in the specified area in the three-dimensional space is displayed by the display device, superimposed on the specified image.

With the above configuration, when a user photographs a calibration pattern with a camera, the user can easily understand where the calibration pattern should be placed in a specific area in three-dimensional space.

In an eleventh aspect, the position at which the calibration pattern should be placed in the specified area is displayed on the display device while being superimposed on the specified image. With this configuration, the user can easily understand where the calibration pattern should be placed when photographing the calibration pattern with a camera.

In a twelfth aspect, the height of the calibration pattern in the multiple captured images captured as images to be used for the calibration is superimposed on the specified image and displayed by the display device. With this configuration, the user can know at what height the captured images at which the calibration pattern has been arranged have already been captured. Therefore, it becomes easy for the user to place the calibration pattern at a height where it has not yet been arranged and capture the captured images, and it is possible to assist the user in determining what kind of captured images should be captured.

In a thirteenth aspect, the height at which the calibration pattern should be placed in the specified area is displayed on the display device while being superimposed on the specified image. With this configuration, the user can easily understand at what height the calibration pattern should be placed when photographing the calibration pattern with a camera.

In a fourteenth aspect, the inclination of the calibration pattern in the multiple captured images captured as images to be used for the calibration is superimposed on the predetermined image and displayed by the display device. With this configuration, the user can know what inclination of the captured images at which the calibration pattern has been arranged has already been captured. Therefore, it becomes easy for the user to place the calibration pattern at an inclination where it has not yet been arranged and capture the captured image, and it is possible to assist the user in determining what kind of captured image should be captured.

In a fifteenth aspect, the display device displays the inclination at which the calibration pattern should be placed in the specified area, superimposed on the specified image. With this configuration, the user can easily understand the inclination at which the calibration pattern should be placed when photographing the calibration pattern with a camera.

In a sixteenth aspect, when the arrangement of the calibration pattern in the specified area satisfies a specified condition, the image captured by the camera is imported as an image to be used for the calibration. With this configuration, the user does not need to perform any operation to import an image to be used for the calibration, reducing the user's effort.

In the seventeenth means, when the number of the plurality of captured images imported as images to be used for the calibration exceeds a predetermined number, the processing device performs the calibration. With this configuration, the user does not need to perform any operation for the calibration, and the user's effort can be reduced.

System Schematic In the first embodiment, an image showing a calibration pattern being photographed and a position of the calibration pattern in a captured photographed image An image showing a calibration pattern and a predetermined straight line being photographed in the second embodiment FIG. 4 is a schematic diagram showing feature points existing on a predetermined straight line in FIG. 3; 5 is a graph showing a comparative example of the distribution of the accuracy improvement effect on the predetermined straight line in FIG. A graph showing the distribution of the accuracy improvement effect on a specific straight line in FIG. Flowchart showing the procedure for supporting calibration operations Image showing the calibration pattern and scanning rectangle Schematic diagram showing the distribution of accuracy improvement effects FIG. 1 is a schematic diagram showing the positions of calibration patterns in a plurality of captured images; Schematic diagram showing the distribution of accuracy improvement effect by using a calibration pattern on multiple captured images FIG. 12 is a schematic diagram showing the distribution of the accuracy improvement effect of FIG. 11 converted into a logarithmic representation. FIG. 13 is a schematic diagram showing a mode of searching for a target arrangement position of a calibration pattern; FIG. 1 is a schematic diagram showing a target arrangement position of a calibration pattern; Schematic diagram showing the inclination of the calibration pattern Schematic diagram showing the inclination distribution of the calibration pattern

First Embodiment
Hereinafter, a first embodiment embodied in a system used in a part assembly process (a work process for a workpiece) by a robot (industrial machine) will be described with reference to the drawings.

As shown in FIG. 1, the system 10 includes a camera 11, a processing device 12, a display device 13, etc.

The camera 11 (industrial camera) is capable of taking still and video images, and transmits the captured image data to the processing device 12. The camera 11 is fixed by a stand 11a or the like so as to take images of a predefined area 20. The predefined area 20 is an area where parts are assembled (work on the workpiece) during the part assembly process by the robot, and has a predetermined width, depth, and height.

The display device 13 is composed of an LCD screen or the like, and displays images based on signals input from the processing device 12.

The processing device 12 is a computer equipped with a CPU, ROM, RAM, an operation unit, a storage device, an input/output interface, etc. The CPU executes various controls using programs and data stored in the ROM and RAM. The operation unit is composed of a keyboard, mouse, etc., and inputs various instructions to the CPU by accepting user operations. The storage device stores various programs and data.

The processing device 12 causes the display device 13 to display the predetermined image 25 (captured image) captured by the camera 11. The predetermined image 25 is an image of the predetermined area 20 that is currently being captured (captured) by the camera 11. Figure 1 shows a state in which a calibration pattern 30 is arranged within the predetermined area 20.

The calibration pattern 30 can have any known configuration, for example, formed in a plate shape with predetermined markers 30a marked in a predetermined arrangement. The calibration pattern 30 is used for the process of determining camera parameters such as the lens center and focal length of the camera 11, i.e., for calibrating the camera 11.

The processing device 12 calibrates the camera 11 based on a plurality of captured images of the calibration pattern 30 captured by the camera 11. In detail, the camera 11 continuously captures the predetermined area 20, and the processing device 12 continuously displays the predetermined image 25 captured by the camera 11 on the display device 13. However, at that stage, the predetermined image 25 has not been imported into the processing device 12 as an image to be used for calibration. When the camera 11 starts capturing the predetermined area 20, the processing device 12 may determine whether calibration is possible. Whether calibration is possible can be determined based on, for example, the brightness of the image captured by the camera 11, the focus, the presence or absence of an obstruction, etc. If the processing device 12 determines that calibration is possible, it may cause the display device 13 to display an instruction to perform calibration. If the processing device 12 determines that calibration is not possible, it may cause the display device 13 to display an instruction to correct the shooting conditions of the camera 11. Furthermore, if calibration is not started within a predetermined period after the processing device 12 has displayed an instruction to perform calibration on the display device 13, the processing device 12 may cause the display device 13 to display an instruction to perform calibration again.

When the arrangement state of the calibration pattern 30 in the specified area 20 satisfies a specified condition, the processing device 12 imports the specified image 25 captured by the camera 11 as an image to be used for calibration. In other words, when importing the specified image 25 to be used for calibration into the processing device 12, the user does not need to perform any operation to import the specified image 25 into the processing device 12.

More specifically, the processing device 12 identifies the position of the calibration pattern 30 based on the data of the captured image captured by the camera 11. A method for identifying the position of the calibration pattern 30 will be described later. The processing device 12 determines that the above-mentioned specified condition is satisfied when the identified position of the calibration pattern 30 is included in any one of a plurality of position ranges that are desirable for performing calibration with high accuracy. The plurality of desirable position ranges are set, for example, near the four corners and near the center of the specified image 25.

Next, we will explain the support device that is applied to the system 10 and supports the user in performing the calibration operation of the camera 11. In this embodiment, the support device (calibration operation support device for an industrial camera) is composed of a processing device 12.

Figure 2 shows an image showing the calibration pattern 30 being photographed and the position of the calibration pattern 30 in the captured photographed image. The processing device 12 displays the position of the calibration pattern 30 in the multiple photographed images captured as images to be used for calibration on the display device 13 by superimposing it on the specified image 25.

More specifically, the processing device 12 identifies the positions of the multiple markers 30a based on the data of the captured image captured as an image to be used for calibration. For example, the processing device 12 identifies the contour line 31 (outline) of the area in which the identified multiple markers 30a exist as the position of the calibration pattern 30. Then, the processing device 12 displays the contour line 31 as a dashed line. The contour lines 32 and 33 represent the position of the calibration pattern 30 in a captured image in which the calibration pattern 30 is placed at a position different from the contour line 31 and captured.

As described above, the contours 31 to 33 are displayed as the positions of the calibration pattern 30 that are already included in the captured captured image. This allows the user to easily determine where the calibration pattern 30 should be placed next as a position of the calibration pattern 30 that is not yet included in the captured captured image. When the user places the calibration pattern 30 in a desired position in the specified area 20, the processing device 12 determines that the above-mentioned specified conditions are satisfied, and captures the specified image 25 captured by the camera 11 as an image to be used for calibration.

The processing device 12 performs calibration each time the number of multiple captured images captured as images to be used for calibration exceeds a predetermined number. Once the number of multiple captured images captured by the processing device 12 exceeds the predetermined number, the processing device 12 performs calibration each time a new captured image is captured. This improves the accuracy of calibration each time calibration is performed. Since the method of performing calibration is well known, a description thereof will be omitted here. After that, the processing device 12 ends the series of processes for performing calibration when the camera parameters obtained as the calibration result are within the appropriate range. The processing device 12 may display the predetermined number of captured images required for calibration, the number of captured images currently captured, etc., on the display device 13. The processing device 12 may also display the degree of achievement or deficiency of the currently obtained camera parameters with respect to the values in the appropriate range on the display device 13.

The present embodiment described above has the following advantages:

- The processing device 12 (calibration operation support device for industrial cameras) causes the display device 13 to display the positions of the calibration pattern 30 in the multiple captured images captured as images to be used for calibration, superimposed on the specified image 25. This allows the user to know in which position in the specified area 20 the captured images in which the calibration pattern 30 has been placed have already been captured. This makes it easy for the user to capture captured images by placing the calibration pattern 30 in a position where it has not yet been placed, thereby facilitating the user's decision as to what type of captured images should be captured.

- When the arrangement state of the calibration pattern 30 in the specified area 20 satisfies a specified condition, the processing device 12 imports the image captured by the camera 11 as an image to be used for calibration. With this configuration, the user does not need to perform any operation to import the image to be used for calibration, reducing the user's effort.

When the number of multiple captured images imported as images to be used for calibration exceeds a predetermined number, the processing device 12 performs calibration. With this configuration, the user does not need to perform operations for calibration, reducing the user's workload.

The above embodiment can be modified as follows. The same parts as those in the above embodiment are designated by the same reference numerals and will not be described.

The processing device 12 can also display on the display device 13 the position at which the calibration pattern 30 should be placed in the specified area 20, superimposed on the specified image 25. Specifically, the processing device 12 displays on the display device 13 one of a number of position ranges desirable for performing accurate calibration, in which a captured image in which the calibration pattern 30 is placed has not been captured, or a specified position within one of those position ranges, superimposed on the specified image 25. With this configuration, the user can easily understand where the calibration pattern 30 should be placed when photographing the calibration pattern 30 with the camera 11.

The processing device 12 can also display the heights of the calibration pattern 30 in the multiple captured images captured as images used for calibration by superimposing them on the specified image 25 on the display device 13. Specifically, the processing device 12 identifies the heights of the various parts of the calibration pattern 30 based on the data of the captured images captured as images used for calibration. The processing device 12 then displays the interiors of the contours 31 to 33 in colors corresponding to the heights of the various parts. With this configuration, the user can know at what height the captured images at which the calibration pattern 30 has been placed have already been captured. This makes it easy for the user to capture captured images at a height at which the calibration pattern 30 has not yet been placed, thereby helping the user to determine what kind of captured images should be captured. It is also possible to calculate the average value of the heights of the various parts of the calibration pattern 30, and to display the interiors of the contours 31 to 33 in colors corresponding to the average heights, or to display the average heights numerically.

The processing device 12 can also display the height at which the calibration pattern 30 should be placed in the specified area 20 on the specified image 25 by the display device 13. Specifically, the processing device 12 displays on the display device 13 one of the height ranges, among a plurality of height ranges desirable for performing calibration with high accuracy, in which the captured image in which the calibration pattern 30 is placed has not been captured, or a predetermined position within one of the height ranges, by superimposing it on the specified image 25. With this configuration, the user can easily understand at what height the calibration pattern 30 should be placed when the calibration pattern 30 is captured by the camera 11. Furthermore, when the user places the calibration pattern 30 at the desired height in the specified area 20, the processing device 12 may determine that the above-mentioned predetermined condition is satisfied and capture the specified image 25 captured by the camera 11 as an image to be used for calibration.

The processing device 12 can also display the inclination of the calibration pattern 30 in the multiple captured images captured as images used for calibration by superimposing it on the specified image 25 on the display device 13. As above, the processing device 12 displays the inside of the contours 31 to 33 in a color corresponding to the height of each part. The user can understand the inclination of the calibration pattern 30 based on the degree of color change inside the contours 31 to 33. With this configuration, the user can understand what inclination of the captured image at which the calibration pattern 30 has been arranged has already been imported. Therefore, it becomes easy for the user to place the calibration pattern 30 at an inclination that has not yet been arranged and import the captured image, and it is possible to assist the user in determining what kind of captured image should be imported. Note that the inclination of the calibration pattern 30 can be calculated based on the height of each part of the calibration pattern 30, and the calculated inclination can be displayed numerically.

The processing device 12 can also display on the display device 13 the inclination at which the calibration pattern 30 should be arranged in the specified area 20, superimposed on the specified image 25. Specifically, the processing device 12 displays on the display device 13 one of the inclination ranges, among a plurality of inclination ranges desirable for performing calibration with high accuracy, in which the captured image in which the calibration pattern 30 is arranged is not captured, or a specified inclination value within one of the inclination ranges. With this configuration, the user can easily understand the inclination at which the calibration pattern 30 should be arranged when the calibration pattern 30 is captured by the camera 11. Furthermore, when the user arranges the calibration pattern 30 in the specified area 20 with the desired inclination, the processing device 12 may determine that the above-mentioned predetermined condition is satisfied, and may capture the specified image 25 captured by the camera 11 as an image to be used for calibration.

Second Embodiment
Hereinafter, a second embodiment will be described with reference to the drawings, in which the processing device 12 superimposes a distribution of the effect of improving the accuracy of calibration in a predetermined region 20 (hereinafter referred to as "accuracy improvement effect") of the calibration pattern 30 in a plurality of captured images captured as images to be used for calibration on a predetermined image 25 and displays it on the display device 13. Note that the same parts as those in the first embodiment are denoted by the same reference numerals and description thereof will be omitted.

Figure 3 is an image showing the calibration pattern 30 and the specified straight line L during shooting. The calibration pattern 30 is formed in a rectangular (rectangular) plate, and a number of feature points 30b are marked in an overall rectangular arrangement (specified arrangement). Here, the specified area 20 is regarded as a two-dimensional plane, and the position of each feature point 30b is regarded as a position on that two-dimensional plane. The specified straight line L is an arbitrary straight line drawn so as to overlap the calibration pattern 30 on that two-dimensional plane.

Figure 4 is a schematic diagram showing feature points 30b that exist on the specified line L in Figure 3. Here, four feature points 30b exist on the specified line L.

Figure 5 is a graph showing a comparative example of the distribution of the accuracy improvement effect on the specified line L in Figure 4. Each feature point 30b has the effect of improving the accuracy of calibration at each position. Therefore, the accuracy improvement effect is quantified by setting the accuracy improvement effect to 1.0 in the area of the calibration pattern 30 where multiple feature points 30b exist (inside the contour line 31) and setting the accuracy improvement effect to 0 in the area where no feature points 30b exist (outside the contour line 31). In this case, if not inside the contour line 31, the accuracy improvement effect is 0 regardless of whether it is close to the calibration pattern 30 or far from it. Therefore, outside the contour line 31, it is not possible to show the difference in the accuracy improvement effect between a position close to the calibration pattern 30 and a position far from the calibration pattern 30.

Figure 6 is a graph showing the distribution of the accuracy improvement effect on the specified straight line L in Figure 4. Here, the accuracy improvement effect is quantified as a specified value that becomes smaller as the position of each feature point 30b becomes the apex and the further away from the apex. In detail, the specified value is a value based on each normal distribution 30c with the position of each feature point 30b as the apex. In other words, the specified value is a value greater than 0 in the entire specified region 20. The distribution 30d of the accuracy improvement effect is a distribution of each total value obtained by summing the above-mentioned specified values at each position. In this case, outside the contour line 31, the closer to the calibration pattern 30, the greater the accuracy improvement effect, and the farther away from the calibration pattern 30, the smaller the accuracy improvement effect. Therefore, outside the contour line 31, the difference in the accuracy improvement effect can be shown between a position close to the calibration pattern 30 and a position far from the calibration pattern 30. Note that, although the above-mentioned specified value on the specified straight line L is shown as a two-dimensional graph in Figure 6, the above-mentioned specified value is a three-dimensional graph in the shape of a cone.

Figure 7 is a flowchart showing the procedure for supporting the calibration operation. This series of processes is executed by the processing device 12.

First, the camera 11 captures an image of the specified area 20 (S10).

Then, the two-dimensional position of each feature point 30b is identified (S11). More specifically, in the specified image 25 (captured image), which is an image of the specified area 20 being captured by the camera 11, the specified area 20 is regarded as a two-dimensional plane, the position of each feature point 30b is regarded as a position on that two-dimensional plane, and the position of each feature point 30b on the two-dimensional plane (hereinafter referred to as the "two-dimensional position") is identified.

Next, it is determined whether or not a scanning rectangle has not yet been created (S12). As shown in FIG. 8, the scanning rectangle 40 is a rectangle (figure) that corresponds to the contour line 31 (not shown) of an area in which multiple feature points 30b exist. In this determination, if it is determined that a scanning rectangle has not yet been created (S12: NO), the process proceeds to S14. On the other hand, in this determination, if it is determined that a scanning rectangle has not yet been created (S12: YES), a scanning rectangle 40 is created (S13). More specifically, a rectangle passing through each of the outermost feature points 30b is created based on the two-dimensional positions of each of the identified feature points 30b.

Next, it is determined whether the image capture condition (predetermined condition) is met (S14). More specifically, if the calibration pattern 30 is placed at the target placement position of the calibration pattern 30 described below or at a position near the target placement position, it is determined that the image capture condition is met, and otherwise it is determined that the image capture condition is not met. The position of the calibration pattern 30 is calculated based on the two-dimensional position of each of the identified feature points 30b. If it is determined that the image capture condition is not met (S14: NO), the process is executed again from S10.

On the other hand, if it is determined in S14 that the image capture condition is met (S14: YES), an image (predetermined image 25) of the specified area 20 being photographed by the camera 11 is captured (S15).

Then, it is determined whether calibration can be performed (S16). In detail, when the number of the captured images exceeds a predetermined number, it is determined that calibration can be performed. After the number of the captured images exceeds the predetermined number once, it is determined that calibration can be performed every time a new image is captured. On the other hand, when the number of the captured images does not exceed the predetermined number, it is determined that calibration cannot be performed. In this determination, when it is determined that calibration cannot be performed (S16: NO), the process proceeds to S18. On the other hand, when it is determined that calibration can be performed (S16: YES), the calibration is performed (S17). Note that, when the calibration of the camera 11 has not been performed, the processing device 12 cannot accurately calculate the position of each feature point 30b of the calibration pattern 30 in the three-dimensional space based on the captured images. Therefore, the processing device 12 cannot accurately grasp the distribution of the accuracy improvement effect in the predetermined area 20 as the three-dimensional space.

Next, a distribution of the accuracy improvement effect is created on a two-dimensional plane (S18). Figure 9 is a schematic diagram showing the distribution of the accuracy improvement effect corresponding to the specified image 25 in Figure 8. The closer to the center of the calibration pattern 30, the higher the value of the accuracy improvement effect contour line 51 becomes.

Then, the distribution of the accuracy improvement effect is displayed by superimposing it (S19). FIG. 10 is a schematic diagram showing the position of the calibration pattern 30 in a plurality of captured images. Here, the position of the calibration pattern 30 is shown by the contour lines 31 and 32. FIG. 11 is a schematic diagram showing the distribution of the accuracy improvement effect corresponding to the predetermined image 25 in FIG. 10, and shows a plurality of accuracy improvement effects superimposed. The closer to the center of the two calibration patterns 30, the higher the accuracy improvement effect. At positions away from the center of the two calibration patterns 30, it is difficult to see the difference in the accuracy improvement effect, but the further away from the center of the two calibration patterns 30, the lower the accuracy improvement effect. FIG. 12 is a schematic diagram showing the distribution of the accuracy improvement effect of FIG. 11 converted into a logarithmic display. In this figure, it is clearly shown that the further away from the center of the two calibration patterns 30, the lower the accuracy improvement effect.

Next, the target placement position of the calibration pattern 30 on the two-dimensional plane is displayed (S20). FIG. 13 is a schematic diagram showing a mode of searching for the target placement position of the calibration pattern 30. Here, for convenience of explanation, the distribution of the accuracy improvement effect by one calibration pattern 30 is shown. While changing the position of the scanning rectangle 40 (scanning), the position where the sum of the accuracy improvement effect in the scanning rectangle 40 is the smallest is searched for. In other words, the position of the calibration pattern 30 where the sum of the sums of the accuracy improvement effect at each position in the area corresponding to the calibration pattern 30 in the specified area 20 is the position where the calibration pattern 30 should be placed in the specified area 20. Then, the position where the sum of the accuracy improvement effect in the scanning rectangle 40 is the smallest is displayed by the display device 13 while being superimposed on the specified image 25. FIG. 14 is a schematic diagram showing the target placement position 60 of the calibration pattern 30. The target placement position 60 is displayed by the same rectangle as the scanning rectangle 40.

Next, it is determined whether calibration has been performed (S21). If it is determined that calibration has not been performed (S21: NO), the process proceeds to S26. On the other hand, if it is determined that calibration has been performed (S21: YES), the three-dimensional position of each feature point 30b is identified (S22). In detail, the predetermined area 20 is regarded as a three-dimensional space using the parameters related to the camera 11 acquired by the calibration, and the position of each feature point 30b is identified as a position in the three-dimensional space (three-dimensional position). After the calibration of the camera 11 is performed, the position of each feature point 30b of the calibration pattern 30 in the three-dimensional space can be accurately calculated based on the captured image using the parameters related to the camera 11 acquired by the calibration.

Next, a distribution of the accuracy improvement effect is created in three-dimensional space (S23). In detail, the same process as S18 and FIG. 9 is applied to the three-dimensional space and executed.

Then, the distribution of the accuracy improvement effect (height and gradient) is displayed in an overlapping manner (S24). In detail, the same process as that of S19 and Figures 10 to 12 is performed but applied to a three-dimensional space.

Next, the target placement position of the calibration pattern 30 in three-dimensional space is displayed (S25). More specifically, the same processing as S20 and FIGS. 13 and 14 is applied to the three-dimensional space and executed, and the target placement position (height and tilt) of the calibration pattern 30 is displayed as a three-dimensional position. Note that in addition to the position of the calibration pattern 30 where the sum of the respective total values of the accuracy improvement effect of each position in the region corresponding to the calibration pattern 30 in the specified region 20 is the smallest, the position of the calibration pattern 30 where the sum of the respective total values of the accuracy improvement effect is the second smallest may be displayed as the target placement position.

Next, it is determined whether calibration has been performed (S26). If it is determined that calibration has not been performed (S26: NO), the process proceeds to S31. On the other hand, if it is determined that calibration has been performed (S26: YES), the inclination of the calibration pattern 30 is identified (S27). In detail, the three-dimensional position of each feature point 30b is identified, and the inclination of the plane through which each feature point 30b passes, that is, the inclination of the calibration pattern 30, is identified. FIG. 15 is a schematic diagram showing the inclination of the calibration pattern 30. Here, the inclination Rx and the inclination Ry are identified among the inclination Rx around the x-axis, the inclination Ry around the y-axis, and the inclination Rz around the z-axis. The inclination Rz around the z-axis is omitted because it has little effect on the accuracy improvement effect. For example, in the arrangement state P1, the inclination Rx = 0° and the inclination Ry = 0°. In arrangement state P2, the inclination Rx = 20° and the inclination Ry = 20°.

Next, a gradient distribution of the calibration pattern 30 is created (S28). More specifically, a distribution is created based on a normal distribution, in which the identified gradients Rx and Ry are at the apex and the values become smaller the further away from the apex.

Next, the tilt distributions are displayed in an overlapping manner (S29). FIG. 16 is a schematic diagram showing the tilt distribution of the calibration pattern 30. In the figure, the tilt distribution in arrangement state P1 and the tilt distribution in arrangement state P2 are displayed in an overlapping manner.

Then, the target arrangement angle of the calibration pattern 30 is displayed (S30). More specifically, based on the tilt distribution in FIG. 16, the arrangement angle with the smallest distribution value within the range of tilts Rx, Ry from -45° to 45° is displayed as the target arrangement angle. Note that if the tilts Rx, Ry are outside the range of -45° to 45°, it may become impossible to recognize the calibration pattern 30.

Next, it is determined whether or not the termination condition is satisfied (S31). In more detail, if at least one of the following is satisfied, it is determined that the termination condition is satisfied, and if none of the following is satisfied, it is determined that the termination condition is not satisfied.

- The user has performed an end operation - The number of multiple captured images imported has exceeded the end determination number - The RMS (reprojection error) is equal to or less than a threshold value - The difference between the previous and current RMS values is equal to or less than a threshold value - The minimum value of the accuracy improvement effect in the specified region 20 is equal to or greater than a determination value - The proportion of the region in the specified region 20 where the minimum value of the accuracy improvement effect is equal to or greater than a predetermined proportion If it is determined in the above determination that the end condition is not met (S31: NO), the process is executed again from S10. On the other hand, if it is determined in the above determination that the end condition is met (S31: YES), this series of processes is terminated (END).

The present embodiment described above has the following advantages. Here, only the advantages that differ from the first embodiment will be described.

- The processing device 12 of the camera 11 causes the display device 13 to display, superimposed on the specified image 25, the distribution of the effect of the calibration pattern 30 in the multiple captured images captured as images used for calibration on improving the accuracy of the calibration in the specified area 20 (accuracy improvement effect). This allows the user to understand to what extent the accuracy improvement effect of the calibration pattern 30 in the multiple captured images has been exerted on which parts of the specified area 20. This makes it easier for the user to place the calibration pattern 30 in a position where the accuracy improvement effect is insufficient and capture the captured image, thereby assisting the user in determining what kind of captured image should be captured.

The calibration pattern 30 includes a plurality of feature points 30b, and the distribution of the accuracy improvement effect is a distribution of total values at each position, where the position of each feature point 30b is the apex, and the total values decrease the further away from the apex. With this configuration, the distribution of the accuracy improvement effect in the specified region 20 can be easily expressed using a predetermined value that is largest at the position of each feature point 30b and decreases the further away from each feature point 30b. Furthermore, because the distribution of the accuracy improvement effect is expressed as a distribution of total values at each position, the process of determining the distribution of the accuracy improvement effect can be prevented from becoming complicated even when the number of captured images captured is large.

The predetermined value is a value based on each normal distribution with the position of each feature point 30b as its apex. With this configuration, the accuracy improvement effect of each feature point 30b in the predetermined region 20 can be reflected even in positions far from each feature point 30b. Therefore, it becomes easier to quantitatively grasp to what extent the accuracy improvement effect extends to which part of the predetermined region 20.

The specified value is a value greater than 0 throughout the entire specified region 20. This configuration makes it possible to avoid the occurrence of parts of the specified region 20 where the accuracy improvement effect is zero. This makes it easier to quantitatively grasp to what extent the accuracy improvement effect extends to which parts of the specified region 20.

- The position of the calibration pattern 30 where the sum of the total values obtained by adding up the predetermined values at each position within the area corresponding to the calibration pattern 30 is smallest, that is, the position of the calibration pattern 30 where the effect of improving accuracy of the calibration pattern 30 in the already captured captured image is smallest, can be determined as the position at which the calibration pattern 30 should be placed in the specified area 20. Then, the position at which the calibration pattern 30 should be placed in the specified area 20 is displayed by the display device 13, superimposed on the specified image 25. Therefore, the user can easily understand where the calibration pattern 30 should be placed when photographing the calibration pattern 30 with the camera 11.

- The specified area 20 is regarded as a two-dimensional plane, the position of each feature point 30b is regarded as a position on the two-dimensional plane, and the distribution of the accuracy improvement effect is superimposed on the specified image 25 and displayed by the display device 13. With this configuration, even if the calibration of the camera 11 has not been performed, the processing device 12 can grasp and display the distribution of the accuracy improvement effect in the specified area 20 as a two-dimensional plane. This makes it easy for the user to determine what kind of captured image should be captured when performing the calibration.

- The position where the calibration pattern 30 should be placed in the specified area 20 as a two-dimensional plane is displayed by the display device 13, superimposed on the specified image 25. With this configuration, even if the calibration of the camera 11 has not been performed, the processing device 12 can grasp the distribution of the accuracy improvement effect in the specified area 20 as a two-dimensional plane, and can therefore display the position where the calibration pattern 30 should be placed, superimposed on the specified image 25.

- After calibration is performed, the parameters related to the camera 11 obtained by the calibration are used to regard the specified area 20 as a three-dimensional space, the position of each feature point 30b as a position within the three-dimensional space, and the distribution of the accuracy improvement effect is superimposed on the specified image 25 and displayed by the display device 13. With this configuration, the processing device 12 of the camera 11 can accurately grasp and display the distribution of the accuracy improvement effect in the specified area 20 as a three-dimensional space. Therefore, it becomes easy for the user to place the calibration pattern 30 at a position in the specified area 20 as a three-dimensional space where the accuracy improvement effect is insufficient, and capture a captured image.

- The position where the calibration pattern 30 should be placed in the specified area 20 as a three-dimensional space is displayed by the display device 13, superimposed on the specified image 25. With this configuration, when the calibration pattern 30 is photographed by the camera 11, the user can easily understand where in the specified area 20 as a three-dimensional space the calibration pattern 30 should be placed.

The predetermined value is not limited to values based on normal distributions with the position of each feature point 30b as a vertex, but can also be a value based on an exponential distribution or triangular distribution with the position of each feature point 30b as a vertex. In other words, the predetermined value can be a value that decreases the further away from the vertex, with the position of each feature point 30b as a vertex. Furthermore, the calibration pattern 30 is not limited to a configuration in which each feature point 30b is uniformly arranged, but can also be a configuration in which the density of feature points 30b increases closer to the center of the calibration pattern 30. Furthermore, the predetermined value can be a value that decreases the further away from the center of the calibration pattern 30 or the further away from the contour line 31.

The first and second embodiments can also be modified as follows. The same parts as those in the first and second embodiments are given the same reference numerals and will not be described.

- The position of the calibration pattern 30 is not limited to the contours 31 to 33, but it is also possible to display only the corners of the contours 31 to 33, or to display the center of the calibration pattern 30.

-The user can also perform the operations required to capture images to be used for calibration.

- Calibration operations can also be performed by the user.

- The camera 11 (industrial camera) may be fixed to the hand of a robot (industrial machine). In this case, the posture of the robot is controlled so that the camera 11 photographs a predetermined area 20 that has been set in advance, and then the camera 11 is calibrated. In this state, the posture of the robot can be controlled so that the position and angle at which the calibration pattern 30 is photographed are changed.

- The camera 11 may have the functions of the processing device 12, and the camera 11 may constitute an industrial camera calibration operation support device. Also, the display device 13 may have the functions of the processing device 12, and the display device 13 may constitute a camera calibration operation support device.

-The above embodiments and modifications can also be implemented in any suitable combination.

10...system, 11...camera, 12...processing device (calibration operation support device for industrial cameras), 13...display device, 20...predetermined area, 25...predetermined image, 30...calibration pattern, 30b...feature points, 30d...distribution of accuracy improvement effect, 31...contour line (position), 32...contour line (position), 33...contour line (position), 60...target placement position.

Claims (8)

1. A calibration operation support device for an industrial camera, which is applied to a system including a camera that captures an image of a preset predetermined area, a display device that displays a predetermined image that is an image of the predetermined area being captured by the camera, and a processing device that calibrates the camera based on a plurality of captured images of a calibration pattern captured by the camera,
A calibration operation support device for an industrial camera, which displays, on the display device, the position of the calibration pattern in the plurality of captured images captured as images to be used for the calibration, superimposed on the specified image. 2. The calibration operation support device for an industrial camera according to claim 1 , wherein the position at which the calibration pattern is to be placed in the predetermined area is displayed by the display device so as to be superimposed on the predetermined image. 3. The calibration operation support device for an industrial camera according to claim 1, wherein the height of the calibration pattern in the plurality of captured images captured as images used for the calibration is superimposed on the specified image and displayed by the display device. 4. The calibration operation support device for an industrial camera according to claim 1, wherein a height at which the calibration pattern should be disposed in the predetermined area is displayed by the display device so as to be superimposed on the predetermined image. 5. The calibration operation support device for an industrial camera according to claim 1, wherein the inclination of the calibration pattern in the plurality of captured images captured as images used for the calibration is superimposed on the predetermined image and displayed by the display device. 6. The calibration operation support device for an industrial camera according to claim 1, wherein the inclination at which the calibration pattern should be arranged in the specified area is displayed on the display device by superimposing the specified image. 7. The calibration operation support device for an industrial camera according to claim 1 , wherein, when an arrangement state of the calibration pattern in the predetermined area satisfies a predetermined condition, the image captured by the camera is imported as an image to be used for the calibration. 8. The calibration operation support device for an industrial camera according to claim 1, wherein when a number of the plurality of captured images captured as images to be used for the calibration exceeds a predetermined number, the calibration is performed by the processing device.

Images