JP7760864B2 - Imaging system and control method thereof - Google Patents

Description

The present invention relates to technology for imaging transparent objects.

A technology is known for imaging transparent objects using a polarization camera with an imaging element in which polarizers with different transmission axis directions are regularly arranged (see Patent Document 1). This technology is expected to be applied to, for example, the detection and inspection of objects made of glass or transparent resin.

Japanese Patent Application Laid-Open No. 2020-17688

When light is shone on a transparent object, some of the light is specularly reflected from the surface of the transparent object. Because specularly reflected light has polarized properties, it can be observed with a polarization camera, and the surface (reflective surface) of the transparent object can be imaged based on the observation results.

However, polarization cameras are subject to not only specularly reflected light but also light diffusely reflected from the subject and background, which can cause noise. Furthermore, if the distance between the light source and the subject is too great, the specular reflected light may be very weak, making it difficult to extract polarization information. Conversely, if the light source and the subject are too close, the specular reflected light may be too bright, resulting in so-called blown-out highlights (reflected light exceeding the dynamic range of the polarization camera), making it impossible to extract polarization information. However, because the shape and reflective properties of transparent objects, the background conditions, and the relative distance and relative position between the transparent object and the light source vary depending on the equipment placement and subject selection, it is difficult to create a system that can reliably extract polarization information for all subjects and all situations. Therefore, to reliably perform detection and inspection using polarization cameras, it is generally necessary to adjust (finish) the shooting conditions depending on the target subject and situation. However, such adjustments require advanced skills and knowledge, as well as trial and error, and the manual effort involved hinders labor-saving and automation of item detection and inspection. Furthermore, adjusting the shooting conditions can be extremely difficult in some cases. For example, adjusting the brightness of the lighting to suit one part of the field of view can result in overexposure or underexposure in other parts. This type of problem can become more pronounced when there are multiple subjects in the field of view of the polarization camera, and each subject has a different relative distance or position from the lighting.

The present invention was made in consideration of the above-mentioned circumstances, and its purpose is to provide technology that can stably extract polarization information regardless of the subject or situation.

The present invention provides an imaging system for imaging a transparent object, comprising: an illumination device for illuminating the transparent object; a polarization camera having an imaging element in which polarizers with different transmission axis directions are regularly arranged; and a processing device, wherein the processing device has: a control means for controlling either or both of the illumination device and the polarization camera to photograph the transparent object under different shooting conditions; an image acquisition means for acquiring a plurality of original images photographed under different shooting conditions; a polarization extraction means for performing a polarization extraction process on each pixel of each of the plurality of original images to extract polarization information, which is information regarding polarization resulting from specular reflection on the transparent object; and an image generation means for combining the plurality of pieces of polarization information extracted from each of the plurality of original images by the polarization extraction process, and generating a polarization information image in which the polarization information is visualized.

Depending on the arrangement of equipment and the selection of the subject, the shape and reflection characteristics of transparent objects, the background conditions, and the relative distance and relative positional relationship between the transparent object and the lighting will vary. Therefore, with the imaging system of the present invention, the same transparent object is photographed multiple times under different shooting conditions to obtain multiple original images. If any of the shooting conditions match, it can be expected that at least one original image suitable for extracting the polarization information of the transparent object will be obtained. Therefore, by using the polarization information extracted from each of the multiple original images, it is possible to stably extract polarization information regardless of the subject or situation.

The image generation means may perform an operation of selecting, for each pixel, the polarization information with the highest reliability from the multiple pieces of polarization information extracted from each of the multiple original images, and generate the polarization information image using the polarization information selected for each pixel.

A high-quality polarization information image can be generated by collecting reliable polarization information from multiple original images. However, it is not guaranteed that polarization information extracted from the same original image will be selected for every pixel. This is because the shape, orientation, and reflective properties of the reflective surface, the intensity of the illumination light reaching the reflective surface, and the effects of diffuse reflection vary depending on the position within the field of view of the polarization camera. Therefore, by selecting the optimal polarization information for each pixel and using it to generate the final polarization information image, a highly reliable polarization information image can be obtained regardless of the position within the field of view. Furthermore, using this method, it is possible to accurately extract polarization information for all subjects even when there are multiple subjects within the field of view of the polarization camera (or even when multiple subjects with different shapes, reflective properties, or postures are mixed together).

The image generating means may consider the strongest degree of polarization among the multiple pieces of polarization information to be the most reliable polarization information. "Strongest degree of polarization" means that the polarization component is most clearly observed. The stronger the degree of polarization (the more clearly the polarization component is observed), the more reliable the polarization information can be considered. Therefore, by selecting the strongest degree of polarization from the multiple pieces of polarization information obtained for a single pixel, it is possible to obtain the most reliable polarization information for that pixel (i.e., the area on the reflective surface of the transparent object that corresponds to that pixel).

The control means may control either or both of the lighting device and the polarization camera so as to obtain multiple original images with different brightness levels. If the specular reflected light is too weak, it becomes difficult to extract polarization information, and if the specular reflected light is too strong, it becomes impossible to extract polarization information due to overexposure. Therefore, in order to accurately extract polarization information, it is essential to observe specular reflected light of an appropriate intensity. By capturing multiple original images with different brightness levels, as in the present invention, it is possible to observe specular reflected light of an appropriate intensity in one of the original images, and it is expected that reliable polarization information can be extracted from that original image.

The control means may perform control to vary the "illumination intensity of the lighting device" as one of the imaging conditions, or may perform control to vary the "exposure time of the polarization camera" as one of the imaging conditions, or may perform control to vary the "gain of the polarization camera" as one of the imaging conditions. The control means may simultaneously control two or more of the "illumination intensity of the lighting device," "exposure time of the polarization camera," and "gain of the polarization camera."

The control means may control the lighting device to obtain a plurality of original images with different lighting directions for the transparent object. Changing the lighting direction changes the state of diffuse reflection in the background, etc. Therefore, by selecting, from the plurality of original images captured with different lighting directions, the image with the least influence of the diffuse reflection component (i.e., noise) and the clearest polarization component, more reliable polarization information can be extracted.

The lighting device may have multiple light sources that can be turned on independently, and the control means may change the lighting direction relative to the transparent object by switching between the light sources that are turned on. Using such a lighting device allows for quick switching of the lighting direction with simple control, thereby shortening the tact time for photography.

The polarization extraction process may include a noise removal process that removes or reduces noise contained in the calculation results after calculating the polarization information for each pixel of the original image. By performing the noise removal process, high-quality polarization information can be extracted.

The noise removal process may include a process of comparing the polarization information of a target pixel and its surrounding pixels to determine whether the polarization information of the target pixel and its surrounding pixels is similar or dissimilar, and deleting the polarization information of the target pixel if it is determined that the polarization information is dissimilar. This process makes it possible to remove unreliable polarization information.

The noise removal process may include a process of comparing the polarization information of a target pixel and its surrounding pixels to determine whether the polarization information of the target pixel and its surrounding pixels is similar or dissimilar, and if it is determined that the polarization information is similar, replacing the polarization information of the target pixel with the polarization information of a pixel selected from the target pixel and its surrounding pixels. This process smooths the polarization information, further improving the signal-to-noise ratio of the polarization information.

The present invention may also provide a control method for an imaging system comprising an illumination device, a polarization camera having an imaging element in which polarizers with different transmission axis directions are regularly arranged, and a processing device, the control method comprising the steps of: photographing a transparent object under different imaging conditions by controlling either or both of the illumination device and the polarization camera; importing into the processing device multiple original images photographed under different imaging conditions; and executing a polarization extraction process using the processing device to extract, for each pixel of each of the multiple original images, polarization information that is information related to polarization resulting from specular reflection on the transparent object; combining the multiple pieces of polarization information extracted from each of the multiple original images by the polarization extraction process; and generating a polarization information image that visualizes the polarization information.

The present invention may also provide a program for causing a processor to execute each step of the above control method.

The present invention may be understood as an imaging system having at least some of the above means, or as a processing device, control device, or image processing device. The present invention may also be understood as a device that detects or inspects transparent objects using polarization information images obtained by an imaging system. The present invention may also be understood as a control method, image processing method, detection method, or inspection method for an imaging system that includes at least some of the above processing, or as a program for implementing such a method or a recording medium on which that program is recorded. The above means and processing can be combined with each other to the greatest extent possible to constitute the present invention.

This invention makes it possible to stably extract polarization information regardless of the subject or situation.

FIG. 1 is a diagram schematically showing the overall configuration of an imaging system according to an embodiment of the present invention. FIG. 2 is a diagram showing a schematic structure of a polarization camera. FIG. 3A is a diagram schematically illustrating the structure of a lighting device, and FIGS. 3B and 3C are diagrams illustrating other configuration examples of the lighting device. FIG. 4 is a block diagram showing the logical configuration (functional configuration) of the processing device. 5A and 5B are diagrams illustrating specular reflection. FIG. 6 is a diagram illustrating a method for extracting polarization components from an image captured by a polarization camera. FIG. 7 is a flowchart showing an example of the operation of the imaging system. FIG. 8 shows an example of a user interface for inputting image processing parameters. FIG. 9 is a flowchart of the polarization information calculation process. 10A to 10D are diagrams showing examples of neighboring pixels. FIG. 11A is a diagram showing an example of specular reflection at a curved surface portion, and FIG. 11B is a diagram showing an example of selecting a mask size when the range into which polarized light is incident is narrow. FIG. 12 is a diagram showing an example in which a large mask size is suitable. FIG. 13 is a flowchart of the noise removal process. FIG. 14 is a diagram showing an example of a comparison range. FIG. 15 is a flowchart of the image generation process. FIG. 16A is a diagram showing a state in which a syringe placed in a nest is observed by a zenith camera, and FIG. 16B is a diagram for explaining the non-uniformity of the intensity of illumination light depending on the position within the field of view. FIG. 17 shows an example of the difference between the polarization intensity images obtained when the illumination intensity is changed stepwise, and a polarization intensity image obtained by combining the images. 18A and 18B are diagrams for explaining the second embodiment.

First Embodiment
(System configuration)
1 is a schematic diagram showing the overall configuration of an imaging system according to a first embodiment of the present invention. This imaging system 1 is a system for capturing an image of a transparent object using a polarization camera 10, and is used, for example, to detect and inspect transparent objects on a factory production line.

The imaging system 1 mainly comprises a polarization camera 10, an illumination device 11, a processing device 12, and a stage 13. The polarization camera 10 is an imaging means having an imaging element with an array of polarizers. The illumination device 11 is a light source for irradiating illumination light L onto a subject (transparent object) W placed on the stage 13. The processing device 12 is a device that controls the entire imaging system 1 and performs information processing using images captured by the polarization camera 10. The stage 13 is a device for placing or holding the subject W.

The polarization camera 10 is positioned so as to capture an image of the subject W placed on the stage 13 from the zenith direction. In an XYZ coordinate system with the X and Y axes parallel to the stage 13 and the Z axis perpendicular to the stage 13, the optical axis of the polarization camera 10 is parallel to the Z axis. In this system, the objective is to capture light specularly reflected by the subject W with the polarization camera 10, so it is desirable for the angle of incidence of the illumination light L on the subject surface to be as close to the Brewster's angle as possible. Therefore, in this embodiment, the illumination device 11 is positioned at approximately the same height as the stage 13 (in practice, the lower end of the illumination device 11 is positioned at approximately the same height as the upper end of the stage 13 so that the illumination light L is not blocked by the stage 13), and the illumination light L is incident on the subject W from approximately directly beside it (perpendicular to the Z axis). This illumination arrangement is also referred to as low-angle illumination. With the combination of a zenith camera and low-angle illumination as in this embodiment, the polarization camera 10 captures the specular reflection R of the illumination light L, which has an incident angle of approximately 45 degrees on the subject surface. In order to bring the angle of incidence closer to Brewster's angle, a lighting arrangement may be adopted in which illumination light L is applied from a position lower than the subject W, or the optical axis of the polarization camera 10 may be tilted to the opposite side from the lighting device 11.

(polarized camera)
An example of the configuration of the polarization camera 10 will be described with reference to Fig. 2. Fig. 2 is a diagram showing a schematic diagram of the structure of the polarization camera 10.

The polarization camera 10 has a structure that combines an image sensor 20 with a polarizer array 21. The image sensor 20 is a device in which photoelectric conversion elements (also called pixels), such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor, are arranged two-dimensionally, and is also called an image sensor. The polarizer array 21 is a device in which numerous polarizers are arranged two-dimensionally, and the position and size are designed so that one polarizer 210 corresponds to one pixel (light receiving element) 200 of the image sensor 20. The polarizer 210 is an optical element that has the property of only passing linearly polarized light in a specific direction (the vibration direction of the linearly polarized light transmitted by the polarizer 210 is called the transmission axis direction of the polarizer 210). The polarizer array 21 of this embodiment has a structure in which polarizers 210 with four different transmission axis directions (0 degrees, 45 degrees, 90 degrees, and 135 degrees) are regularly arranged, as shown in Figure 2. Specifically, the array pattern is such that linearly polarized light components with different transmission axis directions (0 degrees, 45 degrees, 90 degrees, 135 degrees) are incident on four 2x2 pixels of the image sensor 20.

The resolution (pixel pitch) and size (number of pixels) of the image sensor 20 and polarizer array 21 can be designed appropriately depending on the subject and the intended use of the image. The polarizer array 21 can be realized using a wire grid, photonic crystal, or other methods, but either method may be used. Also, while Figure 2 shows four types of polarizers with different transmission axis directions combined into four 2x2 pixels, other variations in the transmission axis directions and arrangement of the polarizers may be used.

(Lighting equipment)
An example of the configuration of the illumination device 11 will be described with reference to Fig. 3A. Fig. 3A is a diagram schematically illustrating the structure of the illumination device 11 of this embodiment, showing the state when the stage 13 is viewed from the polarization camera 10 side.

The lighting device 11 is composed of four rod-shaped lights 30 arranged to surround the stage 13. When all four rod-shaped lights 30 are turned on simultaneously, the subject W on the stage 13 can be illuminated from all four directions (positive X direction, negative X direction, positive Y direction, negative Y direction). Furthermore, the lighting direction can be switched by selectively turning on only one of the rod-shaped lights 30. The rod-shaped lights 30 are composed, for example, of multiple LED light sources arranged on a substrate and a diffuser plate arranged to cover them.

The structure of the illumination device 11 is not limited to that shown in FIG. 3A . For example, a circular illumination device 11 as shown in FIG. 3B may be used. Even with this structure, the subject W can be simultaneously illuminated from all directions. Furthermore, the illumination direction can be arbitrarily switched by selectively turning on the LED light sources. Instead of omnidirectional illumination, a configuration in which the subject W is illuminated only from a specific direction, as shown in FIG. 3C , may be used. For example, if the range of the subject W is limited or the angle of the subject surface to be detected is known or limited, it is sufficient to illuminate from the required direction. Alternatively, even when the illumination device 11 as shown in FIG. 3C is used, omnidirectional imaging may be performed by changing the relative position of the illumination device 11 and the subject W (e.g., by rotating the subject W on a stage).

(Processing device)
An example of the configuration of the processing device 12 will be described with reference to Fig. 4. Fig. 4 is a block diagram showing the logical configuration (functional configuration) of the processing device 12 of this embodiment.

The processing device 12 mainly comprises a camera control unit 40, an illumination control unit 41, and an image processing unit 42. The camera control unit 40 controls the polarization camera 10. For example, the camera control unit 40 controls the imaging conditions of the polarization camera 10 (exposure time, gain, etc.), captures image data from the polarization camera 10, and calibrates the polarization camera 10. The illumination control unit 41 controls the illumination device 11. For example, the illumination control unit 41 controls the illumination conditions (emission intensity, emission time, etc.) and controls the on/off of each light source. The image processing unit 42 processes the images captured from the polarization camera 10.

The image processing unit 42 of this embodiment has functions such as an image acquisition unit 420, a polarization extraction unit 421, a polarization information image generation unit 422, a display unit 423, and a parameter reception unit 424. The image acquisition unit 420 acquires an image captured by the polarization camera 10 (hereinafter also referred to as the "original image") via the camera control unit 40. The polarization extraction unit 421 performs polarization extraction processing to extract polarization information from the original image. Polarization information is information related to polarization resulting from specular reflection on a transparent object. The polarization information image generation unit 422 generates an image representing polarization information (hereinafter also referred to as the "polarization information image") based on the extraction results of the polarization extraction processing. The display unit 423 performs processing to display the generated polarization information image on a display device. The parameter reception unit 424 has a function to accept changes to various condition settings (parameters) that determine the operation of the imaging system 1. Details of each process of the image processing unit 42 will be described later.

The processing device 12 may be configured, for example, by a computer including a processor such as a CPU or GPU, a memory serving as a main memory, a storage serving as an auxiliary memory, a display device, an input device such as a mouse or a touch panel, a network I/F, etc. The computer may be a general-purpose computer such as a personal computer, a tablet terminal, a smartphone, or a field computer, an embedded computer, a dedicated device, or a device that utilizes computer resources of another device such as the polarization camera 10 or a programmable logic controller (PLC). The configuration shown in FIG. 4 is realized by loading a program stored in a storage device or the like into memory and executing it with a processor. However, part or all of the configuration shown in FIG. 4 may be configured with a dedicated device such as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA). Furthermore, part of the configuration or part of the processing shown in FIG. 4 may be executed by another device using cloud computing or distributed computing.

(Transparent object detection using a polarization camera)
This section explains the basic principles of transparent object detection using a polarization camera.

As shown in FIG. 5A , when light 50 is obliquely incident on a transparent object W, such as glass or transparent resin, most of the incident light 50 becomes refracted light 51 and passes through the transparent object W, while a portion of the incident light 50 is specularly reflected at the surface (interface) of the transparent object W and becomes reflected light 52. This specular reflection results in a difference in reflectance between the p-wave (polarized component whose electric field vibration direction is parallel to the plane of incidence) and the s-wave (polarized component whose electric field vibration direction is perpendicular to the plane of incidence (i.e., parallel to the reflecting surface)) contained in the incident light 50. FIG. 5B , with the horizontal axis representing the angle of incidence and the vertical axis representing the reflectance, shows an example of the difference in reflectance between the s-wave and the p-wave. As shown in this example, the reflectance of the s-wave monotonically increases with the angle of incidence α, while the reflectance of the p-wave gradually decreases as the angle of incidence α increases from 0 degrees, reaching zero at a certain angle (called the Brewster angle). Therefore, the reflected light 52 observed in the case of specular reflection is dominated by the s-wave, i.e., the polarized component vibrating in a direction parallel to the surface of the transparent object W, which is the reflecting surface. By utilizing this property and capturing polarized light with a polarization camera, the surface (reflective surface) of the transparent object W can be imaged.

By the way, not only polarized light due to specular reflection but also light diffusely reflected from the subject and background enters a polarization camera. When imaging transparent objects, the diffusely reflected light component is unnecessary and can cause artifacts, so it is first necessary to extract only the polarized light component from the polarization camera image. To achieve this, this embodiment utilizes the difference in transmission characteristics caused by polarizers.

Referring to Figure 6, we will explain how to extract polarization components from images captured by a polarization camera. The intensity of the polarization component received by the light receiving element is maximum when the polarization direction θ and the transmission axis direction of the polarizer are aligned. It decreases as the transmission axis direction deviates from the polarization direction θ, and is minimum when the transmission axis direction and the polarization direction θ are perpendicular. In other words, as the transmission axis direction of the polarizer is changed from 0 degrees to 180 degrees, the intensity of the polarization component received by the light receiving element changes like a sine wave. On the other hand, the intensity of the diffusely reflected light component contained in the incident light remains constant, regardless of the transmission axis direction of the polarizer. Therefore, it is possible to extract only information about the polarization component contained in the incident light based on the change in light intensity when observed through multiple types of polarizers with different transmission axis directions (transmission axis direction dependence).

For example, as shown in Figure 6, consider a case in which incident light containing a polarization component with a polarization direction θ is observed through four polarizers 210 with transmission axis directions of 0 degrees, 45 degrees, 90 degrees, and 135 degrees. In this case, the values of the pixels 200 corresponding to each polarizer 210 (corresponding to the light intensity of the transmitted light) differ depending on the transmission axis direction. By fitting a sine wave to these four pixel values, the polarization direction θ and polarization intensity of the polarization component can be estimated. In Figure 6, the angle at which the fitting curve is maximized represents the polarization direction θ of the polarization component, and the amplitude of the fitting curve represents the polarization intensity (also called the degree of polarization). By applying this polarization extraction process to each pixel of the image obtained by the polarization camera 10, information (such as polarization direction and polarization intensity) about the polarized light specularly reflected by a transparent object W present within the field of view of the polarization camera 10 can be captured.

The polarization extraction process performed by the polarization extraction unit 421 in this embodiment also follows the same basic principle of extracting polarization information based on changes in light intensity (transmission axis direction dependency) when observed through multiple types of polarizers with different transmission axis directions. However, to further improve accuracy and usability, unique processing such as selection of mask size (a parameter that defines the range of neighboring pixels to be referenced when calculating polarization) and noise removal has been added. Details will be provided later.

(Example of operation of imaging system)
An example of the operation of the imaging system 1 will be described with reference to the flowchart of FIG.

In step S70, a transparent object that will serve as subject W is placed on stage 13. The subject W may be transported and positioned using a robot or transport device, or a worker may set the subject W on stage 13.

In step S71, the lighting control unit 41 of the processing device 12 sets the lighting conditions for the lighting device 11. In this embodiment, the lighting device 11 can switch the lighting intensity between four levels, 1 to 4, and the lighting control unit 41 first sets the lighting intensity to 1 (the darkest state).

In step S72, the subject W is photographed. Specifically, the lighting control unit 41 turns on the lighting device 11 according to the given lighting conditions, and irradiates the subject W with illumination light. Then, with the subject W illuminated, the camera control unit 40 controls the polarization camera 10 according to the given imaging conditions to photograph the subject.

In step S73, the image acquisition unit 420 of the processing device 12 captures the image (original image) captured in step S72 from the polarization camera 10. The captured original image data is stored in memory or storage and is then processed by the image processing unit 42.

In step S74, the processing device 12 checks whether image capture under all shooting conditions has been completed. If not, the process returns to step S71 and performs image capture under the next shooting condition. That is, in step S71, the lighting control unit 41 changes the illumination intensity of the lighting device 11 to 2, in step S72, the subject W is captured again, and in step S73, an original image corresponding to illumination intensity 2 is captured. In this embodiment, the processes of steps S71 to S74 are repeated four times, and four original images of the same subject W with different brightnesses are captured by the processing device 12. When image capture under all shooting conditions has been completed, the process proceeds to step S75.

In step S75, the user inputs the image processing parameters to be used by the image processing unit 42 using the user interface provided by the parameter receiving unit 424. Note that if there is no need to change the image processing parameters, the processing of step S74 may be skipped. Furthermore, the image processing parameters may be set before capturing an image.

Figure 8 shows an example of a user interface (UI) for inputting image processing parameters provided by the parameter receiving unit 424. This UI screen consists of a mask size setting 80, a noise reduction setting 81, a display image setting 82, a full image display area 83, a magnified image display area 84, and a magnification setting 85. The mask size setting 80 is a UI for setting the "mask size," which defines the range of neighboring pixels to be referenced when calculating polarization information from the original image. In the example of Figure 8, the user is prompted to select a mask size from 2x2, 3x3, 4x4, and 5x5; however, the user may also be prompted to input an arbitrary size. The noise reduction setting 81 is a UI for setting conditions for the "noise reduction process," which removes or reduces noise contained in the polarization information calculation results. In the example of Figure 8, the user can select whether or not to apply noise reduction. If "yes," the user can also input parameters to be used in the noise reduction process (comparison range, polarization angle difference threshold, polarization intensity difference threshold).

The display image settings 82 are a UI for setting the type of image to be displayed in the full image display area 83 and the enlarged image display area 84. In the example of Figure 8, the original image, polarization intensity image, polarization direction image, and brightness image can be selected. The full image display area 83 is an area where the full image (low resolution) is displayed, and the enlarged image display area 84 is an area where a portion of the image (the portion indicated by the frame 86) is displayed at high resolution. The enlargement settings 85 are a UI for setting the magnification and range of the enlarged image to be displayed in the enlarged image display area 84.

In step S76, the polarization extraction unit 421 calculates polarization information from the multiple original images captured in step S73. Details of the polarization information calculation process will be described later.

In step S77, the polarization extraction unit 421 applies noise removal processing to the calculation results of step S76. Note that if noise removal is set to "No" in the UI of FIG. 8, the processing of step S77 is skipped. Details of the noise removal processing will be described later.

In step S78, the polarization information image generation unit 422 combines the multiple pieces of polarization information extracted from the multiple original images by the polarization extraction unit 421 to generate a polarization information image. Details of the image generation process will be described later. In this embodiment, three types of images are generated as polarization information images: a polarization intensity image, a polarization direction image, and a brightness image. The polarization intensity image is an image that represents the polarization intensity (degree of polarization) using shades, and the polarization direction image is an image that represents the polarization direction using shades or pseudo-colors. The brightness image is an image that represents the light intensity of the polarized component using shades.

In step S79, the display unit 423 displays the image selected in the display image settings 82 of the UI in Figure 8 in the full image display area 83 and enlarged image display area 84.

Here, when the user changes the display image setting 82 on the UI of Figure 8, the images displayed in the full image display area 83 and the enlarged image display area 84 change accordingly. Using this UI allows the user to visually compare the appearance of the original image with the polarization information (polarization intensity, polarization direction, brightness), making it easy to, for example, check whether the polarization information calculation and noise removal have been performed appropriately, or check the shape and state of transparent objects. Note that if the mask size setting 80 or noise removal setting 81 is changed on the UI of Figure 8, steps S76 to S79 may be executed again. Using such a UI allows the user to fine-tune image processing parameters (mask size, whether or not to perform noise removal, comparison range, threshold, etc.) while checking the calculation results (polarization information image).

In this embodiment, the lighting conditions of the lighting device 11 were changed as the shooting conditions, but images with different brightness levels can also be captured by changing the imaging conditions of the polarization camera 10. For example, the camera control unit 40 of the processing device 12 may control the exposure time of the polarization camera 10. The longer the exposure time, the greater the amount of light received by the imaging element, and therefore brighter images can be captured. Alternatively, the camera control unit 40 may control the gain of the polarization camera 10. Gain is the amplification factor of the imaging element, and the higher the gain, the brighter the image obtained. Of course, it is also possible to control both the lighting conditions of the lighting device 11 and the imaging conditions of the polarization camera 10.

(Polarization information calculation process)
An example implementation of the polarization information calculation process (step S76 in FIG. 7 ) of the polarization extraction unit 421 will be described with reference to FIG. 9 and FIGS. 10A to 10D. FIG. 9 is a flowchart of the polarization information calculation process, and FIGS. 10A to 10D are diagrams showing examples of neighboring pixels. Here, the size (number of pixels) of the original image is assumed to be M rows and N columns, the pixel in row i and column j is represented as (i, j), and the pixel value (light intensity) of that pixel is represented as I(i, j) (i = 1, 2, ..., M, j = 1, 2, ..., N). FIGS. 10A to 10D show enlarged portions of the original image, with each rectangle representing a pixel, the arrow within the pixel indicating the transmission axis direction, and the symbols in the margins representing the pixel's row number and column number.

In step S10, the polarization extraction unit 421 selects one original image to be processed from among multiple original images captured under different shooting conditions in steps S71 to S74 of FIG. 7. In this embodiment, in step S10, the original images captured at illumination intensity 1, illumination intensity 2, illumination intensity 3, and illumination intensity 4 are selected in that order.

In step S11, the polarization extraction unit 421 selects a pixel of interest from the original image to be used for calculation. In step S11, the pixels of interest are selected in order from (1,1) to (M,N).

In step S12, the polarization extraction unit 421 selects neighboring pixels to be referenced when calculating polarization information for the pixel of interest. The range of neighboring pixels to be selected here is determined by the mask size setting 80 in FIG.
The mask size is determined by the mask size specified in . First, an example of the default mask size of 2 × 2 will be explained. When the mask size is 2 × 2, as shown in FIG. 10A, three pixels (i-1, j-1), (i-1, j), and (i, j-1) adjacent to the pixel of interest (i, j) are selected as neighboring pixels. Note that if the pixel of interest is on the boundary of the original image and there are no neighboring pixels, the processing of steps S12 and S13 may be skipped, or an appropriate value may be padded.

In step S13, the polarization extraction unit 421 calculates polarization information for the pixel of interest based on the light intensities (pixel values) of the pixel of interest and its neighboring pixels. If the light intensities when the transmission axis directions are 0 degrees, 45 degrees, 90 degrees, and 135 degrees are I1, I2, I3, and I4, respectively, the brightness I0, polarization direction θ, and polarization intensity D of the pixel of interest can be calculated using the following formulas.

In the example of FIG. 10A, the transmission axis directions corresponding to the four pixels (i-1, j-1), (i-1, j), (i, j-1), and (i, j) are 90 degrees, 135 degrees, 45 degrees, and 0 degrees, respectively, and therefore I1 to I4 in equations (1) to (3) are given by the following equations.
I1 = I(i, j)
I2=I(i,j-1)
I3=I(i-1,j-1)
I4=I(i-1,j)

The brightness I0 in equation (1) is an index representing the intensity of the incident light entering the pixel of interest. The polarization direction θ in equation (2) represents the principal axis direction of the polarization component contained in the incident light entering the pixel of interest. The polarization intensity D in equation (3) is an index representing the degree of polarization of the polarization component contained in the incident light entering the pixel of interest.

The above formulas (1) to (3) are merely examples, and the indices corresponding to the brightness, polarization direction, and polarization intensity may be calculated using other formulas. For example, the coefficients in formulas (1) to (3) may be replaced with other coefficients, or formula (3) may be replaced with the following formula (4):

In step S14, the polarization extraction unit 421 checks whether the calculation of polarization information has been completed for all pixels in the original image (i.e., whether i = M and j = N), and if there are any unprocessed pixels, it returns to step S11 and performs processing on the next pixel of interest.

In step S15, the polarization extraction unit 421 checks whether the polarization information calculation process has been completed for all original images, and if there are any unprocessed original images, it returns to step S10 to process the next original image.

Three items of polarization information (polarization direction θ, polarization intensity D, and brightness I0) are calculated for every pixel in every original image through the above-described process. The results of the polarization information calculation process are saved in memory or storage in a data format that associates the coordinates (i,j) of each pixel with the polarization direction θ(i,j), polarization intensity D(i,j), and brightness I0(i,j), and are then used for subsequent processing.

Next, we will explain a processing example when the mask size is larger than 2x2, with reference to Figures 10B to 10D.

As shown in Figure 10B, when the mask size is 3x3, in step S12, the eight pixels (i-1,j-1), (i-1,j), (i-1,j+1), (i,j-1), (i,j+1), (i+1,j-1), (i+1,j), and (i+1,j+1) adjacent to the pixel of interest (i,j) are selected as neighboring pixels. In the example of Figure 10B, the nine pixel group consisting of the pixel of interest and its neighboring pixels includes one pixel with a transmission axis direction of 0 degrees, two pixels with a transmission axis direction of 45 degrees, two pixels with a transmission axis direction of 135 degrees, and four pixels with a transmission axis direction of 90 degrees. When there are multiple pixels with the same transmission axis direction, the calculation in step S13 simply involves substituting representative values for the pixels in each transmission axis direction into I1 to I4 in equations (1) to (3).

For example, when the average value is used as the representative value, I1 to I4 are given by the following formula.
I1 = I(i, j)
I2={I(i,j-1)+I(i,j+1)}/2
I3={I(i-1,j-1)+I(i-1,j+1)+I(i+1,j-1)+I(i+1,j+1)}/4
I4={I(i,j-1)+I(i,j+1)}/2

In addition to the average value, the mode, median, maximum value, minimum value, etc. may also be used as the representative value. Alternatively, instead of using the representative value, I1 to I4 may be estimated by fitting a sine wave to the values of the nine pixels, or the brightness I0, polarization direction θ, and polarization intensity D may be calculated directly.

Figure 10C shows an example where the mask size is 4x4. When the mask size is 3x3, the range selected is the eight neighboring pixels plus seven more pixels: (i-2, j-2), (i-2, j-1), (i-2, j), (i-2, j+1), (i-1, j-2), (i, j-2), (i+1, j-2). In the example of Figure 10C, the selected range includes four pixels each at 0 degrees, 45 degrees, 90 degrees, and 135 degrees. In this case, as with the 3x3 case, you can either substitute representative values for each transmission axis direction for I1 to I4, or perform sine wave fitting on the values of the 16 pixels.

Figure 10D shows an example with a 5x5 mask size. The range selected is the 15 neighboring pixels from a 4x4 mask size plus an additional nine pixels: (i-2, j+2), (i-1, j+2), (i, j+2), (i+1, j+2), (i+2, j-2), (i+2, j-1), (i+2, j), (i+2, j+1), (i+2, j+2). In the example of Figure 10D, the selected range includes nine 0-degree pixels, six 45-degree pixels, six 135-degree pixels, and four 90-degree pixels. In this case, as with the 3x3 case, you can substitute representative values for I1 to I4 for each transmission axis direction, or perform sine wave fitting on the values of the 25 pixels.

In this embodiment, the mask size, i.e., the pixel range to be referenced when calculating polarization information, can be changed as desired. The user can select an appropriate mask size depending on the shape and condition of the subject, the purpose of the system, etc.

For example, as shown in Figure 11A, when the polarization camera 10 captures specular reflection from a curved portion of the subject W, the reflecting surface is extremely small, so the range in which polarized light is incident may be a narrow area of just a few pixels wide (depending on the resolution of the polarization camera 10). In such a case, if the mask size is too large, as shown in Figure 11B, pixels in which no polarized light is incident may also be referenced, which may reduce the detectability of polarization information. Therefore, it is preferable to select a mask size that is smaller than the range in which polarized light is incident. This allows for appropriate extraction of polarization information in a narrow area.

On the other hand, as shown in Figure 12, when inspecting for surface contamination or foreign matter, due to the reflective characteristics of a flat surface (specular reflection, scattering strength), it is expected that similar polarization characteristics will appear over a relatively wide area of the image. In such cases, if the mask size is too small, the inspection may be sensitive to local differences in reflectance, making the inspection unstable. Therefore, it is preferable to set a somewhat larger mask size to average out disturbing factors such as local differences in reflectance. This can improve the stability of the inspection.

In this embodiment, four mask sizes, 2x2, 3x3, 4x4, and 5x5, are shown as examples, but the mask sizes are not limited to these. For example, a mask larger than 5x5 may be used, or a mask with a shape other than a square may be used. Furthermore, the user may be allowed to design the size and shape of the mask as desired.

(Noise removal processing)
The polarization information calculation process described above utilizes the principle of detecting polarization by regarding local pixel value differences as differences in transmission due to differences in the direction of the polarizer's transmission axis. Therefore, for example, when a high-spatial-frequency brightness change occurs in an edge or texture area of a subject, this may be mistaken for a pixel value difference due to polarization, resulting in a false detection of polarization. The noise removal process is a process for removing or reducing noise due to such false detection from the calculation results of the polarization information calculation process.

An example implementation of the noise removal process (step S77 in Figure 7) will be described with reference to Figures 13, 14A, and 14B. Figure 13 is a flowchart of the noise removal process, and Figures 14A and 14B are diagrams showing examples of comparison ranges.

In step S20, the polarization extraction unit 421 reads the parameters set in the noise removal setting 81 in Figure 8. Here, four parameters are read: noise removal on/off, comparison range, polarization angle difference threshold, and polarization intensity difference threshold. If noise removal is set to "off," subsequent processing is skipped (step S21). If noise removal is set to "on," proceed to step S22.

In step S22, the polarization extraction unit 421 reads the calculation results of the polarization information calculation process from memory or storage. Here, data is obtained that associates the coordinates (i, j) of each pixel with the polarization direction θ(i, j), polarization intensity D(i, j), and brightness I0(i, j) extracted from each of the multiple original images.

In step S23, the polarization extraction unit 421 selects one original image to be processed from the plurality of original images. In this embodiment, the original images captured at illumination intensity 1, illumination intensity 2, illumination intensity 3, and illumination intensity 4 are selected in this order.

In step S24, the polarization extraction unit 421 selects the target pixel to be processed. In step S24, the target pixels are selected in order from (1,1) to (M,N).

In step S25, the polarization extraction unit 421 compares the polarization directions of the target pixel and its surrounding pixels to determine whether they are similar. As shown in FIG. 14A, when the comparison range is set to "1," a comparison is made between the target pixel (i, j) and the three surrounding pixels (i, j+1), (i+1, j), and (i+1, j+1). Also, as shown in FIG. 14B, when the comparison range is set to "2," a comparison is made between the target pixel (i, j) and the eight surrounding pixels (i-1, j-1), (i-1, j), (i-1, j+1), (i, j-1), (i, j+1), (i+1, j-1), (i+1, j), and (i+1, j+1). If even one combination is found in which the difference in polarization direction is greater than the polarization angle difference threshold, the combination is determined to be "dissimilar," and the process proceeds to step S27. On the other hand, if the combination is determined to be "similar," the process proceeds to step S26.

In step S26, the polarization extraction unit 421 compares the polarization intensity between the target pixel and its surrounding pixels to determine whether the polarization intensities are similar. The method for selecting the comparison range is as shown in Figures 14A and 14B. If even one combination is found in which the difference in polarization intensity is greater than the polarization intensity difference threshold, it is determined to be "dissimilar" and the process proceeds to step S27. On the other hand, if it is determined to be "similar," the process proceeds to step S28.

If either the polarization direction or polarization intensity is "dissimilar," the polarization information of the target pixel is an unusual value (outlier) compared to the surrounding pixels, and is likely to be noise. Therefore, in step S27, the polarization extraction unit 421 deletes the polarization information of the target pixel (polarization direction θ and polarization intensity D).

If both the polarization direction and polarization intensity are "similar," in step S28, the polarization extraction unit 421 selects the pixel with the greatest polarization intensity among the target pixel and the pixels in the comparison range, and replaces the polarization direction and polarization intensity of the target pixel with the polarization direction and polarization intensity of the selected pixel. This process achieves a smoothing effect that reduces local variations in the polarization direction and polarization intensity.

In step S29, the polarization extraction unit 421 checks whether noise removal processing has been completed for all pixels, and if there are any unprocessed pixels, returns to step S24 to perform processing on the next target pixel.

In step S30, the polarization extraction unit 421 checks whether noise removal processing has been completed for all original images, and if there are unprocessed original images, returns to step S23 to perform processing on the next original image.

The above-described process removes noise (outliers) and smooths the polarization direction θ and polarization intensity D, which are the results of the polarization information calculation process. The corrected data is then overwritten and saved in memory or storage.

(Image generation processing)
An example of implementation of the image generation process (step S78 in FIG. 7) will be described with reference to the flowchart in FIG.

In step S40, the polarization information image generating unit 422 first selects a target pixel (i
, j) is selected. Each time the loop of steps S40 to S45 is repeated, a target pixel is selected in order from (1,1) to (M,N). The polarization information image generator 422 then obtains polarization information extracted from each of the multiple original images for the target pixel (i,j). Here, if the polarization information extracted from pixel (i,j) of the kth original image is denoted as PIk(i,j), in this example, polarization information PI1(i,j) to PI4(i,j) corresponding to each of the four original images captured at illumination intensities 1 to 4 is read from memory. Note that the polarization information PIk(i,j) includes three items of information: polarization intensity Dk(i,j), polarization direction θk(i,j), and brightness I0k(i,j) (the subscript k represents the original image number).

In step S41, the polarization information image generation unit 422 compares the multiple pieces of polarization information PI1(i,j) through PI4(i,j) acquired in step S40, and considers the polarization information with the strongest polarization intensity Dk(i,j) to be the most reliable polarization information and adopts it as the polarization information for the target pixel (i,j). Polarization intensity is an index that represents the degree of polarization; the higher this value, the more clearly the polarization component is observed. Therefore, by adopting the piece with the strongest polarization intensity from the multiple pieces of polarization information PI1(i,j) through PI4(i,j) acquired under different shooting conditions, the most reliable polarization information for pixel (i,j) (i.e., the area on the reflective surface of the transparent object corresponding to pixel (i,j)) can be obtained. The following explanation will continue assuming that PIN(i,j), the polarization information for the nth original image, is adopted.

In step S42, the polarization information image generation unit 422 writes the polarization intensity value Dn(i,j) of the polarization information PIn(i,j) selected in step S41 to pixel (i,j) of the polarization intensity image. Similarly, in step S43, the polarization information image generation unit 422 writes the polarization direction value θn(i,j) of the polarization information PIn(i,j) selected in step S41 to pixel (i,j) of the polarization direction image. Also, in step S44, the polarization information image generation unit 422 writes the brightness value I0n(i,j) of the polarization information PIn(i,j) selected in step S41 to pixel (i,j) of the brightness image.

By performing the above-described steps S40 to S44 for each pixel (step S45), a polarization information image can be obtained by combining the good portions (i.e., highly reliable polarization information) of the multiple polarization information PI1 to PI4. In the above embodiment, the polarization intensity D itself is used as an index representing the reliability of the polarization information, but the reliability of the polarization information may be calculated using a different index. For example, the index calculated by (I3-I1) ² + (I2-I4) ² (an index corresponding to the contrast strength of the four pixels) may be used as the reliability of the polarization information. Furthermore, because dark areas of an image are generally more susceptible to noise, the brightness of the image may also be taken into consideration. For example, if MAX(I1, I2, I3, I4) is equal to or less than a threshold value (i.e., in the case of a dark area of the image), it is assumed that there is no polarization, and the reliability may be set to 0 (the minimum value).

The advantages of the polarization information image of this embodiment will be explained using an example in which the above-described imaging system 1 is applied to detection and inspection in the manufacturing process of syringes filled with drug solutions.

In the manufacturing process of pre-filled syringes, robots are typically used to pick and place the syringes and inject the liquid. If a syringe is in an incorrect position or lying on its side, it could be damaged or the manufacturing equipment could malfunction. Therefore, a system is needed to monitor the syringe's position and orientation, and output an error or stop the equipment if a problem occurs.

16A shows a state in which syringes 101 placed in nest 100 are observed by polarization camera 10 installed at the zenith. Two syringes 101b are not inserted into the holes in nest 100 and are lying on their side. However, because syringes 101 are colorless and transparent, even if they are photographed using, for example, normal transmitted illumination and an optical camera, the boundary between syringe 101 and the background (white nest 100) is barely discernible in the image. Therefore, it is difficult to detect the lying syringes 101b in a normal optical image.

In the imaging system 1 of this embodiment, as shown in Figure 16B, low-angle lighting devices 11 (light sources 11r, 11l, 11t, and 11b) are installed around the nest 100, and the polarization camera 10 captures the specular reflection (polarized light) on the surface of the syringe 101. The orientation of the reflective surface of the syringe 101 (the orientation in the XY plane of the normal to the reflective surface) can be any angle between 0 and 360 degrees, but by illuminating the syringe 101 from all sides as shown in Figure 16B, the polarization camera 10 can observe the specular reflection from the reflective surface in any orientation.

However, when comparing syringe 101r on the right side of the field of view with syringe 101l on the left side, for example, the intensity of the specularly reflected light observed from each syringe 101r, 101l will not be the same. This is because light from light source 11r, located to the right of nest 100, is incident on the right-facing reflective surface of each syringe 101r, 101l, but there is a difference in the intensity of the illumination light incident on the reflective surface between syringe 101r located closer to light source 11r and syringe 101l located farther from light source 11r. Therefore, it is not possible to capture all subjects within the field of view of polarization camera 10 under completely uniform lighting conditions. If the illumination intensity of light source 11r is set to match the proximal syringe 101r, the polarized component in the distal syringe 101l may be weak, making it difficult to extract polarization information. Conversely, if the illumination intensity of light source 11r is set to match the distal syringe 101l, overexposure may occur in the proximal syringe 101r, making it impossible to extract polarization information.

Accordingly, imaging system 1 measures polarization multiple times under different imaging conditions. Images 104a-104d in Figure 17 are schematic examples of polarization intensity images obtained when the illumination intensity is gradually increased. It can be seen that the optimal illumination intensity varies depending on the position within the field of view. By collecting and combining the portions of these images 104a-104d with high polarization intensity, it is possible to obtain a high-quality polarization information image 105 in which the polarization information of all subjects present within the field of view has been appropriately extracted. In polarization information image 105, a clearly circular image appears for syringe 101a in the correct position, and a clearly linear band-shaped image appears for syringe 101b that is lying on its side. Using this polarization information image 105 makes it possible to stably inspect the position and orientation of syringes and detect syringes in an incorrect position or orientation.

Computer-based image recognition technology may be used to detect syringes from the polarization information image 105 and determine their posture. For example, a syringe in the correct posture will exhibit specular reflection from the inner circle and the flange surrounding it. By focusing on this image feature and detecting the shape of the circle and flange from the polarization information image using a method such as pattern matching, it is possible to automatically recognize syringes in the correct posture. In the case of a syringe lying on its side, for example, first, a set of adjacent pixels (pixel region) within a specific polarization direction range (e.g., 0° to 60°) is extracted, and then pixel regions with widths and lengths greater than or equal to a threshold are detected from the extracted pixel region. This type of processing makes it possible to automatically recognize the strip-shaped pixel region of a syringe lying on its side.

In the above embodiment, multiple images were taken while switching the lighting conditions (lighting intensity) of the lighting device 11. However, multiple images may be taken while switching the imaging conditions (exposure conditions or gain) of the polarization camera 10, or control may be exercised to switch both the lighting conditions of the lighting device 11 and the imaging conditions of the polarization camera 10. In other words, as long as multiple original images with different brightnesses can be obtained, any method for changing the imaging conditions may be used.

Second Embodiment
In the first embodiment, the subject was photographed while illuminated from all directions. This method has the advantage of being able to capture reflective surfaces in all directions within the field of view in a single shot, thereby shortening the time required to capture the original image. However, when observing the specular reflection of illumination light from one direction, the method of simultaneous illumination from all directions may result in noise due to diffuse reflection of illumination light from other directions. Figure 18A shows an example. Light incident from light source 11r is specularly reflected from the slope on the right side of subject W and observed by polarization camera 10. If strong light is incident from light source 11l in another direction, the diffuse reflection light in the background passes through the transparent subject W and is observed together with the specular reflection light. For example, in a situation where light source 11r is far from subject W and light source 11l is close to subject W, the strong diffuse reflection light may drown out the weak specular reflection light, significantly reducing the accuracy of extracting polarization information.

In order to reduce this type of diffuse reflection noise, in the second embodiment, multiple shots are taken while switching the illumination direction relative to the subject W. For example, as shown in Figure 18B, the first shot is taken with only light source 11l turned on, and then the second shot is taken with only light source 11r turned on. This makes it possible to observe the diffuse reflection of the illumination light from light source 11l and the specular reflection of the illumination light from light source 11r separately. This reduces the situation in which the weak specular reflection is drowned out by the strong diffuse reflection.

Now, with reference to the flowchart in Figure 7, we will explain an example of the operation of the imaging system 1 of the second embodiment.

In step S70, a transparent object that will serve as subject W is placed on stage 13. The subject W may be transported and positioned using a robot or transport device, or a worker may set the subject W on stage 13.

In step S71, the lighting control unit 41 of the processing device 12 sets the lighting conditions (lighting direction) of the lighting device 11. In this embodiment, the lighting device 11 has four rod-shaped lights (light sources) 30 arranged to surround the stage 13, and each rod-shaped light 30 can be independently switched on/off and its lighting intensity can be controlled. The lighting control unit 41 sets the lighting conditions so that only one of the four rod-shaped lights 30 is turned on and the other three are turned off.

In step S72, the subject W is photographed. Specifically, the lighting control unit 41 turns on the lighting device 11 according to the given lighting conditions, and irradiates the subject W with illumination light. At this time, the subject W on the stage 13 is illuminated from only one direction. Then, with the subject W illuminated, the camera control unit 40 controls the polarization camera 10 according to the given imaging conditions to photograph the subject W.

In step S73, the image acquisition unit 420 of the processing device 12 captures the image (original image) captured in step S72 from the polarization camera 10. The captured original image data is stored in memory or storage and is then processed by the image processing unit 42.

In step S74, the processing device 12 checks whether image capture under all shooting conditions has been completed. If not, the process returns to step S71 to capture image capture under the next shooting condition. That is, in step S71, the lighting control unit 41 changes the lighting direction of the lighting device 11, in step S72, the subject W is captured again, and in step S73, a new original image is captured. In this embodiment, the processing of steps S71 to S74 is repeated four times, and four original images of the same subject W but with different lighting directions are captured by the processing device 12. When image capture under all shooting conditions has been completed, the process proceeds to step S75. The processing from this point onwards is the same as in the first embodiment, so a description thereof will be omitted.

As mentioned above, changing the lighting direction changes the diffuse reflection conditions in the background, etc. Therefore, by selecting from multiple original images taken with different lighting directions the one with the least influence of the diffuse reflection component (i.e., noise) and the clearest polarization component, it is possible to extract more reliable polarization information.

In this embodiment, a lighting device 11 with multiple light sources that can be turned on independently is used, and the lighting direction is changed by switching the light source that is turned on. This method has the advantage of being able to change the lighting direction quickly with simple control, thereby shortening the tact time for photography.

(others)
The above-described embodiment merely exemplifies the configuration of the present invention, and the present invention is not limited to the specific embodiment described above, and various modifications are possible within the scope of the technical concept thereof.

For example, the arrangement and configuration of the polarization camera and lighting device are not limited to those shown in Figure 1 and Figures 3A to 3C, and can be designed appropriately depending on the subject and application. In other words, it is sufficient if the specular reflection of the illumination light can be observed by the polarization camera. Furthermore, multiple polarization cameras can be installed, or movable polarization cameras and lighting devices can be used.

In the above embodiment, polarizers with 0-degree, 45-degree, 90-degree, and 135-degree angles were used, but the setting of the transmission axis direction is not limited to this. All that is required is that the imaging element be able to observe light that has passed through polarizers with different transmission axis directions, and be able to extract polarization information contained in the incident light from the observation results.

The UI screen in Figure 8 allows the user to set the mask size used in the polarization information calculation process and the parameters used in the noise removal process, but the UI screen configuration and parameter items are merely examples. Any UI that allows the user to change the parameters used in the polarization extraction process is acceptable. In other words, it is sufficient if the user can change the polarization extraction logic as appropriate depending on the subject, application, etc. Also, in the above embodiment, the parameter receiving unit 424 provided a user interface for setting parameters, but the parameter receiving unit 424 may also receive parameter change instructions from the user from an external device via a network.

The image brightness switching control in the first embodiment and the lighting direction switching control in the second embodiment may be performed together. This makes it possible to both uniformize the lighting conditions across the entire field of view and reduce diffuse reflection noise, enabling more reliable extraction of polarization information.

In addition, in the second embodiment, the lighting direction is changed by switching the light source that is turned on, but the configuration of the lighting device is not limited to this. For example, the lighting direction may be changed by using a lighting device 11 that illuminates from one direction as shown in Figure 3C and changing the relative positional relationship between the lighting device 11 and the subject W.

<Appendix 1>
An imaging system (1) for imaging a transparent object (W), comprising:
an illumination device (11) for illuminating a transparent object (W);
a polarization camera (10) having an imaging element (20) in which polarizers (210) with different transmission axis directions are regularly arranged;
a processing device (12),
The processing device (12)
a control means (40, 41) for controlling either or both of the illumination device (11) and the polarization camera (10) in order to photograph the transparent object (W) under different photographing conditions;
image acquisition means (420) for acquiring a plurality of original images taken under different photographing conditions;
a polarization extraction means (421) for executing a polarization extraction process for extracting polarization information, which is information about polarization resulting from specular reflection on the transparent object (W), for each pixel of each of the plurality of original images;
and an image generation means (422) that combines multiple pieces of polarization information extracted from each of the multiple original images by the polarization extraction process and generates a polarization information image that visualizes the polarization information.

<Appendix 2>
A control method for an imaging system (1) including an illumination device (11), a polarization camera (10) having an imaging element (20) in which polarizers (210) having different transmission axis directions are regularly arranged, and a processing device (12), comprising:
A step of photographing a transparent object (W) under different photographing conditions by controlling either or both of the lighting device (11) and the polarization camera (10);
A step of inputting a plurality of original images taken under different photographing conditions into the processing device (12);
a step of executing a polarization extraction process by the processing device (12) for each pixel of each of the plurality of original images to extract polarization information, which is information regarding polarization resulting from specular reflection on the transparent object (W), and synthesizing the plurality of pieces of polarization information extracted from each of the plurality of original images by the polarization extraction process to generate a polarization information image that visualizes the polarization information.

1: Imaging system 10: Polarization camera 11: Illumination device 12: Processing device 13: Stage 20: Imaging element 21: Polarizer array 30: Rod-shaped illumination

Claims (13)

1. An imaging system for imaging a transparent object, comprising:
an illumination device for illuminating a transparent object;
a polarization camera having an image sensor in which polarizers with different transmission axis directions are regularly arranged;
a processing device;
The processing device includes:
a control means for controlling one or both of the illumination device and the polarization camera in order to photograph the transparent object under different photographing conditions;
image acquisition means for acquiring a plurality of original images captured under different photographing conditions;
a polarization extraction means for executing a polarization extraction process to extract, for each pixel of each of the plurality of original images, polarization information that is information regarding polarization resulting from specular reflection on the transparent object;
an image generating means for synthesizing the plurality of pieces of polarization information extracted from the plurality of original images by the polarization extraction process and generating a polarization information image by visualizing the polarization information ,
The image generating means performs an operation of selecting, for each pixel, the polarization information with the highest reliability from the plurality of pieces of polarization information extracted from each of the plurality of original images, and generates the polarization information image using the polarization information selected for each pixel.
An imaging system characterized by: 2. The imaging system according to claim 1 , wherein said image generating means regards the strongest degree of polarization among said plurality of pieces of polarization information as the most reliable polarization information. The imaging system according to any one of claims 1 to 2 , characterized in that the control means controls one or both of the lighting device and the polarization camera so as to obtain a plurality of original images with different brightness levels. 4. The imaging system according to claim 3 , wherein the control means controls the illumination intensity of the illumination device to be different as the imaging condition. 5. The imaging system according to claim 3 , wherein the control means controls the polarization camera to vary an exposure time as the imaging condition. 6. The imaging system according to claim 3 , wherein the control means controls the polarization camera to vary a gain as the imaging condition. 3. The imaging system according to claim 1, wherein the control means controls the lighting device so as to obtain a plurality of original images with different lighting directions for the transparent object. The lighting device has a plurality of light sources that can be turned on independently,
8. The imaging system according to claim 7 , wherein the control means changes the illumination direction of the transparent object by switching the light source to be turned on. The imaging system according to any one of claims 1 to 8, characterized in that the polarization extraction process includes a noise removal process that removes or reduces noise contained in the calculation results after calculating polarization information for each pixel of the original image . 10. The imaging system according to claim 9, wherein the noise removal process includes a process of determining whether the polarization information of a target pixel and its surrounding pixels is similar or dissimilar by comparing the polarization information of the target pixel and its surrounding pixels, and deleting the polarization information of the target pixel if it is determined that the polarization information is dissimilar . 11. The imaging system according to claim 9, wherein the noise removal process includes a process of comparing polarization information between a target pixel and its surrounding pixels to determine whether the polarization information of the target pixel and the surrounding pixels is similar or dissimilar, and if it is determined that the polarization information is similar, replacing the polarization information of the target pixel with polarization information of a pixel selected from the target pixel and the surrounding pixels . A control method for an imaging system including an illumination device, a polarization camera having an imaging element in which polarizers with different transmission axis directions are regularly arranged, and a processing device, comprising:
Photographing a transparent object under different photographing conditions by controlling one or both of the illumination device and the polarization camera;
A step of importing a plurality of original images taken under different photographing conditions into the processing device;
and performing a polarization extraction process by the processing device to extract, for each pixel of each of the plurality of original images, polarization information that is information about polarization resulting from specular reflection on the transparent object, and synthesizing the plurality of pieces of polarization information extracted from each of the plurality of original images by the polarization extraction process to generate a polarization information image that visualizes the polarization information,
In the step of generating the polarization information image, an operation of selecting the polarization information with the highest reliability from the plurality of pieces of polarization information extracted from each of the plurality of original images is performed for each pixel, and the polarization information image is generated using the polarization information selected for each pixel.
A control method comprising: A program for causing a processor to execute each step of the control method according to claim 12 .