JP7705046B2 - Three-dimensional measurement device, three-dimensional measurement method and program - Google Patents

Description

The present invention relates to a three-dimensional measurement device, a three-dimensional measurement method, and a program.

The active stereo method is a method of 3D measurement using a camera and projector. The camera and projector are set up facing in the same direction, a pattern of light is projected, and the corresponding pixels on the projector are identified from the brightness values observed by the camera, which determines the parallax between the camera and projector and calculates the 3D position.

With active stereo, if the subject is semi-transparent and partially transmits light, the brightness values observed by the camera are a mixture of reflections from multiple layers, making it impossible to identify the corresponding pixel on the projector. For this reason, frame-based cameras that capture one frame at a time are sometimes used. Frame-based cameras output the absolute amount of brightness for each frame. However, when using a frame-based camera, reflections from multiple layers can be identified by lighting up the pixels on the projector one at a time and capturing one frame at a time, but as many captures as there are pixels are required.

An event-based vision camera is a camera equipped with an event-based vision sensor that uses a different method from frame-based cameras. In an event-based vision camera, each pixel operates asynchronously and outputs the brightness change of each pixel with a time resolution of microseconds. Compared to frame-based cameras, event-based vision cameras have the advantages of high time resolution, a wide dynamic range, and the ability to capture only changes.

In 3D measurement using an event-based vision camera, light is projected onto the subject and the reflected light is observed by the event-based vision camera. Assuming that the subject does not transmit light and there is no scattering due to the surrounding environment, when the projector shines light onto only one pixel, the scattered light in the corresponding microscopic area on the subject's surface is observed by the event-based vision camera, and an event occurs at the corresponding pixel. Note that scattering due to the surrounding environment refers to scattered light that occurs when light hits something unrelated to the subject of observation, such as a wall or floor, but does not affect the subject of observation.

In three-dimensional measurements using an event-based vision camera, the subject is scanned by changing the lit pixels at regular intervals. When performing such scanning, if the scanning speed is sufficiently slow compared to the time resolution of the event-based vision camera, events caused by scattered light from the irradiation range at each time can be observed individually. When using a laser-irradiation projector that uses MEMS (Micro Electro Mechanical Systems) as the projector, a similar observation can be made even if all pixels are constantly lit, because the laser irradiation position changes over time as the MEMS is driven.

When the geometric relationship between the projector and the event-based vision camera is known, the three-dimensional position of the corresponding subject can be calculated from the relationship between the lit pixel position, which is the position of the pixel that is lit in the projector, and the ignited pixel position, which is the position of the pixel where the event is ignited in the event-based vision camera, at each time. Note that in an event-based vision camera, an event being ignited at a certain pixel means that the amount of change in luminance value at a certain pixel in the event-based vision camera exceeds a threshold value, and an event detection signal is generated.

If transmission and internal scattering are not taken into account, when light reflected and scattered at a certain point on the subject ignites an event at one pixel on the event vision sensor, an event occurs at this pixel once per projector scanning period T. In this case, there is a one-to-one correspondence between the projector's lit pixel positions and the event ignition pixel positions, and it is possible to identify the corresponding projector lit pixel position from the event ignition time at each event ignition pixel position, and the three-dimensional position can be found by calculating the parallax between the camera and projector at the scattering point from this relationship.

However, if the subject transmits light and scattering occurs at various points on the subject's surface and inside, events may be fired at multiple pixels on the event-based vision camera in response to the lighting of a single pixel on the projector. As a result, multiple events may be observed for a single pixel on the event-based vision camera during one illumination period due to light reaching that pixel from different reflection positions.

Figure 13 is an explanatory diagram that explains the firing of events at multiple pixels of an event-based vision camera, and the observation of multiple events at one pixel of the event-based vision camera. Figure 13 shows two types of lasers with different irradiation timing, i.e., irradiation angles, and the path each laser takes from when it is irradiated until it reaches the camera. Parallax can be calculated from the event occurrence position and occurrence timing, and the lit pixel position of the projector at the same timing, and the three-dimensional position of the reflection position on the front and back surfaces can be calculated.

However, in event-based vision cameras, events are often missed, noise is generated, and delays in event firing occur due to insufficient bandwidth or light. If an event is missed, it is impossible to calculate the 3D position, and if there is a noise event, noise will also occur in the restored 3D information. Furthermore, if there is a delay in the event firing time, a mismatch occurs in the pixel correspondence between the event vision sensor and the projector, making it impossible to obtain the correct parallax.

Light reflected from the back of the subject is particularly susceptible to these problems, as it often has an insufficient amount of light to pass through the front. Thus, 3D measurement using an event-based vision camera can sometimes result in poor measurement accuracy. Note that a delay in the event firing time means that the event firing time recorded in the event detection signal is shifted from the time when the actual luminance change occurred. The time when the actual luminance change occurred is the time when the projector irradiated the observation point with light, or the time when irradiation ended.

In view of the above circumstances, the present invention aims to provide a technology that improves the accuracy of three-dimensional measurements using an event vision-based camera.

One aspect of the present invention is a three-dimensional measuring device that includes an event time series acquisition unit that acquires, for each pixel of the event vision-based camera, an event time series that is a time series indicating the presence or absence of an ignition, which is a phenomenon in which an event detection signal indicating that a luminance change in a pixel of the event vision-based camera is generated, the event time series being a time series generated based on the output of an irradiation device that irradiates an electromagnetic wave to a measurement object and scans the electromagnetic wave at a predetermined period, and an event correlation strength acquisition unit that acquires, for each pixel of the event vision-based camera, a correlation strength time series that indicates a time change in the strength of the cross-correlation between the event time series and a template function that is a predetermined function of the period, an irradiation situation estimation unit that estimates the angle or position at which the electromagnetic wave detected by each pixel is irradiated from the irradiation device based on the correlation strength time series, and a shape estimation unit that estimates the three-dimensional shape of the measurement object based on the result of estimation by the irradiation situation estimation unit.

One aspect of the present invention is a three-dimensional measuring device that includes: an event time series acquisition unit that acquires, for each pixel of the event vision-based camera, an event time series that is a time series indicating the presence or absence of an ignition, which is a phenomenon that generates an event detection signal indicating that a luminance change of a pixel of the event vision-based camera is detected, and the event time series is a time series generated based on the output of an irradiation device that irradiates an electromagnetic wave to a measurement object and scans the electromagnetic wave at a predetermined period, and the event time series is a time series indicating the presence or absence of an ignition, which is a phenomenon that generates an event detection signal indicating that a luminance change of a pixel of the event vision-based camera has been detected; a cluster estimation unit that executes the following processes: clustering for each ignition indicated by the event time series, estimating clusters whose distance between clusters is the period based on the results of the clustering, and estimating the angle or position at which the electromagnetic wave detected by each pixel is irradiated from the irradiation device for the clusters whose distance between clusters is estimated to be the period based on the values of each element of the cluster; and a shape estimation unit that estimates the three-dimensional shape of the measurement object based on the results of the estimation by the cluster estimation unit.

One aspect of the present invention is a three-dimensional measurement method including: an event time series acquisition step for acquiring, for each pixel of the event vision-based camera, an event time series that is a time series indicating the presence or absence of an ignition, which is a phenomenon that generates an event detection signal indicating that a luminance change in a pixel of the event vision-based camera has been detected, the event time series being a time series generated based on the output of an irradiation device that irradiates an electromagnetic wave to a measurement object and scans the electromagnetic wave at a predetermined period, and an event correlation strength acquisition step for acquiring, for each pixel, a correlation strength time series that indicates a time change in the strength of the cross-correlation between the event time series and a template function that is a predetermined function of the period; an irradiation situation estimation step for estimating an angle or position at which the electromagnetic wave detected by each pixel is irradiated from the irradiation device based on the correlation strength time series; and a shape estimation step for estimating a three-dimensional shape of the measurement object based on the result of the estimation in the irradiation estimation step.

One aspect of the present invention is a three-dimensional measurement method including: an event time series acquisition step for acquiring, for each pixel of the event vision-based camera, an event time series that is a time series indicating the presence or absence of an ignition, which is a phenomenon in which an event detection signal indicating that a change in luminance of a pixel of the event vision-based camera is generated, the event time series being a time series generated based on the output of an irradiation device that irradiates an electromagnetic wave on a measurement object and scans the electromagnetic wave at a predetermined period, the event time series being a time series indicating the presence or absence of an ignition, the event detection signal indicating that a change in luminance of a pixel of the event vision-based camera has been detected; a cluster estimation step for executing the following steps: clustering for each ignition indicated by the event time series; a process for estimating clusters whose inter-cluster distance is the period based on the results of the clustering; and a process for estimating, for the clusters whose inter-cluster distance is estimated to be the period, an angle or position at which the electromagnetic wave detected by each pixel is irradiated from the irradiation device based on the value of each element of the cluster; and a shape estimation unit for estimating a three-dimensional shape of the measurement object based on the results of the estimation in the cluster estimation step.

One aspect of the present invention is a program for causing a computer to function as the above-mentioned three-dimensional measurement device.

This invention makes it possible to improve the accuracy of three-dimensional measurements using an event vision-based camera.

FIG. 1 is an explanatory diagram illustrating an overview of a three-dimensional measurement system according to an embodiment. 5A and 5B are diagrams showing an example of one scan of a laser irradiated by an irradiation device in the embodiment. FIG. 2 is a diagram illustrating an example of a light receiving surface of an event vision-based camera in an embodiment. FIG. 4 is a diagram showing an example of a graph of an event time series in the embodiment. FIG. 4 is a diagram showing an example of a template function according to the embodiment. FIG. 13 is a diagram showing an example of a graph showing a correlation strength time series in the embodiment. FIG. 1 is a diagram showing an example of a hardware configuration of a three-dimensional measuring apparatus according to an embodiment. FIG. 2 is a diagram showing an example of the functional configuration of a control unit included in the three-dimensional measuring apparatus of the embodiment. 10 is a flowchart showing an example of a processing flow executed by an illumination device and an event vision-based camera in the three-dimensional measurement system of the embodiment. 1 is a flowchart showing an example of a flow of processing executed by a three-dimensional measuring apparatus in the three-dimensional measuring system of the embodiment. FIG. 13 is a diagram showing an example of the configuration of a control unit in a modified example. 10 is a flowchart showing an example of a flow of processing executed by a three-dimensional measuring apparatus that executes a cluster estimation process in a modified example. FIG. 1 is an explanatory diagram illustrating that multiple events are observed at one pixel of an event-based vision camera.

(Embodiment)
An overview of a three-dimensional measurement system 100 according to an embodiment will be described with reference to Figures 1 to 6. The three-dimensional measurement system 100 is a system that irradiates an irradiation target 9 with electromagnetic waves and estimates the three-dimensional shape of the irradiation target 9 based on the scattering of the electromagnetic waves. Note that the term "scattering" in the description of the three-dimensional measurement system 100 is used to indicate a phenomenon in which the propagation direction of the electromagnetic waves changes. Therefore, the three-dimensional measurement system 100 will be described with reflection as a type of scattering. First, an overview of the processing executed by the three-dimensional measurement system 100 will be described using an example in which the electromagnetic waves are lasers.

＜Overview＞
1 is an explanatory diagram for explaining an overview of a three-dimensional measurement system 100 according to an embodiment. The three-dimensional measurement system 100 includes an irradiation device 1, an event vision-based camera 2, and a three-dimensional measurement device 3. The irradiation device 1 is a device that irradiates a laser, and details of the irradiation device 1, the event vision-based camera 2, and the three-dimensional measurement device 3 will be described later. The event vision-based camera 2 detects the laser scattered by an irradiation target 9, and outputs a signal indicating the detection (hereinafter referred to as an "event detection signal"). The three-dimensional measurement device 3 estimates the three-dimensional shape of the irradiation target 9 based on the event detection signal.

Figure 2 is a diagram showing an example of one scan of the laser irradiated by the irradiation device 1 in the embodiment. As shown in Figure 2, the irradiation device 1 irradiates the laser by scanning it. In other words, the irradiation device 1 irradiates the laser while changing the angle of irradiation, for example.

Figure 3 is a diagram showing an example of the light receiving surface of the event vision-based camera 2. The event vision-based camera 2 has a light receiving unit 21. The light receiving unit 21 is present on the light receiving surface of the event vision-based camera 2. The light receiving unit 21 has a plurality of event elements 210 arranged in an array. The event elements 210 are elements that convert light into an electrical signal by photoelectric conversion. The electrical signal generated by the conversion is specifically an event detection signal.

The event detection signal is input to the three-dimensional measuring device 3. FIG. 4 is a diagram showing an example of the timing at which the three-dimensional measuring device 3 in the embodiment acquires the event detection signal. More specifically, it is a diagram showing an example of the timing at which the three-dimensional measuring device 3 acquires the event detection signal output by one event element 210 for that event element 210. The horizontal axis of FIG. 4 indicates time, and the vertical axis indicates intensity. The peaks in FIG. 4 indicate the event detection signal. The three-dimensional measuring device 3 obtains information indicating the timing at which the event detection signal is generated for each event element 210, as shown in FIG. 4.

In the example of FIG. 4, the event detection signal has two peaks during a time interval T. The time interval T is the time it takes for the laser to scan once. Therefore, the laser is scanned with a period T. Hereinafter, one of the two peaks that exist during the time interval T in FIG. 4 will be referred to as the left peak, and the other will be referred to as the right peak.

The left and right peaks are generated by lasers with different irradiation timings, for example. Different irradiation timings specifically mean different phases of irradiation. The phase is the time elapsed from the start of each unit period. In other words, the start time of the mth scan (m is an integer equal to or greater than 1) is defined as T0_m, and the elapsed time from the start time T0_m in one scan with the start time T0_m is the phase of irradiation. A unit period is a period with a length of a period T. Since the laser operation is repeated with a period T, the left peak in FIG. 4 appears repeatedly at approximately the same timing in the time interval T. Also, since the laser operation is repeated with a period T, the right peak in FIG. 4 appears repeatedly at approximately the same timing in the time interval T.

In the example of FIG. 4, there are two types of peaks, a left peak and a right peak, but the peaks that appear within the time interval T and that repeatedly appear at approximately the same period of the cycle T do not necessarily have to be of two types. There may be N types of peaks that appear within the time interval T and that repeatedly appear at approximately the same period of the cycle T (N is an integer equal to or greater than 1).

For data showing that N types of peaks repeatedly appear at approximately the same period T, as shown in FIG. 4, the three-dimensional measuring device 3 obtains the strength of cross-correlation with a given function (hereinafter referred to as a "template function") of a predetermined period T, as shown in the graph in FIG. 5. FIG. 6 is an example of a graph showing the strength of cross-correlation. Specifically, obtaining the strength of cross-correlation means performing a convolution integral.

Figure 5 is a diagram showing an example of a template function in an embodiment. The horizontal axis of Figure 5 indicates time, and the vertical axis of Figure 5 indicates intensity. Specifically, the function expressing the graph of Figure 5 is a function indicating that a Gaussian function appears repeatedly with a period T. For ease of explanation, the following describes the three-dimensional measurement system 100 using as an example a case in which the template function is a function indicating that a Gaussian function appears repeatedly with a period T.

Figure 6 is a diagram showing an example of a graph showing the strength of cross-correlation in an embodiment. The vertical axis of Figure 6 shows the strength of cross-correlation, and the horizontal axis shows the time difference between the peak of a Gaussian distribution in a unit period and the peak of an event detection signal input to the three-dimensional measurement device 3 in a unit period that includes that Gaussian distribution. Therefore, the horizontal axis also shows the phase.

In fact, the event detection signal varies in latency, which is the time from when a laser is incident on the event vision-based camera 2 to when the event detection signal is output from the event vision-based camera 2. One of the reasons for this variation is that, for example, the probability of photoelectric conversion in which the event vision-based camera 2 converts electromagnetic waves into electrons follows quantum mechanics such as Fermi's golden rule. Another reason for this variation is that, for example, there is a limit to the amount of event detection signal that the event vision-based camera 2 can output per unit time. Due to this limit, even if, for example, many electrons are excited, only a small amount of the event detection signal is output, resulting in variation in latency.

The distribution showing the variation in latency is expressed, for example, by a Gaussian distribution. For simplicity of explanation, the following will be described using an example in which the distribution showing the variation in latency is a Gaussian distribution. When the latency of the event detection signal output from the event vision-based camera 2 varies according to a Gaussian distribution, the timing at which the event detection signal enters the three-dimensional measurement device 3 also varies according to a Gaussian distribution.

Therefore, the probability that the three-dimensional measuring device 3 acquires an event detection signal is highly likely to be the probability that a Gaussian distribution is repeated with a period T, as shown in FIG. 5, for example. As is clear from the explanation so far, the position of the peak of the Gaussian distribution in each unit period is the median of the latency variation. The position of the peak of the Gaussian distribution is, for example, located at the time when the time difference from the start time of the unit period is the most likely that an event detection signal will be input to the three-dimensional measuring device 3. The shape of the Gaussian distribution does not depend only on the latency, but may also depend on the amount of noise, as will be described in detail later.

As mentioned above, the process of obtaining the strength of cross-correlation is specifically a convolution integral. Therefore, the process of obtaining the strength of cross-correlation can express frequently occurring phenomena more clearly than infrequently occurring phenomena. For example, consider the case in which a peak that does not have a period T and appears only once in the graph in Figure 4 is mixed in outside the range of the Gaussian distribution indicated by the template function.

Even if such a situation occurs, by obtaining the strength of the cross-correlation with the template function, it is possible to obtain a result in which the peak that appears with the period T is emphasized rather than such a peak that appears only once. Since the period T is the period of the laser scan, the peak that appears with the period T is specifically N types of event detection signals. Note that emphasizing specifically means obtaining a result with a stronger correlation strength. Note that a peak that does not have a period T and appears only once is an example of noise.

Therefore, the process of obtaining the cross-correlation strength is a process of extracting the event detection signal that occurs due to the laser scanning.

Based on the results thus obtained indicating the strength of cross-correlation, the three-dimensional measuring device 3 obtains the phase of the peak that appears within the unit period and that satisfies an intensity condition, which is a predetermined condition regarding the strength of cross-correlation. An example of the intensity condition will be described in detail later, but for example, the condition is that the strength of cross-correlation is a peak that is equal to or greater than a threshold value.

As described above, by obtaining the strength of cross-correlation, N types of event detection signals, which are peaks that appear at a period T, are extracted. As shown in FIG. 6, the information indicating the strength of cross-correlation is, specifically, information indicating the timing of the N types of event detection signals in a unit period. Therefore, by obtaining the strength of cross-correlation, it is possible to obtain information indicating the phase of the laser irradiation that caused each of the N types of event detection signals.

If the irradiation target 9 is, for example, composed of N layers of translucent material, then each of the N types of event detection signals will have a one-to-one correspondence with one of the N layers. Corresponding to one of the layers means that the nth (n is an integer between 1 and N) event detection signal among the N types of event detection signals is an event detection signal caused by laser scattering in the nth layer of the N layers.

The reason why this can be said is as follows. Multiple lasers with different irradiation phases can be incident on one event element 210. However, if the layer on which the laser is reflected is the same regardless of the irradiation phase, lasers irradiated with different phases will not be incident on the same event element 210, based on the principle that the reflection angle is proportional to the incidence angle. Therefore, each of the N types of event detection signals output by one event element 210 is a signal generated due to the laser being reflected from a different layer.

The irradiation target 9 does not necessarily have to be made of N layers of translucent material. Instead of N layers, the irradiation target 9 may be a material that has N locations that scatter the laser (hereinafter referred to as "electromagnetic wave scattering locations"). In such a case, each of the N types of event detection signals is an event detection signal that is generated due to the laser being scattered at a mutually different electromagnetic wave scattering location. In other words, there is a one-to-one correspondence between the N locations of electromagnetic wave scattering locations and the N types of event detection signals.

In this way, the three-dimensional measurement system 100 estimates the phase of the laser irradiation that generated each type of event detection signal for each event element 210, and estimates the three-dimensional shape of the irradiation target 9 based on the estimated results. This concludes the overview of the three-dimensional measurement system 100.

<More detailed description>
The three-dimensional measurement system 100 will be described again with reference to Figures 1 to 10 including Figures 1 to 6. As described above, the three-dimensional measurement system 100 includes the irradiation device 1, the event vision-based camera 2, and the three-dimensional measurement device 3. The three-dimensional measurement system 100 performs three-dimensional measurement of the irradiation target 9. Performing three-dimensional measurement of the irradiation target 9 means estimating the three-dimensional shape of the irradiation target 9. As will be described later, the estimation of the three-dimensional shape is performed by the three-dimensional measurement device 3. Therefore, the irradiation target 9 is the measurement target of the three-dimensional measurement system 100 or the three-dimensional measurement device 3.

The irradiation device 1 irradiates the irradiation target 9 with a laser and scans the laser at a predetermined period T. In other words, the irradiation device 1 repeats the process of completing one scan of the irradiation target 9 when the period T has elapsed.

The irradiation target 9 may be any object capable of scattering the laser irradiated by the irradiation device 1, which is incident at multiple positions. In other words, the irradiation target 9 may be any object having portions at multiple positions that have a predetermined reflectance greater than 0 and less than 1 for the frequency of the laser irradiated by the irradiation device 1. The irradiation target 9 may be, for example, an N-layer scatterer (N is an integer equal to or greater than 1). Each of the N layers may be any object that has a predetermined reflectance greater than 0 and less than 1. Note that since the Kramers-Kronig relationship generally holds for materials, scattering will occur unless the material is a perfect absorber.

Figure 2 will be explained in more detail. More specifically, Figure 2 is a diagram showing the change over time in the position on the camera where the laser arrives, in a scene where the laser emitted by the irradiation device 1 is directly received by the camera placed in a situation where there is no scatterer between the camera and the irradiation device 1. The directions of the vertical axis and the horizontal axis in Figure 2 are two mutually perpendicular directions in real space. Point D1 indicates the position on the camera when the laser irradiated at the start of one cycle arrives at the camera. The direction of the arrow in Figure 2 indicates the position on the camera where the laser arrives, which moves over time. This change occurs because the angle at which the irradiation device 1 irradiates the laser changes over time.

In this way, there is a one-to-one relationship between the angle at which the irradiation device 1 irradiates the laser and the phase of the irradiation. Note that the start time of the scan is the start time of the unit period, and the end time of the scan is the end time of the unit period.

Figure 1 shows two types of lasers with different timing of laser irradiation from the irradiation device 1, i.e., different angles of irradiation, and the path of each laser from when it is irradiated until it reaches the event vision-based camera 2. That is, Figure 1 shows the path of a laser irradiated at a first phase and a laser irradiated at a second phase different from the first phase.

Figure 1 shows three optical paths of the laser. The first optical path is an optical path that travels from the irradiation device 1 to part R1 of the irradiation target 9, is scattered at part R1, and reaches the event vision-based camera 2. The second optical path is an optical path that travels from the irradiation device 1 to part R1 of the irradiation target 9, passes through part R1 to part R2 of the irradiation target 9, is scattered at part R2, passes through part R3 of the irradiation target 9, and reaches the event vision-based camera 2. The third optical path is an optical path that travels from the irradiation device 1 to part R3 of the irradiation target 9, is scattered at part R3, and reaches the event vision-based camera 2.

The first optical path and the second optical path are, for example, examples of optical paths of a laser irradiated with a first phase. In this case, the third optical path is an example of an optical path of a laser irradiated with a second phase.

The irradiation device 1 has a control unit 11 including a processor 91 such as a CPU (Central Processing Unit) and a memory 92 connected by a bus, and executes a program. By executing the program, the irradiation device 1 functions as a device including the control unit 11, a user interface 12, a light source 13, and a memory unit 14.

More specifically, in the irradiation device 1, the processor 91 reads out a program stored in the storage unit 14 and stores the read out program in the memory 92. The processor 91 executes the program stored in the memory 92, whereby the irradiation device 1 functions as a device including the control unit 11, the user interface 12, the light source 13, and the storage unit 14.

The control unit 11 controls the operation of various functional units equipped in the irradiation device 1. The control unit 11 controls the operation of the light source 13, for example. Under the control of the control unit 11, the light source 13 starts emitting laser light. Also, under the control of the control unit 11, the light source 13 ends laser emission. Under the control of the control unit 11, the light source 13 changes the angle of laser emission. By changing the angle of laser emission according to time, for example, scanning as shown in FIG. 2 is performed.

The user interface 12 includes input devices such as a mouse, keyboard, and touch panel. The user interface 12 may also include an interface that connects these input devices to the irradiation device 1. The user interface 12 also includes a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display. The user interface 12 may also include an interface that connects these display devices to the three-dimensional measurement device 3.

Light source 13 is a light source that emits laser light. In other words, light source 13 is a laser device.

The storage unit 14 is configured using a computer-readable storage medium device such as a magnetic hard disk device or a semiconductor storage device. The storage unit 14 stores various information related to the irradiation device 1.

The event vision-based camera 2 is a camera that asynchronously detects changes in luminance of each pixel, and outputs a signal indicating that a change in luminance of a pixel has been detected, in combination with information indicating the position of the detected pixel and information indicating the time of detection. In other words, the event vision-based camera 2 is a so-called event vision-based camera.

Hereinafter, a signal indicating that a luminance change has been detected is referred to as an event detection signal. There are two types of luminance change: an increase in luminance and a decrease in luminance. Therefore, the luminance change has a direction. For example, the direction of the luminance change is a positive direction when the luminance increases, and a negative direction when the luminance decreases. Therefore, the event detection signal also includes information indicating the direction of the luminance change. The three-dimensional measurement device 3 does not need to use all of the input event detection signals to estimate the three-dimensional shape of the irradiation target 9. For example, it is possible to use an event detection signal in one of the two directions of luminance change and not use the other event detection signal.

In this way, the event vision-based camera 2 outputs an event detection signal, information indicating the pixel that detected the pixel luminance change (hereinafter referred to as "detection pixel information"), and information indicating the time when the pixel luminance change was detected (hereinafter referred to as "detection time information"). The detection pixel information is information indicating the pixel that detected the pixel luminance change, and therefore is information indicating the pixel that output the event detection signal. The detection time information is information indicating the time when the pixel luminance change was detected, and therefore is information indicating the time when the event detection signal was output.

In the event vision-based camera 2, pixels are arranged in an array to form the light receiving section 21. The electromagnetic waves incident on the event vision-based camera 2 are, for example, lasers irradiated by the irradiation device 1, and whose propagation direction is changed by the irradiation target 9.

Figure 3 will be explained in more detail. As described above, the event element 210 generates an electrical signal by photoelectric conversion, and the generated electrical signal is specifically an event detection signal. Since photoelectric conversion is a phenomenon governed by quantum mechanics such as Fermi's Golden Rule, the event detection signal is a signal that is probabilistically output with a probability according to the intensity of the received electromagnetic wave. Each of the event elements 210 is a pixel of the event vision-based camera 2.

Since the event detection signal is a signal that is output probabilistically with a probability according to the intensity of the received electromagnetic wave, the event detection signal may not be generated within a unit period even if the event element 210 receives light. The phenomenon in which an event detection signal is generated is also referred to as ignition. Hereinafter, the event element 210 that generates the event detection signal is also referred to as the ignited event element 210.

The relationship between the optical path of the laser and the igniting event element 210 will be described. When estimating the three-dimensional shape of the irradiation target 9, it is desirable to uniquely determine the irradiation angle for the igniting event element 210. The reason is as follows. If information on the irradiation angle at which a certain event element 210 ignited can be obtained, it is possible to estimate from what angle the laser scattered at a certain position was irradiated, using the positional relationship between the irradiation device 1 and each event element 210. As a result, it is possible to estimate the three-dimensional shape of each layer of the irradiation target 9. For this reason, it is desirable to uniquely determine the irradiation angle for the igniting event element 210. In other words, it is desirable that the time series of data for one event element 210 indicating the presence or absence of ignition is a time series indicating only one ignition per unit period. Hereinafter, the time series of data indicating the presence or absence of ignition is referred to as an event time series.

However, as described above, since the laser light is scattered at different locations in the irradiated target 9, there may be laser light incident on the same event element 210 even at different irradiation angles. As a result, the event time series of one event element 210 may indicate that multiple types of ignitions occur in one or more unit periods, each of which has a different phase of laser irradiation that causes the ignition. In addition, ignitions are generated by photoelectric conversion as described above, but as is well known, thermal noise (i.e., photons) is also converted to electrons. As a result, ignitions may not occur even when no light is received. As a result, without analyzing the event time series, it is not possible to determine the cause of an ignition, such as whether it is caused by laser reception or noise, or, if it is caused by laser reception, which irradiation phase it belongs to.

The three-dimensional measuring device 3, which will be described later, estimates the cause of the firing included in the event time series of each event element 210. However, it is not necessary to estimate the cause of all firings. For example, it is sufficient to only estimate the phase of one irradiation. By estimating the phase for multiple firings with different causes, for example, if the irradiated object 9 has multiple layers, it is possible to perform three-dimensional measurement for the multiple layers. Furthermore, by estimating the phase for multiple firings with different causes, for example, if the irradiated object 9 has multiple electromagnetic wave scattering sites, it is possible to estimate the positions in three-dimensional space for the multiple electromagnetic wave scattering sites.

The three-dimensional measuring device 3 estimates the three-dimensional shape of the irradiation target 9 based on the event detection signal, the detection pixel information, and the detection time information. More specifically, the three-dimensional measuring device 3 estimates the three-dimensional shape of the irradiation target 9 based on information on the phase and angle of irradiation of the irradiation device 1 (hereinafter referred to as "irradiation setting information") and information indicating the positional relationship between the irradiation device 1, the irradiation target 9, and each pixel of the event vision-based camera 2 (hereinafter referred to as "system positional relationship information").

Next, the process executed by the three-dimensional measuring device 3 to estimate the three-dimensional shape of the irradiation target 9 will be described with reference to Figures 4 to 6. The three-dimensional measuring device 3 generates a time series (hereinafter referred to as an "event time series") of data indicating the presence or absence of ignition (hereinafter referred to as "ignition presence/absence data") for each pixel based on the event detection signal, detection pixel information, and detection time information. In other words, samples of the event time series are ignition presence/absence data. Hereinafter, the process of generating an event time series for each pixel based on the event detection signal, detection pixel information, and detection time information will be referred to as the event time series generation process.

Figure 4 will be explained in more detail. The graph shown in Figure 4 is an example of an event time series graph. The event time series graph shows the timing at which an event detection signal is output from the event vision-based camera 2 (i.e., the timing of firing). Therefore, more specifically, Figure 4 shows the timing of firing when a unit period of cycle T occurs six times.

Figure 4 shows that multiple firings occur in one unit period. The reason multiple firings occur is that, for example, as described above in Figure 1, there are differences in the locations where the laser light is scattered in the irradiated target 9, and therefore the scattering of the laser light irradiated in the first phase and the scattering of the laser light irradiated in the second phase are incident on one event element 210.

The three-dimensional measurement device 3 acquires information indicating the time change in the strength of the cross-correlation between the event time series and the template function (hereinafter referred to as "event correlation strength"). Hereinafter, the process of acquiring information indicating the time change in the event correlation strength (hereinafter referred to as "correlation strength time series") based on the event time series and the template function is referred to as the event correlation strength acquisition process. The template function is, for example, a function that expresses a Gaussian function that is repeated with a period T.

Figure 5 will be explained in more detail. The template function shown in Figure 5 indicates that there are more than five and not more than six unit periods. Figure 5 indicates that one Gaussian function occurs in each unit period. The Gaussian function for each unit period has the same half-width regardless of the unit period. In addition, the difference between the time of the peak of the Gaussian function for each unit period and the start time of each unit period is the same regardless of the unit period. In other words, the phase of the peak of the Gaussian function is the same regardless of the unit period.

Figure 6 will be explained in more detail. Figure 6 is a diagram showing an example of a graph showing a correlation strength time series in an embodiment. A function expressing a graph of the cross-correlation strength time series is, for example, a cross-correlation function obtained by convolution of a template function with an event time series. The cross-correlation function indicates the similarity and phase difference of two signals. When two signals are similar and the phase difference between them is τ, the cross-correlation function has a peak of intensity according to the similarity at the position of time τ. When the phase of the template signal is 0, the phase difference represents the phase of the event time series. Therefore, the process of obtaining the strength of the cross-correlation with the template function corresponds to the process of obtaining the phase of each output event detection signal.

The values of the half-width and peak time of the Gaussian function are predetermined values according to the amount of noise and latency. The amount of noise is the amount of noise occurring in the event detection signal. The latency is the difference between the actual time when a change in luminance of the electromagnetic wave incident on the event vision-based camera 2 occurs and the time recorded in the event detection signal, which is a difference that varies probabilistically.

As described above, the event detection signal is output from the event vision-based camera 2 probabilistically due to various factors, so latency depends on the probability that the event detection signal is output. Also, as described above, the event detection signal depends on quantum mechanics such as Fermi's Golden Rule and the amount of event detection signal that the event vision-based camera 2 can output per unit time. Therefore, the event detection signal is a signal that is output probabilistically with a probability that depends on the intensity of the received electromagnetic wave. In this way, since the output of the event detection signal is a probabilistic phenomenon that depends on the intensity of the received electromagnetic wave, it is possible to define an index called latency.

The lower the latency, i.e., the longer the time until the event detection signal is output, the later it is desirable to set the position of the peak of the Gaussian function within each unit period. The reason is as follows. As described above, the process of obtaining the strength of the cross-correlation with the Gaussian function repeated with a period T corresponds to the process of determining the unit period to which each ignition presence/absence data belongs. Therefore, it is desirable for the time of the peak of the Gaussian function to approximately match the latency time.

Furthermore, as shown by the theory of Fourier transform, the narrower the time width of the spectrum, the wider the frequency width. Therefore, the wider the half-width of the Gaussian function in Figure 5, the narrower the frequency width can be. Narrowing the frequency width corresponds to a process that further removes signals other than the target frequency component. Therefore, the wider the half-width of the Gaussian function, the more the effects of noise can be reduced. Therefore, it is desirable for the half-width of the Gaussian function to be wider the greater the amount of noise. In this way, the template function has a bandpass effect.

So far, the template function has been described as a function that represents the repetition of a Gaussian function. However, the template function may be any function that represents a function that is repeated with a period T, and may also be a function that represents the repetition of a function other than a Gaussian function.

The three-dimensional measuring device 3 estimates the phase of the irradiation that caused the ignition for each pixel based on the obtained correlation intensity time series. More specifically, based on information indicating the position of the start time of a unit period on the time axis of the correlation intensity time series (hereinafter referred to as "start time position information"), the three-dimensional measuring device 3 obtains the phase of a peak indicated by the correlation intensity time series that satisfies the above-mentioned intensity condition, which is a predetermined condition related to the strength of the event correlation intensity, among peaks that belong to any one unit period.

The intensity condition may be any condition that can select only the peaks indicating ignition caused by the reception of the laser irradiated by the irradiation device 1 from among the multiple peaks indicated by the correlation intensity time series. The intensity condition may be, for example, a condition that the event correlation intensity is the Nth strongest (N is a predetermined integer equal to or greater than 1) from the top in order of strength. The intensity condition may be, for example, a condition that the strength of the event correlation intensity is equal to or greater than a predetermined strength. The intensity condition may be, for example, a condition that the maximum value within a unit period of the cross-correlation function is equal to or greater than a threshold value (hereinafter referred to as an "extreme value condition"). For example, when the intensity condition is an extreme value condition, the phases of the peaks can be acquired not limited to a predetermined number. As a result, the three-dimensional measuring device 3 can estimate the three-dimensional shape of the irradiation target 9 more accurately than when the number of peaks for which the phases are acquired is predetermined.

The event correlation strength is a cross-correlation, so the greater the absolute value of the vertical axis of the graph of the ignition presence/absence data, the greater the value. As for noise, as described above, a template function with a half-width set in advance to take into account the effects of noise is used to perform processing equivalent to a bandpass, so the correlation strength is low. Therefore, the position of the peak of the mutual intensity time series that satisfies the intensity condition is the timing of ignition caused by the reception of the laser irradiated by the irradiation device 1. Therefore, the phase of the peak of the mutual intensity time series that satisfies the intensity condition is the phase at which ignition occurs due to the reception of the laser irradiated by the irradiation device 1. For example, if the intensity condition is that the strength of the event correlation strength is the strongest, the position of the peak that satisfies the intensity condition indicates the timing of ignition caused by the reception of the laser irradiated by the irradiation device 1.

Here, we will explain the effect of a missing event, which is an event in which no ignition occurs despite receiving laser light. Missing events are events that occur infrequently. Therefore, as a result of the action of the template function, the bandpass function of the template function reduces the event correlation strength of the missing event. Therefore, even if a missing event exists, the three-dimensional measurement device 3 can estimate the timing of ignition caused by receiving the laser light emitted by the irradiation device 1. Note that an event in which the event detection signal is output only little by little because there is a limit to the amount of event detection signal that the event vision-based camera 2 can output per unit time, resulting in the event detection signal being output with a latency that exceeds the unit period, is an example of a missing event.

In this way, the three-dimensional measuring device 3 acquires the phase of a peak that satisfies the intensity condition among the peaks indicated by the correlation intensity time series and that belong to any one unit period as the phase at which ignition occurred. Hereinafter, the process of estimating the phase at which ignition occurred based on the correlation intensity time series is referred to as the ignition phase estimation process. As described above, there is a one-to-one relationship between the phase of irradiation and the angle of irradiation, so estimating the phase at which ignition occurred is equivalent to estimating the angle of laser irradiation that caused the ignition.

The three-dimensional measuring device 3 estimates the three-dimensional shape of the irradiated object 9 based on the result of the firing phase estimation process. Specifically, for example, the three-dimensional measuring device 3 estimates the three-dimensional shape of the irradiated object 9 based on information indicating the phase at which one or more firings occurred, acquired for each pixel, the irradiation setting information, and the system position relationship information. For example, based on the irradiation setting information, it is estimated at what angle the laser was irradiated to cause each peak of each pixel, and based on the estimation result and the system position relationship information, the three-dimensional shape of the irradiated object 9 is estimated according to optical theory such as geometric optics. Hereinafter, the process of estimating the three-dimensional shape of the irradiated object 9 based on the result of the firing phase estimation process, the irradiation setting information, and the system position relationship information is referred to as the shape estimation process.

Figure 7 is a diagram showing an example of the hardware configuration of a three-dimensional measuring device 3 of an embodiment. The three-dimensional measuring device 3 has a control unit 31 including a processor 93 such as a CPU and a memory 94 connected by a bus, and executes a program. By executing the program, the three-dimensional measuring device 3 functions as a device including the control unit 31, input unit 32, communication unit 33, storage unit 34, and output unit 35.

More specifically, in the three-dimensional measuring device 3, the processor 93 reads out a program stored in the storage unit 34 and stores the read out program in the memory 94. When the processor 93 executes the program stored in the memory 94, the three-dimensional measuring device 3 functions as a device including a control unit 31, an input unit 32, a communication unit 33, a storage unit 34, and an output unit 35.

The control unit 31 controls the operation of various functional units of the three-dimensional measurement device 3. The control unit 31 executes, for example, an event time series generation process. The control unit 31 executes, for example, an event correlation strength acquisition process. The control unit 31 executes, for example, an ignition phase estimation process. The control unit 31 executes, for example, a shape estimation process.

The input unit 32 includes input devices such as a mouse, keyboard, and touch panel. The input unit 32 may include an interface that connects these input devices to the three-dimensional measuring device 3. Information indicating, for example, one or both of the half-width and peak phase of the Gaussian function included in the template function may be input to the input unit 32. The user may change, via the input unit 32, one or both of the half-width and peak phase of the Gaussian function included in the template function.

The communication unit 33 includes an interface for connecting the three-dimensional measuring device 3 to an external device. The communication unit 33 communicates with the external device via a wired or wireless connection. The external device is, for example, a device that transmits the start time position information. The device that transmits the start time position information is, for example, the irradiation device 1. The communication unit 33 acquires the start time position information by communicating with the device that transmits the start time position information.

The external device is, for example, the event vision-based camera 2. The communication unit 33 acquires the event detection signal, the detection pixel information, and the detection time information by communicating with the event vision-based camera 2.

The external device is, for example, a device that transmits irradiation setting information. The communication unit 33 acquires the irradiation setting information by communicating with the device that transmits the irradiation setting information. The external device is, for example, a device that transmits system positional relationship information. The communication unit 33 acquires system positional relationship information by communicating with the device that transmits the system positional relationship information. The external device is, for example, a device that transmits information indicating a template function (hereinafter referred to as "template information"). The communication unit 33 acquires the template information by communicating with the device that transmits the template information.

The irradiation setting information, system positional relationship information, or template information may be stored in advance in the storage unit 34, or may be input via the input unit 32.

The storage unit 34 is configured using a computer-readable storage medium device such as a magnetic hard disk device or a semiconductor storage device. The storage unit 34 stores various information related to the three-dimensional measuring device 3. The storage unit 34 stores various information resulting from the processing executed by the control unit 31, for example. The storage unit 34 may have irradiation setting information, system positional relationship information, or template information stored in advance, for example, as described above.

The output unit 35 includes a display device such as a CRT display, a liquid crystal display, or an organic EL display. The output unit 35 may include an interface that connects these display devices to the three-dimensional measurement device 3.

FIG. 8 is a diagram showing an example of the functional configuration of the control unit 31 provided in the three-dimensional measurement device 3 of the embodiment. The control unit 31 includes an event information acquisition unit 311, an event time series acquisition unit 312, an event correlation strength acquisition unit 313, an ignition phase estimation unit 314, a shape estimation unit 315, a communication control unit 316, a memory control unit 317, and an output control unit 318.

The event information acquisition unit 311 acquires event information. The event information includes an event detection signal, detection pixel information indicating the pixel that output the event detection signal, and detection time information indicating the time when the event detection signal was output.

The event time series acquisition unit 312 acquires an event time series for each pixel. The event time series acquisition unit 312 acquires an event time series by, for example, executing an event time series generation process for each pixel. The event correlation strength acquisition unit 313 executes an event correlation strength acquisition process. The ignition phase estimation unit 314 executes an ignition phase estimation process. The shape estimation unit 315 executes a shape estimation process.

The communication control unit 316 controls the operation of the communication unit 33. The memory control unit 317 controls the operation of the memory unit 34. The output control unit 318 controls the operation of the output unit 35.

Figure 9 is a flowchart showing an example of the processing flow executed by the irradiation device 1 and the event vision-based camera 2 in the three-dimensional measurement system 100 of the embodiment.

The irradiation device 1 starts irradiating the laser (step S101). That is, the irradiation device 1 starts irradiating the irradiation target 9 with the laser. Next, the event vision-based camera 2 receives the laser scattered by the irradiation target 9 (step S102). The event vision-based camera 2 outputs an event detection signal with a probability according to the received light intensity (step S103). The control unit 11 provided in the irradiation device 1 determines whether or not a predetermined end condition for irradiation (hereinafter referred to as the "irradiation end condition") is satisfied (step S104). If the irradiation end condition is satisfied (step S104: YES), the irradiation ends. On the other hand, if the irradiation end condition is not satisfied (step S104: NO), the irradiation device 1 changes the angle of irradiation (step S105). Since the laser continues to be irradiated continuously after step S101, the laser irradiated at the changed angle due to the change in the irradiation angle is irradiated to the irradiation target 9. The irradiation end condition is, for example, a condition that a predetermined number of scans have been completed. The condition for ending irradiation may be, for example, the passage of a predetermined period of time.

Figure 10 is a flowchart showing an example of the flow of processing executed by the three-dimensional measurement device 3 in the three-dimensional measurement system 100 of the embodiment.

The event information acquisition unit 311 acquires event information for a predetermined period (step S201). The predetermined period may be a period determined in advance, or may be a period from when the irradiation device 1 starts to irradiate the laser to when it ends.

After step S201, the event time series acquisition unit 312 acquires an event time series (step S202). In step S202, the event time series acquisition unit 312 generates an event time series for each pixel of the event vision-based camera 2 based on the event information, for example, by executing an event time series generation process for each pixel. Next, the event correlation strength acquisition unit 313 executes an event correlation strength acquisition process for each pixel (step S203). That is, the event correlation strength acquisition unit 313 acquires a correlation strength time series for each pixel based on the event time series and the template function. In step S203, the template function is read out, for example, from the memory unit 34.

Next, the firing phase estimation unit 314 executes firing phase estimation processing for each pixel (step S204). That is, the firing phase estimation unit 314 estimates the phase at which firing occurred for each pixel based on the correlation strength time series. Next, the shape estimation unit 315 executes shape estimation processing (step S205). That is, the shape estimation unit 315 estimates the three-dimensional shape of the irradiation target 9 based on the result of the firing phase estimation processing, the irradiation setting information, and the system positional relationship information.

The processing of steps S202 to S205 may be executed in parallel with steps S101 to S105, or may be executed after the processing of steps S101 to S105 is completed. Step S201 may be executed before the start of step S101. In this case, however, the event detection signal acquired before the start of step S101 is an event detection signal generated by noise, and is therefore a signal that has no correlation with the shape of the irradiation target 9.

The three-dimensional measuring device 3 of the embodiment configured in this manner uses a template function, which is a predetermined function of the period T, to estimate the three-dimensional shape of the irradiated object 9 based on the strength of the cross-correlation between the template function and an event time series, which is a time series of data indicating the presence or absence of ignition. By using the template function as described above, the effects of noise and missing events are suppressed, and it is possible to estimate the phase of laser irradiation for ignition caused by laser reception. If the phase of laser irradiation for ignition caused by laser reception can be estimated, the accuracy of three-dimensional measurement is improved as described above. Therefore, the three-dimensional measuring device 3 can improve the accuracy of measurement in three-dimensional measurement using an event vision-based camera.

The three-dimensional measurement system 100 of this embodiment configured in this manner includes a three-dimensional measurement device 3. Therefore, the three-dimensional measurement system 100 can improve the measurement accuracy in three-dimensional measurement using an event vision-based camera.

(Modification)
The three-dimensional measuring device 3 may execute a cluster estimation process instead of executing the event correlation intensity acquisition process and the ignition phase estimation process. In the cluster estimation process, each ignition indicated by the event time series is clustered by a clustering method such as k-means, and a process of estimating clusters with a distance between the clusters having a period T based on the clustering result is executed. In the cluster estimation process, the phase of each element of the estimated cluster is then estimated based on the element value of each cluster and the start time position information, thereby estimating the phase of laser irradiation for ignition caused by laser reception. As described above, the phase of irradiation and the angle of irradiation have a one-to-one relationship, so estimating the phase of laser irradiation is estimating the angle of laser irradiation. In this way, in the cluster estimation process, the phase of laser irradiation for ignition caused by laser reception is estimated.

The distance between clusters in the cluster estimation process is defined as the time difference that occurs most frequently in the set of differences in the values in the time axis direction between any two elements belonging to the cluster. Since ignition caused by laser reception occurs frequently with a period T, the execution of the cluster estimation process also estimates the phase of laser irradiation for ignition caused by laser reception, in the same way as the execution of the event correlation intensity acquisition process and the ignition phase estimation process.

Note that each firing indicated by the event time series refers to a sample that indicates the presence of an firing among each sample of the event time series. Hereinafter, the control unit 31 that executes the cluster estimation process instead of executing the event correlation strength acquisition process and the firing phase estimation process will be referred to as the control unit 31a.

Figure 11 is a diagram showing an example of the configuration of the control unit 31a in the modified example. The control unit 31a differs from the control unit 31 in that it includes a cluster estimation unit 319 instead of the event correlation strength acquisition unit 313 and the firing phase estimation unit 314.

Figure 12 is a flowchart showing an example of the flow of processing executed by the three-dimensional measurement device 3 that executes the cluster estimation processing in the modified example. Processing similar to that in Figure 10 is denoted by the same reference numerals as in Figure 10, and description thereof will be omitted.

After the processing of step S201 is executed, the processing of step S202 is executed. After the execution of step S202, a cluster estimation process is executed (step S203a). After step S203a, the shape estimation unit 315 estimates the three-dimensional shape of the irradiation target 9 based on the result of the cluster estimation process (step S205a). Specifically, for example, the three-dimensional measurement device 3 estimates the three-dimensional shape of the irradiation target 9 based on information indicating the phase at which one or more ignitions occurred, obtained for each pixel obtained by the cluster estimation process, irradiation setting information, and system positional relationship information.

The three-dimensional measuring device 3 may store the scanning period T in advance in the storage unit 34, or may obtain the peak position by, for example, frequency converting the event time series. The process of obtaining the peak position by frequency converting the event time series is performed, for example, by the control unit 31 or the control unit 31a. More specifically, the process of obtaining the peak position by frequency converting the event time series is performed, for example, by the cluster estimation unit 319. More specifically, the process of obtaining the peak position by frequency converting the event time series is performed, for example, by the ignition phase estimation unit 314.

The event detection signal may pass through a noise removal filter or the like after it is output by the event vision-based camera 2 and before it is input to the three-dimensional measuring device 3. In such a case, the three-dimensional measuring device 3 acquires an event detection signal with even more reduced noise, further improving the accuracy of estimating the shape of the irradiation target 9.

The three-dimensional measurement system 100 may include multiple event vision-based cameras 2. In such a case, it is also possible to estimate the angle-dependent response function of the irradiated target 9 to the electromagnetic waves based on the result of estimating the phase of the laser irradiation for ignition caused by reception of the laser by the three-dimensional measurement device 3. This is because, since there are multiple event vision-based cameras 2, it is possible to obtain information on reflected light reflected from the same location but in different directions. The process of estimating the response function is performed by the control unit 31 or the control unit 31a. More specifically, the process of estimating the response function may be performed by, for example, the shape estimation unit 315.

In addition, when the three-dimensional measurement system 100 is equipped with multiple event vision-based cameras 2, it is possible to estimate the shape of the irradiation target 9 more robustly than when the three-dimensional measurement system 100 is equipped with one event vision-based camera 2.

Note that when the three-dimensional measurement system 100 includes multiple event vision-based cameras 2, it is not necessarily necessary to use a template function. When the three-dimensional measurement system 100 includes multiple event vision-based cameras 2, instead of the process of obtaining the strength of cross-correlation between the template function and the event time series, a process of obtaining the strength of cross-correlation between the event time series generated by each event vision-based camera 2 may be executed.

The arrangement of the illumination device 1, event vision-based camera 2, and illumination target 9 does not necessarily have to be as shown in FIG. 1. The illumination device 1, event vision-based camera 2, and illumination target 9 may form an angle of 90 degrees or more with the line of sight of the illumination device 1 and event vision-based camera 2. In such a case, the event vision-based camera 2 receives backscattered light.

When the refractive index is non-uniform inside the irradiation target 9, the shape estimation unit 315 may estimate the shape of the irradiation target 9 by numerical analysis using Maxwell's equations or the like, using information indicating the distribution of the refractive index inside the irradiation target 9 (hereinafter referred to as "refractive index distribution information"). The refractive index distribution information may be stored in advance in the storage unit 34, or may be input by the user via the input unit 32.

The irradiation device 1 and the three-dimensional measurement device 3 may be implemented using multiple information processing devices that are communicatively connected via a network.

Up to this point, the three-dimensional measurement system 100 has been described using an example in which three-dimensional measurement is performed by irradiating a laser, but the three-dimensional measurement can be performed by irradiating electromagnetic waves and does not necessarily have to be performed by irradiating a laser.

Up to this point, the irradiation device 1 has been described as a device that irradiates electromagnetic waves by changing the angle at which the electromagnetic waves are irradiated. However, the irradiation device 1 may irradiate electromagnetic waves by changing the position rather than the angle at which the electromagnetic waves are irradiated. For example, the irradiation device 1 may be a liquid crystal or DLP (Digital Lighting Projection) type projector, and in such a case, the pixels or lines may be turned on in sequence for each pixel or each line, so that the condition that different pixels or lines are turned on at different phases is satisfied. Even with such a change in the irradiation position, it is possible to perform the scanning illustrated in FIG. 2. Furthermore, the irradiation device 1 may irradiate electromagnetic waves by changing not only the position but also the angle and position, so that the condition that different sets of angles and positions are different at different phases is satisfied. Even in this case, it is possible to perform the scanning illustrated in FIG. 2.

In this way, any device that irradiates electromagnetic waves by changing the angle or position at which the electromagnetic waves are irradiated so that the irradiation device 1 scans the scanning target at a predetermined cycle can be used.

When the irradiation device 1 irradiates electromagnetic waves by changing the angle or position, not necessarily the angle, estimating the phase means estimating the angle or position of the irradiation of the electromagnetic waves. Therefore, the ignition phase estimation unit 314 that estimates the phase estimates the state of the irradiation of the electromagnetic waves according to the state of the irradiation of the electromagnetic waves of the irradiation device 1. For example, if the irradiation device 1 is a device that radiates electromagnetic waves by changing only the angle, the ignition phase estimation unit 314 is a device that estimates the angle of irradiation. If the irradiation device 1 is a device that radiates electromagnetic waves by changing the position, the ignition phase estimation unit 314 estimates the position of irradiation. If the irradiation device 1 is a device that radiates electromagnetic waves by changing the angle and position, the ignition phase estimation unit 314 estimates the angle and position of irradiation. In this way, the ignition phase estimation unit 314 estimates the angle or position at which the light detected by each pixel is irradiated from the irradiation device 1 based on the correlation intensity time series.

In addition, instead of obtaining an event time series by executing an event time series generation process, the event time series acquisition unit 312 may acquire an event time series of each pixel obtained by executing an event time series generation process by an external device. In addition, the ignition phase estimation unit 314 is an example of an irradiation situation estimation unit.

All or part of the functions of the irradiation device 1 and the three-dimensional measurement device 3 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. The program may be transmitted via a telecommunications line.

The above describes an embodiment of the present invention in detail with reference to the drawings, but the specific configuration is not limited to this embodiment, and includes designs that do not deviate from the gist of the present invention.

100...3D measurement system, 1...irradiation device, 2...event vision-based camera, 3...3D measurement device, 11...control unit, 12...user interface, 13...light source, 14...storage unit, 21...light receiving unit, 210...event element, 31...control unit, 32...input unit, 33...communication unit, 34...storage unit, 35...output unit, 311...event information acquisition unit, 312...event time series acquisition unit, 313...event correlation strength acquisition unit, 314...firing phase estimation unit, 315...shape estimation unit, 316...communication control unit, 317...storage control unit, 318...output control unit, 319...cluster estimation unit, 91...processor, 92...memory, 93...processor, 94...memory, 9...irradiation target

Claims (6)

an event time series acquisition unit that acquires, for each pixel of the event vision-based camera, an event time series that is a time series generated based on the output of an event vision-based camera that detects scattering of the electromagnetic waves irradiated by an irradiation device that irradiates an electromagnetic wave onto a measurement object and scans the electromagnetic wave at a predetermined period, the event time series being a time series indicating the presence or absence of an ignition, which is a phenomenon that generates an event detection signal indicating that a change in luminance of a pixel of the event vision-based camera has been detected; and
an event correlation strength acquisition unit that acquires, for each pixel, a correlation strength time series that indicates a time change in a strength of a cross-correlation between the event time series and a template function that is a predetermined function of the period;
an irradiation situation estimation unit that estimates an angle or a position at which light detected by each of the pixels is irradiated from the irradiation device based on the correlation intensity time series;
a shape estimation unit that estimates a three-dimensional shape of the measurement object based on a result of the estimation by the irradiation situation estimation unit;
A three-dimensional measuring device comprising: The three-dimensional measurement device according to claim 1, wherein the template function is a function that represents a Gaussian function that is repeated at the period. Each value of the half-width and the peak time of the Gaussian function is a value that is predetermined according to the amount of noise generated in the event detection signal and the latency, which is a difference between the actual time when the luminance change of the electromagnetic wave incident on the event vision-based camera occurs and the time recorded in the event detection signal and which varies probabilistically.
The three-dimensional measuring apparatus according to claim 2 . The greater the amount of noise, the wider the half-width.
The three-dimensional measuring apparatus according to claim 3 . an event time series acquisition step for acquiring an event time series for each pixel of the event vision-based camera, the event time series being a time series generated based on the output of an event vision-based camera that detects scattering of the electromagnetic waves irradiated by an irradiation device that irradiates an electromagnetic wave onto a measurement object and scans the electromagnetic wave at a predetermined period, the event time series being a time series indicating the presence or absence of an ignition, which is a phenomenon that generates an event detection signal indicating that a change in luminance of a pixel of the event vision-based camera has been detected;
an event correlation strength acquisition step of acquiring, for each pixel, a correlation strength time series indicating a time change in a strength of a cross-correlation between the event time series and a template function that is a predetermined function of the period;
an irradiation situation estimation step of estimating an angle or a position at which the electromagnetic wave detected by each of the pixels is irradiated from the irradiation device based on the correlation strength time series;
a shape estimation step of estimating a three-dimensional shape of the measurement object based on a result of the estimation in the irradiation situation estimation step;
A three-dimensional measurement method having the above structure. A program for causing a computer to function as the three-dimensional measuring apparatus according to any one of claims 1 to 4 .

Images