JP7583527B2 - Material Identification Device - Google Patents

Description

Application of Article 30, Paragraph 2 of the Patent Act (1) Published on February 19, 2019, in the Proceedings of the 2019 Spring Meeting of the Acoustical Society of Japan (Abstracts and Papers on CD-ROM), pp. 1419-1420, presentation number 2-2-5 (2) Published on February 19, 2019, in the Proceedings of the 2019 Spring Meeting of the Acoustical Society of Japan (Abstracts and Papers on CD-ROM), pp. 715-716, presentation number 2-3-6 (3) Published on June 18, 2019, at the website address https://blog.bitis.co.jp/ jp/column-blog2-0, (4) Published on October 1, 2019 in Ultrasonic Techno, September/October 2019 issue, pages 9-14, published by Japan Industrial Publishing.

The present invention relates to a material identification device that can obtain information about the material that constitutes an object, and in particular to a material identification device that can identify the degree of hardness of an object.

A head-mounted device emits a localization ultrasonic pulse from an ultrasonic transmitter and receives the reflection from an obstacle with a two-channel ultrasonic receiver. The device uses descending FM ultrasonic waves as the localization ultrasonic pulse and extends the reflected sound by 50 times in time so that it can be heard as an audible sound. Research is being conducted to investigate the recognition characteristics of human echolocation in imitation of bats (Non-Patent Document 1).

There is also research that uses a directional ultrasonic transmitter/receiver to determine the frequency characteristics of the reflection coefficient from the sound reflected from an object, and shows that such characteristics change for plywood and plasterboard walls (Non-Patent Document 2). However, even if the reflection coefficient of the reflected sound changes depending on the object, there is no description of a method for determining the material of the target from the reflected sound. In addition, since the directionality of ultrasonic waves is assumed, nothing is disclosed about the characteristics or trends of the reflection coefficient of reflected sound, including measurements using a non-directional ultrasonic transmitter.

Recently, there have been many technologies that use cameras and laser radar to recognize objects and detect the distance and position of objects ahead (for example, Patent Document 1). However, object recognition devices that use visible light, such as cameras and laser radar, cannot identify the material of an object, and there may be cases where it is not easy to grasp the substance and material of an identified object. Furthermore, for applications where it is important to be mindful of the leakage of personal information, it may be necessary to handle images with care, and there may be cases where using object recognition devices that use visible light is undesirable.

Tadayuki Sasaki, Tatsuya Ifukube, "Obstacle Recognition Method Modeled on Bat Echolocation" (January 30, 1987), Noise Research Society Document No. H87-01-7, Hearing Research Society Document No. H-87-7 Masahiro Harazono, Kosuke Nogo, "Measurement of ultrasonic reflection coefficient of flat panels in real environments," Journal of the Acoustical Society of Japan, 72(9), 544550 (2016)

JP 2007-187618 A

The present invention has been made in consideration of the above-mentioned background art, and aims to provide a material identification device that can obtain information about the material that constitutes an object using ultrasound.

To achieve the above object, the material identification device of the present invention comprises an ultrasonic transmitter that generates an ultrasonic transmission wave, an ultrasonic receiver that detects a reflected wave of the ultrasonic wave from the ultrasonic transmitter, and a processing device that analyzes and processes the waveform of the reflected wave, the ultrasonic transmitter generates a transmission wave whose frequency changes according to a predetermined rule, and the processing device determines a material identification index that reflects the degree of hardness of the object based on the waveform of the reflected wave.

In the above-mentioned material identification device, the ultrasonic transmitter generates a transmission wave whose frequency changes according to a predetermined rule, so that the reflected wave detected by the ultrasonic receiver contains information about the material that constitutes the object that formed the reflected wave (e.g., information about the hardness of the object) as a response to various frequencies. The processing device determines a material identification index that reflects the degree of hardness of the object based on the waveform of the reflected wave, and such a material identification index is information about the hardness, etc., of the material that constitutes the object, and serves as a reference for identifying the material of the object.

According to a specific aspect of the present invention, in the above-mentioned material identification device, the material identification index is information reflecting at least one of the hardness, density, and thickness of the object. In this case, the material identification index includes information on not only the hardness of the object, but also the density and thickness, and the state of the object can be accurately identified using the material identification index.

According to yet another aspect of the present invention, the processing device compares the target waveform pattern obtained for the target with multiple reference waveform patterns obtained for multiple known types of substances, and identifies the substance type as a material identification index for the target based on the degree of similarity, with respect to the waveform pattern of the reflected wave. In this case, it becomes possible to determine which of multiple known types of substances the target that formed the reflected wave is.

According to yet another aspect of the present invention, the processing device compares the frequency spectrum of the reflected waves as the waveform patterns of the reflected waves. In this case, the reflected waves as responses to various frequencies can be classified in terms of frequency distribution patterns, and the reliability of the material identification index can be increased in terms of identifying the material.

According to yet another aspect of the present invention, the processing device acquires one short-time spectrum with a time width less than twice the time width of the transmitted wave, or collects multiple short-time spectra with a time width narrower than the time width of the transmitted wave at a predetermined time difference, and extracts features common to the multiple short-time spectra. In this case, focusing on the short-time spectrum makes it easier to extract and classify features, and simplifies the judgment process.

According to yet another aspect of the present invention, the processing device detects the distance and direction to an object corresponding to the material identification indicator. In this case, it is assumed that some method for identifying the direction of the reflected wave is used. By checking the distance and direction to an object for which a material identification indicator has been determined, information regarding the placement of the object, including information regarding hardness, etc., can be obtained.

According to yet another aspect of the invention, the processing device detects a material identification indicator for at least one of a plurality of objects that are in overlapping orientations but at different distances. For example, information regarding the degree of hardness of an object that covers a rear object can be obtained, or information regarding the degree of hardness of an object that is covered by a front object can be obtained.

According to yet another aspect of the invention, the processing device detects material identification indicators for an object that transmits ultrasound and an object behind it.

According to yet another aspect of the present invention, multiple ultrasonic receivers are arranged at a distance from the ultrasonic transmitter, and the processing device extracts reflected waves detected by the multiple ultrasonic receivers that have a high correlation with the transmitted waves. In this case, noise can be reduced from the reflected waves, and the reliability of the determination of the material identification index can be increased.

According to yet another aspect of the present invention, the processing device determines the spatial arrangement pattern by mapping objects corresponding to the material identification index. In this case, the arrangement state of the materials in the space of interest can be understood as the spatial arrangement pattern.

According to yet another aspect of the present invention, the processing device tracks changes in the spatial arrangement pattern over time. In this case, it is possible to grasp state changes and dynamic phenomena such as the movement or stationary state of the target object, and this can be applied to the monitoring of various special events such as accidents, situations requiring assistance, and crime occurrences.

FIG. 1 is a block diagram illustrating a configuration of a material identifying apparatus according to the present invention. FIG. 1 is a conceptual diagram illustrating an example of installation of a material identifying device. FIG. 2 is a perspective view illustrating an example of the arrangement of an ultrasonic transmitter and an ultrasonic receiver. 1A and 1B are diagrams illustrating a transmission wave emitted from a sound wave transmitter. (A) shows an example of a reflected wave received by an ultrasonic receiver, (B) shows a waveform obtained by cross-correlating the reflected wave with the transmitted wave, and (C) shows a specific arrangement of a target when the reflected wave shown in (A) is obtained. 13A to 13C are conceptual diagrams illustrating a method for calculating correlation function values. FIG. 1 is a conceptual diagram illustrating a method for calculating a time spectrum pattern. (A) to (D) show time spectrum patterns obtained by conducting experiments in which a transmitted wave was applied to various samples and a reflected wave was detected. (A) shows a time spectrum pattern obtained by performing a different experiment, and (B) shows a pattern obtained by subtracting it from the case of iron. FIG. 10 is a diagram for explaining the arrangement of ultrasonic transmitting/receiving units and samples in the experiment shown in FIG. 1A to 1C are charts showing the results of measuring the time spectrum pattern of a low-density urethane foam. 13A and 13B are diagrams illustrating an experiment conducted on the transparency of an ultrasonic transmitting and receiving device. 10 is a flowchart illustrating an overview of the operation of the material identifying apparatus. 10 is a flowchart illustrating details of a substance detection operation.

The following describes one embodiment of the material identification device with reference to the drawings.

Figure 1 is a block diagram showing an embodiment of a material identification device. The material identification device 100 includes an ultrasonic wave detection transmitting/receiving device 20 that transmits and receives ultrasonic waves, and a processing device 80 that operates the ultrasonic wave detection transmitting/receiving device 20 to analyze and process the waveform of the reflected wave RW, which is a response to the transmitted wave SW.

The ultrasonic wave detection transmitting/receiving device 20 has an ultrasonic transmitter 31, first, second, third, and fourth ultrasonic receivers 41, 42, 43, and 44, an amplifier device 50, and a microcomputer 60. The ultrasonic receivers 41 to 44 are arranged at a distance from the ultrasonic transmitter 31 in a predetermined direction, as will be described in detail later. The ultrasonic transmitter 31 and the ultrasonic receivers 41 to 44 are fixed integrally to a support 27, and constitute an ultrasonic transmitting/receiving unit 28.

2, the ultrasonic wave detection transmitting/receiving device 20 is housed in a case 20a so that the ultrasonic wave transmission/reception unit 28 is exposed, and is installed facing the monitoring target via a mount device or other fixture 20b. In the illustrated case, the ultrasonic wave detection transmitting/receiving device 20 is fixed to the ceiling side of the indoor ID, and is positioned so that humans OB1, furniture and other non-biological objects OB2, etc. present in the indoor ID can be observed while analyzing them using ultrasound. In the illustrated case, a single ultrasonic wave detection transmitting/receiving device 20 is installed in the indoor ID, but it is also possible to install multiple ultrasonic wave detection transmitting/receiving devices 20 with different operation timings and ultrasonic characteristics, for example. In addition, the orientation and position of the ultrasonic wave detection transmitting/receiving device 20 can be changed by the fixture 20b. It goes without saying that the ultrasonic wave detection transmitting/receiving device 20 can be installed outdoors, not limited to the indoor ID as illustrated.

Returning to FIG. 1, the ultrasonic transmitter 31 is an ultrasonic speaker with wide directionality. The ultrasonic transmitter 31 is a combination of a voice coil and a piezoelectric element, and converts an electrical signal into ultrasonic waves. The ultrasonic transmitter 31 can generate sound waves of, for example, 20 Hz to 100 kHz, but in this embodiment, ultrasonic waves are emitted using a range of about 25 kHz to 100 kHz.

The first ultrasonic receiver 41 is an ultrasonic microphone with wide directionality. The first ultrasonic receiver 41 is a microphone that uses MEMS technology and converts ultrasonic waves into electrical signals. The first ultrasonic receiver 41 may be a conventional capacitor type microphone, or may be one that uses a piezoelectric element. The second to fourth ultrasonic receivers 42 to 44 have a similar structure to the first ultrasonic receiver 41. The first to fourth ultrasonic receivers 41 to 44 can detect sound waves of, for example, 20 Hz to 100 kHz, but in this embodiment, ultrasonic detection is performed using a range of approximately 25 kHz to 70 kHz.

With reference to FIG. 3, the positional relationship between the ultrasonic transmitter 31 and the first to fourth ultrasonic receivers 41 to 44 will be described. For convenience, the center of the front surface of the ultrasonic transmitter 31 is assumed to be at the origin O of the XYZ coordinate system. The first and second ultrasonic receivers 41, 42 are arranged at a predetermined distance in the X direction from the ultrasonic transmitter 31, and the third and fourth ultrasonic receivers 43, 44 are arranged at a predetermined distance in the Y direction from the ultrasonic transmitter 31. The X direction corresponds to the horizontal direction on the paper, but does not correspond to the horizontal direction when the ultrasonic detection transmitting/receiving device 20 is installed at an angle. The X direction corresponds to the vertical direction on the paper, but does not correspond to the vertical direction when the ultrasonic detection transmitting/receiving device 20 is installed at an angle. When considered on the XY plane, the distance to the target OB can be known taking into account the speed of sound from the average value of the reception times ΔT1 and ΔT2 of the reflected waves RW1 and RW2 from the same target received by the first and second ultrasonic receivers 41 and 42 based on the transmission time T0 of the transmission wave SW from the ultrasonic transmitter 31, and the direction of the target OB can be known taking into account the speed of sound from information such as the difference between the reception times ΔT1 and ΔT2 and the interval between the first and second ultrasonic receivers 41 and 42. When considered on the YZ plane, the distance to the target OB can be known taking into account the speed of sound from the average value of the reception times ΔT3 and ΔT4 of the reflected waves RW3 and RW4 from the same target received by the third and fourth ultrasonic receivers 43 and 44 based on the transmission time T0 of the transmission wave SW from the ultrasonic transmitter 31, and the direction of the target OB can be known taking into account the speed of sound from information such as the difference between the reception times ΔT3 and ΔT4 and the interval between the third and fourth ultrasonic receivers 43 and 44. In the case of a real three-dimensional space, the ultrasonic transmitter 31 and the first ultrasonic receiver 41 are considered as two foci, and a first elliptical surface is considered in which the sum of the distances from these two foci corresponds to the distance corresponding to the reception time ΔT1, and the ultrasonic transmitter 31 and the second ultrasonic receiver 42 are considered as two foci, and a second elliptical surface is considered in which the sum of the distances from these two foci corresponds to the distance corresponding to the reception time ΔT2. The object OB exists on or near the intersection line or circumference of these elliptical surfaces. Similarly, the ultrasonic transmitter 31 and the third ultrasonic receiver 43 are considered as two foci, and a third elliptical surface is considered in which the sum of the distances from these two foci corresponds to the distance corresponding to the reception time ΔT3, and the ultrasonic transmitter 31 and the fourth ultrasonic receiver 44 are considered as two foci, and a fourth elliptical surface is considered in which the sum of the distances from these two foci corresponds to the distance corresponding to the reception time ΔT4. The object OB exists on or near the intersection line or circumference of these elliptical surfaces. As a result, the object OB exists on or near the intersection point of the first to fourth elliptical surfaces as described above. In addition, when the target OB has a spatial spread, the time width of the reflected waves RW1 to RW4 becomes wider than the time width of the original transmission wave SW, and the reception times ΔT1 to ΔT4 have a certain width, which also provides information about the size or magnitude of the target OB. In the following, for simplicity, the placement detection in the XY plane may be described with a focus on the ultrasonic transmitter 31 and the first and second ultrasonic receivers 41, 42.

The amplifier device 50 shown in FIG. 1 has a transmission amplifier 55 that amplifies the electrical signal input to the ultrasonic transmitter 31 to a desired intensity, and receiving amplifiers 51-54 that amplify the electrical signals output from the first to fourth ultrasonic receivers 41-44. The transmission amplifier 55 is a high-power, high-speed amplifier, and the receiving amplifiers 51-54 are high-speed amplifiers.

The microcomputer 60 includes a DA converter 65, first to fourth AD converters 61 to 64, a microprocessor 68, and a memory 69. The DA converter 65 receives commands from the microprocessor 68 and sequentially converts a series of digital signals in a predetermined format corresponding to the transmission wave SW into analog signals at the corresponding timing. The first to fourth AD converters 61 to 64 sequentially convert a series of analog signals corresponding to the reflected waves RW received from the first to fourth ultrasonic receivers 41 to 44 via the receiving amplifiers 51 to 54 into digital signals in a predetermined format. At this time, the microprocessor 68 temporarily distributes and stores the outputs of the first to fourth AD converters 61 to 64 in the memory 69 with time-related information attached, and outputs them to the processing device 80 at the appropriate timing.

In the ultrasonic detection transmitting/receiving device 20, the combination of the ultrasonic transmitter 31, the transmitting amplifier 55, and the DA converter 65 is called the ultrasonic transmitting device 25. Also, the combination of the first ultrasonic receiver 41, the receiving amplifier 51, and the first AD converter 61 is called the first ultrasonic receiving device 21, the combination of the second ultrasonic receiver 42, the receiving amplifier 52, and the second AD converter 62 is called the second ultrasonic receiving device 22, the combination of the third ultrasonic receiver 43, the receiving amplifier 53, and the third AD converter 63 is called the third ultrasonic receiving device 23, and the combination of the fourth ultrasonic receiver 44, the receiving amplifier 54, and the fourth AD converter 64 is called the fourth ultrasonic receiving device 24.

The processing device 80 is, for example, a personal computer, and includes a calculation device 81, a storage device 82, an input/output device 83, a communication device 84, and a display 86. The processing device 80 is connected to the ultrasound detection transmitting/receiving device 20 via a communication cable 91 so as to be capable of digital communication. The processing device 80 is installed in a position away from the indoor ID shown in FIG. 2. In other words, the processing device 80 remotely manages the ultrasound detection transmitting/receiving device 20.

The arithmetic unit 81 is the part that manages the operation of the processing device 80, controls the operation of the ultrasonic wave detection transmitting/receiving device 20, and obtains information on the monitored object using ultrasonic waves. The storage device 82 is the part that stores the programs that operate the processing device 80 itself and the ultrasonic wave detection transmitting/receiving device 20, and can also store information obtained on the monitored object. The input/output device 83 is the part that is operated by the operator of the processing device 80, and can also output information related to the processing results by the arithmetic unit 81. The communication device 84 transmits and receives monitoring information, management information, command information, etc. to and from an external management server via a public communication network such as the Internet or a dedicated communication network that has been independently constructed. The display 86 operates under the control of the arithmetic unit 81, and is the part that displays the material spatial arrangement information, etc. obtained for the monitored object, which will be described later.

The calculation device 81 operates the ultrasonic wave detection transmitting/receiving device 20 via the microcomputer 60, and generates a transmission wave (hereinafter also referred to as FM type ultrasonic wave) SW whose frequency changes according to a predetermined rule from the ultrasonic transmitter 31 to the surroundings. The waveform of the transmission wave SW is, for example, one in which the frequency increases linearly with time, but is not limited to this, and one in which the frequency decreases linearly or one in which the frequency changes nonlinearly can be used. The pattern of the transmission wave SW is stored in the data area 82a of the storage device 82 as ultrasonic transmission data US0. The ultrasonic transmission data US0 is not limited to one corresponding to a single waveform, and multiple waveforms can be prepared in advance and waveform data according to the purpose and situation can be selected from among them. The calculation device 81 operates the ultrasonic wave detection transmitting/receiving device 20 via the microcomputer 60, and captures the reflected wave RW as a response to the transmission wave SW as ultrasonic reflection data US1 to US4 for each of the first to fourth ultrasonic receivers 41 to 44 and stores it in the data area 82b of the storage device 82. The arithmetic device 81 extracts waveform data having a high correlation with the transmission wave SW from the ultrasonic reflection data US1 to US4, determines the distance and direction to the object (i.e., spatial arrangement), and stores it in the data area 82c of the storage device 82 as a labeled object, while analyzing the frequency distribution of the waveform data to determine a material identification index reflecting the degree of hardness of the object, and stores this material identification index in the data area 82c as information associated with the labeled object. The information stored in the data area 82c is material spatial arrangement information combining the spatial arrangement and the material identification index for the multiple labeled objects that form the reflected wave RW. In other words, the arithmetic device 81 determines the spatial arrangement pattern by mapping the object corresponding to the material identification index. In this case, the arrangement state of the material in the space of interest (actually the arrangement state of the material identification index reflecting the degree of hardness) can be grasped as a spatial arrangement pattern. In this case, the spatial arrangement of the multiple labeled objects may include the spatial size. The arithmetic device 81 causes the ultrasonic wave detection transmitting/receiving device 20 to periodically emit a transmission wave SW at a predetermined timing, and the material spatial arrangement information is stored in the data area 82c in sequence as monitoring information over time. The monitoring information over time is a collection of continuous frame data, similar to a moving image. The arithmetic device 81 can determine the identity of the labeled object between frame data of the monitoring information over time, and if the amount of displacement of objects of similar material or size is less than a predetermined amount, the arithmetic device 81 can label the objects as the same object, and track or trace the movement of the labeled object according to the transition of the frame data, and store such tracking data in the data area 82d as object movement information. At this time, the arithmetic device 81 can classify the importance of the object to be tracked according to its material identification index or size, and can remove it from the monitoring target. In the above, the arithmetic device 81 tracks the change over time in the spatial arrangement pattern of the object characterized by the material identification index stored in the data area 82c. In this case, it is possible to grasp state changes and dynamic phenomena such as the movement or stationary state of the target object. If the movement of the tracked object being monitored exhibits abnormal behavior or if it moves to an unacceptable location, the computing device 81 can display this on the display 86, and if deemed necessary, can report this to an external management server via the communication device 84. When installed in an area of a public facility where the manager is not present or in a private space, the material identification device 100 can be used to monitor various special events such as accidents, situations requiring assistance, and crimes.

In the above explanation, a processing device 80 is provided separately from the microcomputer 60 to process the ultrasonic reflection data US1 to US4, obtain material spatial arrangement information, and track labeled objects, but all or part of these functions of the processing device 80 can be incorporated into the microcomputer 60, and operations can be completed by the microcomputer 60 alone, for example.

Figure 4 (A) is a diagram explaining an example of the generation of a transmission wave SW, where the horizontal axis indicates time and the vertical axis indicates frequency. In the illustrated example, a single transmission wave SW has a width of about 1 ms, and such a transmission wave SW is repeated in a 100 ms cycle. A single transmission wave SW is an FM type ultrasonic wave whose frequency decreases, and in the illustrated example, the frequency decreases abruptly but continuously from 80 kHz to 25 kHz. Figure 4 (B) shows the waveform of a single transmission wave SW, and it can be seen that the frequency decreases over time. Note that the transmission wave SW is not limited to an ultrasonic wave whose frequency gradually decreases as shown in Figure 4 (A) and the like, but may be an ultrasonic wave whose frequency gradually increases, and there may be a region in which the frequency is kept constant in the initial time zone or the intermediate time zone. Also, in FIG. 4(A), the transmission wave SW is repeated at a cycle of 100 ms, but the repetition cycle of the transmission wave SW can be set appropriately according to the time it takes to obtain an echo depending on the size of the monitored object and its space as shown in FIG. 2.

Figure 5 (A) illustrates the reflected waves RW1, RW2 received by the first and second ultrasonic receivers 41, 42 or the ultrasonic reflection data US1, US2 obtained therefrom, and Figure 5 (B) illustrates a waveform obtained by cross-correlating the reflected waves RW1, RW2 or the ultrasonic reflection data US1, US2 obtained therefrom with the transmission data of the transmission wave SW. Figure 5 (C) illustrates an example of the arrangement of objects when the reflected waves RW1, RW2 shown in Figure 5 (A) etc. are obtained, i.e., an example of an object arranged in front of the ultrasonic transmission/reception unit 28 shown in Figure 3. In this case, the first object PO1 located at the position (0,0.75) is cylindrical with a diameter of 5 mm extending in the Z direction, the second object PO2 located at the position (-0.5,1) is cylindrical with a diameter of 30 mm extending in the Z direction, the third object PO3 located at the position (0.4,1.2) is cylindrical with a diameter of 30 mm extending in the Z direction, and the fourth object PO4 located at the position (0,1.6) is cylindrical with a diameter of 50 mm extending in the Z direction. The cross-correlated waveform shown in FIG. 5(B) shows peaks RP11 to RP4 corresponding to the first to fourth objects PO1 to PO4. In other words, the distance and direction to the first to fourth objects PO1 to PO4 can be determined by using the peaks RP11 to RP4.

As shown in Figure 5 (B), by taking the cross-correlation between the reflected wave RW1 or reflected wave RW2 and the transmitted wave SW, the reflected waves RW1 and RW2 that have a waveform similar to the transmitted wave SW are emphasized, and waveforms that differ from the transmitted wave SW are reduced. In other words, the noise in the reflected waves RW1 and RW2 is reduced from the waveform shown in Figure 5 (A) obtained by the correlation function because it is different from the transmitted wave SW, and the reflected waves RW1 and RW2 from objects such as objects PO1 to PO4 are enhanced because the transmitted wave SW is similar.

6(A) is a conceptual diagram explaining the reflected wave RW or ultrasonic reflection data obtained from it, and FIG. 6(B) is a conceptual diagram explaining the transmitted wave SW or the corresponding ultrasonic transmission data. As shown by the dotted line, the reflected wave RW is sampled at regular intervals and converted into digital data (specifically, any of the ultrasonic reflection data US1 to US4). The sampling period is, for example, 1 μs, but is not limited to this. The sampling period of the transmitted wave SW is the same as that of the reflected wave RW at the stage of digital data (specifically, ultrasonic transmission data US0) before being converted into analog. When taking the cross-correlation between the reflected wave RW and the transmitted wave SW, in the first target section ID0 of the reflected wave RW (corresponding to the time width IS0 of the transmitted wave SW), N multiplications are performed on the sample value of the reflected wave RW at each time point in section ID0 and the sample value of the transmitted wave SW at each time point, and the sum of all N multiplication values is normalized by the square root of the sum of squares of each waveform to be used as the correlation function value at time t0. Next, as shown in Figure 6 (C), the transmitted wave SW is shifted by one point, and the sample values of the reflected wave RW at each time point in section ID1 are multiplied N times by the sample values of the transmitted wave SW at each time point, and all of these N multiplication values are added together and normalized by the square root of the sum of the squares of each waveform to obtain the correlation function value at time t1. This process is repeated to obtain correlation function values one after another, and when these values are plotted on the time axis, a correlation function waveform is obtained. The above is an example of a calculation to obtain the cross-correlation between the reflected wave RW and the transmitted wave SW, and the correlation can be obtained by various other calculation methods.

As described above, when there are not many objects PO1-PO4 or when the objects PO1-PO4 are relatively far apart, as shown in Figure 5 (B), the distance and direction to the first to fourth objects PO1-PO4 can be determined. However, when the multiple objects being observed are close to each other or there are many of them, there is a possibility of interference. For this reason, correlation waveforms are used to calculate the distance and direction. For example, when the receivers are on the left and right, the distance and direction to the cylinder, which is an object, are calculated from the correlation waveform (RP1R) between the transmitted wave SW and the right received wave (RW1R), and the correlation waveform (RP1L) between the transmitted wave SW and the left received wave (RW1L). Specifically, the distance from the object was calculated from the average ((TR+TL)/2) of the time difference (TR) from the transmitted wave SW to the right correlation waveform RP1R and the time difference (TL) from the transmitted wave SW to the left correlation waveform RP1L, while the direction was calculated from the difference (ΔT) between the time difference to the right (TR) and the time difference to the left (TL). In other words, if TR and TL are large, the distance from the transmitter to the cylinder is farther, and if the difference ΔT between TR and TL is the same, the cylinder is in the front direction, and the larger the difference ΔT, the further away it is from the front direction.

Below, a method for analyzing the frequency distribution of the waveform data of the reflected wave RW will be described. For data regions extracted in which the reflected wave RW1 has a correlation with the transmitted wave SW, an FFT (Fast Fourier Transform) is performed on the waveform data of interest in this data region to analyze the Fourier transform spectrum, i.e., frequency distribution. Note that an FFT can also be performed on the entire region of the data of the reflected wave RW, which is the received wave. The analysis of frequency distribution is not limited to FFT, and various methods such as wavelet transform can be used.

Let us assume that the reflected wave RW has a waveform as shown in FIG. 7(A). For example, if the reflected wave RW is sampled over a period of 100 ms at a period of 1 μs immediately after the transmission wave SW is emitted, 100×1000 points, i.e., 100,000 sample values, are obtained. From these sample values, for example, the first 100 sample values (shown as frame length in the figure) are subjected to FFT to obtain a spectrum SP1 (the leftmost side of FIG. 7(B)), then shifted by 50 points (shown as frame period in the figure) corresponding to a time shorter than the frame length, and the spectrum SP1 is obtained from the next 100 sample values by FFT (the next left side of FIG. 7(B)), then shifted by another 50 points, and the spectrum is obtained from the next 100 sample values by FFT, and so on. Here, the specific frame period is, for example, 50 μs, and the specific frame length is, for example, 100 μs=0.1 ms. When a spectrum is obtained by the above method, 100,000 sample values are shifted by 50 points each to obtain a spectrum, so 2,000 short-time spectra are obtained. Such short-time spectra are hereinafter referred to as short-time spectra. A collection of short-time spectra can be expressed as a three-dimensional diagram with the time axis as the horizontal axis, frequency as the front to back axis as the depth axis, and power (intensity) as the vertical axis. However, by expressing power (intensity) as "density", it can be displayed as a two-dimensional pattern just like a "voiceprint", and such a pattern with visible density is called a density-type time spectrum pattern. In such a time spectrum pattern, it is predicted that the areas with high density are reflected sounds from objects, and a predicted reflected wave RW having a length of about 1 ms or more has a total of 20 or more short-time spectra obtained every 50 μs over a period of 0.1 ms. Among these, the areas with high density are regarded as the reflected wave RW, and the short-time spectrum pattern there is used as the data for pattern matching. Such short-time spectra or an average of them is also called a local time spectrum pattern or simply a spectrum pattern. Calculating the entire time spectrum of the received wave would require a large amount of calculation, so for simplicity's sake, the waveform with high correlation (waveform of interest) is considered to be the reflected wave RW, and only its time spectrum is used as the pattern matching data. Focusing on the short-time spectrum makes it easier to extract features and classify them, simplifying the judgment process.

A single short-time spectrum can also be used as the time spectrum pattern. For example, the point at which the correlation function is maximized is calculated, and the waveform with the length of the transmission wave SW starting from that point is used as the waveform of interest. A frequency spectrum can be analyzed or calculated in a range 1 to 2 times the length of the transmission wave SW, centered on the waveform of interest, and such a frequency spectrum can be used as a single short-time spectrum.

The short-time spectrum has a time width narrower than that of the transmitted wave SW. For short-time spectra, if the frame length (sample width) is short, the precision of the spectrum (more precisely, the frequency resolution) decreases, making it difficult to identify the material. Conversely, if the frame length (sample width) is long, the time resolution decreases, but the frequency resolution increases. In reality, the reflected wave RW is also about 1 ms, and if the sample width is made wider than that, signals other than the reflected wave will also be picked up, so the frame length (sample width) is set to 2 ms or less.

Figures 8(A) to 8(D) show the frequency spectrum, or time spectrum pattern, of the reflected wave RW obtained by performing an experiment in which the transmitted wave SW was applied to various samples and the reflected wave RW was detected. In order to obtain the time spectrum patterns shown in Figures 8(A) to 8(D), the ultrasonic transmission/reception unit 28 (without the third and fourth ultrasonic receivers 43 and 44) shown in Figure 3 was oriented horizontally, and plates of various materials measuring 25 cm x 30 cm were placed 60 cm in front of the unit. Figure 8(A) shows the case of a 1 mm thick iron plate, Figure 8(B) shows the case of a 2.3 mm thick styrene plate, Figure 8(C) shows the case of a 5 mm thick lauan plywood, and Figure 8(D) shows the case of a 5 mm thick urethane sponge. Note that Figures 8(A) to 8(D) are arranged in order from hardest material to softest material. As is clear from Figures 8(A) to 8(D), the frequency peak patterns formed around 50 kHz differ for each material. In the steel plate of FIG. 8(A), there is a maximum peak at a frequency of 50 kHz, with relatively large peaks around 30 kHz and 70 kHz. In the styrene plate of FIG. 8(B), in addition to the large peak around 50 kHz, there is a maximum peak around 70 kHz, with a relatively large peak below that around 30 kHz. In the lauan plywood of FIG. 8(C), there is no peak around 50 kHz, with a flat peak between 30 kHz and 70 kHz. In the urethane sponge of FIG. 8(D), there is a weakening around 50 kHz, and there is no noticeable peak overall between 30 kHz and 70 kHz, but there is a large fluctuation in amplitude.

Figure 9 (A) shows the time spectrum patterns obtained by performing another experiment on various samples. Also, Figure 10 is a diagram explaining the arrangement of the ultrasonic transmission/reception unit and the samples in the experiment shown in Figure 9 (A). The transmission wave SW output from the wave transmission/reception unit 28 was lowered in frequency from 80 kHz to 20 kHz in the pattern shown in Figure 4 (A). The wave transmission/reception unit 28 was set on a cylindrical support SM installed on the floor surface FL, and the ultrasonic transmission/reception unit 28 (without the third and fourth ultrasonic receivers 43 and 44) was placed above the sample SA. The distance from the ultrasonic transmission/reception unit 28 to the sample SA was 1 m, and the size of the sample SA was 20 cm x 30 cm. The thickness of the sample SA was 1 to 3 mm. Specifically, the sample SA was an iron plate, plywood, vinyl, elastomer, styrene board, and plain rubber, and the hardness values were 100, 96.85, 85, 72, and 25, respectively. In Fig. 9(A), it can be seen that peak waveforms showing differences in hardness are obtained in the frequency ranges shown as changes 1 to 6. For example, in the 25-30 kHz range, the level decreases for softer materials such as styrene board and plain rubber (change 1), while above 35 kHz, the level increases for soft materials such as vinyl (changes 2 to 6). For harder materials such as plywood, the level decreases in the 40-50 kHz range (between changes 3 and 4), but no such decrease is observed for steel plates. Although the elastomer is buried in the figure and difficult to see, it is at a level that is intermediate between hard and soft. Fig. 9(B) is a spectral pattern showing the difference with the spectrum of the steel plate, and it can be seen that the level increases and decreases depending on the frequency for soft and hard materials. However, the material should not be judged only by the local increase or decrease in the level in a specific frequency range, but by the overall difference in the increase and decrease pattern (frequency characteristics).

The above explanation is merely an example. Experiments are conducted in which a transmission wave SW is applied to samples SA of various sizes and hardnesses other than those mentioned above, and the reflected wave RW is detected. The time spectrum patterns of the reflected wave RW obtained are stored, and the processing device 80 can determine various information such as hardness and size of various targets OB by analyzing and learning them in advance.

Figures 11(A) to 11(C) show the reflected sound spectrum from a 2 cm to 10 cm thick urethane foam (50 cm x 50 cm) placed 1 m in front of the ultrasonic transmitter 31. Figure 11(A) shows the time spectrum pattern of a 2 cm thick urethane foam, Figure 11(B) shows the time spectrum pattern of a 5 cm thick urethane foam, and Figure 11(C) shows the time spectrum pattern of a 10 cm thick urethane foam. In general, the ultrasonic waves penetrated the urethane foam and the reflected sound was small, but the 46 to 47 kHz component weakened as the thickness increased. From this, it is inferred that the urethane foam used in this experiment is less dense, so the emitted sound is generally transmitted, but absorption occurs around 46 to 47 kHz. In this way, it is inferred that the focus point differs depending on the material, and conversely, the change in the focus point predicts differences in materials.

As is clear from the above, if the time spectrum patterns of the reflected wave RW are measured in advance for a large number of predicted materials and registered as templates in the storage device 82, the material or material of the object or object OB present in front of the ultrasonic transmission/reception unit 28 can be estimated from the similarity of the time spectrum pattern of the reflected wave RW to the many registered templates. The material or material determined in this manner is a material identification index reflecting the degree of hardness of the object OB. In other words, in the waveform pattern of the reflected wave RW, the local time spectrum pattern (target waveform pattern) obtained for the object or object OB is compared with multiple templates (reference waveform patterns) obtained for multiple known types of materials, and the material type is identified as the material identification index of the object or object OB based on the high degree of similarity. The similarity or similarity of the local time spectrum patterns as described above can be determined by correlation and various other statistical methods. Furthermore, the similarity of the local time spectrum patterns as described above can be determined by using various machine learning methods such as deep learning that construct a multilayered neural network, and pattern identification using AI is also possible.

The material identification index is not limited to identifying the material or material of the target OB. The material identification index may be something like hardness indicating the degree of hardness. The material identification index may be a hardness classification in which hardness is divided into multiple stages. For example, it is possible to classify hard plate-like objects such as (1) iron plates and plywood, (2) transparent vinyl boards and transparent vinyl boards, and (3) soft plate-like objects such as styrene boards and rubber. It has been experimentally confirmed that these classifications (1) to (3) have clearly different slopes of the entire time spectrum pattern (the ratio of the intensity of the low frequency (around 30 kHz) to the intensity of the high frequency (50 to 60 kHz)). The material identification index may be something that distinguishes between categories of materials. For example, it may be possible to distinguish between metal solids, ceramic solids, organic solids, etc. From the viewpoint that organic solids also include biological materials such as meat, it is thought that it is also possible to distinguish between living organisms and objects harder than them. The material identification index may include information on density such as porosity and fiber. Materials with low density, such as sponges and fibers, will clearly have a larger transmitted component, and the time spectrum pattern will be significantly different from that of materials with high density, such as rubber sheets.

Although a detailed explanation is omitted, it is believed that as the thickness of the target object or target OB increases, a downward shift occurs in which the power of the time spectrum pattern of the reflected wave RW decreases overall, and if the power value is calibrated according to the distance to the target OB, it becomes possible to estimate the thickness of the target OB.

12(A) and 12(B) are diagrams for explaining an experiment conducted on the transparency of the ultrasonic wave detection transmitting/receiving device 20. FIG. 12(A) shows a concentration-type time spectrum pattern when a glass plate is placed 100 cm in front of the ultrasonic transmitting/receiving unit 28, and FIG. 12(B) shows a concentration-type time spectrum pattern when a glass plate is placed 100 cm in front of the ultrasonic transmitting/receiving unit 28 and a thin furoshiki cloth is placed 70 cm in front of the ultrasonic transmitting/receiving unit 28 so as to cover the glass plate. As is clear from FIG. 12(B), it can be seen that the presence of the furoshiki cloth and the glass plate can be detected simultaneously. From the above, it can be said that the material identification device 100 of the embodiment can detect objects and people that are hidden by a partition or curtain such as a thin curtain and cannot be visually seen. In addition, the material identification device 100 of the embodiment can detect materials such as glass and mirrors that are visually transparent and difficult to see.

By utilizing the above phenomenon, the processing device 80 can detect a material identification indicator for at least one of multiple targets that have overlapping orientations but different distances, and can detect the material identification indicators of the target through which the ultrasonic waves pass and the target behind it by comparing the transmitted waves SW and the reflected waves RW. Specifically, it is possible to detect not only the presence of a glass surface behind the furoshiki as described above, but also the presence of a hard object behind the urethane foam, and to detect differences in the thickness of the urethane foam due to reflected waves from a hard object behind the urethane foam.

Although a detailed explanation is omitted, since fine particles such as fog and smoke do not generally block ultrasonic waves, the material identification device 100 of the embodiment can be said to be capable of detecting objects or people in places that are difficult to see due to fog or smoke.

As described above, the material identification device 100 of the embodiment does not perform object detection using the visible appearance of a material, and therefore can identify objects with the same pattern but different materials as different objects. In other words, if the material identification device 100 of the embodiment is mounted on a moving body, for example, when the moving body heads toward an object ahead, it is possible to easily predict in advance the impact of a collision with the object ahead.

The material identification device 100 of the embodiment performs measurements using ultrasound, and therefore can be said to be highly reliable in terms of preventing leakage of personal information, etc., from the viewpoint of observing human movements in private spaces such as private rooms and medical facilities.

The operation of the material identification device 100 will be described below with reference to FIG. 13. The processing device 80 obtains information on the arrangement and material of one or more objects OB arranged in front of the ultrasonic transmission/reception unit 28 from the data obtained using the ultrasonic detection transmission/reception device 20 (step S1). In other words, the processing device 80 obtains material spatial arrangement information.

Next, the processing device 80 refers to the storage device 82 and compares the target OB identified by the material spatial arrangement information with past data to determine its identity, labels those determined to be the same object, and tracks or traces the movement of the labeled object or target OB (step S2).

The processing device 80 determines whether an event has occurred that requires an alarm or other notification, such as whether the movement or stopping of the labeled object or target OB is abnormal, or whether the labeled object or target OB is moving to an unacceptable location (step S3). In a hospital or care facility, for example, abnormal behavior would mean a patient or a person receiving care falling, and moving to an unacceptable location would be, for example, the actions of an intruder from outside who has slipped through other security check mechanisms.

If the processing device 80 determines that the movement of the labeled object or target OB is behaving abnormally, or that the labeled object or target OB is moving to an unacceptable location, it displays an alarm on the display 86 or the like, and reports the abnormal situation to an external management server via the communication device 84 (step S4).

The operation of step S1 in FIG. 13 will be described in detail below with reference to FIG. 14. The processing device 80 operates the ultrasonic detection transmitting/receiving device 20, generates a transmission wave SW, which is an FM type ultrasonic wave, from the ultrasonic transmitter 31 in the surrounding area, and captures a reflected wave RW as a response to the transmission wave SW as ultrasonic reflection data for each of the first to fourth ultrasonic receivers 41 to 44 (step S11). After that, the processing device 80 performs correlation processing to extract waveform data that is correlated with the transmission wave SW from the ultrasonic reflection data (step S12). Next, the processing device 80 determines the distance and direction (i.e., spatial arrangement) to the target from the waveform data that is determined to be correlated in step S12, and stores it in the storage device 82 (step S13). Meanwhile, the processing device 80 analyzes the frequency distribution from the waveform data that is determined to be correlated in step S12 using a method such as FFT to obtain a spectrum pattern (step S14). Next, the processing device 80 performs pattern matching for the object whose spectral pattern was obtained in step S14, comparing it with the registered template stored in the storage device 82 (step S15), and determines a material identification index reflecting the degree of hardness of the object, such as estimating the material of the object OB and estimating the hardness level of the object OB (step S16). Finally, the processing device 80 performs integration by combining the spatial arrangement and the material identification index while labeling one or more objects OB corresponding to the reflected wave RW, to form material spatial arrangement information (step S17).

As is clear from the above description, in the material identification device 100 of the above embodiment, the ultrasonic transmitter 31 generates a transmission wave SW whose frequency changes according to a predetermined rule, so that the reflected wave RW detected by the ultrasonic receivers 41-44 contains information about the material that constitutes the target OB that formed the reflected wave RW (for example, information about the hardness of the target OB) as a response to various frequencies. The processing device 80 determines a material identification index that reflects the degree of hardness of the target OB based on the waveform of the reflected wave RW, and such a material identification index is information about the hardness, etc., of the material that constitutes the target OB, and serves as a reference for identifying the material of the target OB.

This invention is not limited to the above embodiment, and can be implemented in various forms without departing from the spirit of the invention. For example, the arrangement of the four ultrasonic receivers 41-44 shown in FIG. 3 is merely an example, and the distance and direction from the ultrasonic transmitter 31 can be changed as appropriate. Also, the spatial arrangement of the target OB can be determined using three ultrasonic receivers without providing four ultrasonic receivers 41-44.

The ultrasonic transmitter 31 can be made directional, and the ultrasonic receivers 41 to 44 can also be made independently directional. In this case, the ultrasonic transmitter 31 and ultrasonic receiver 41 can be moved in a scanning manner to determine the direction of the target OB.

OB... object, OB1... human, OB2... object, PO1-PO4... object, RW... reflected wave, RW1-RW4... reflected wave, SW... transmitted wave, 1-4AD... number, 41-44... ultrasonic receiver, 51-54... receiving amplifier, 61-64... AD converter, 20... ultrasonic detection transmitting/receiving device, 21-24... ultrasonic receiving device, 25... ultrasonic transmitting device, 27... support, 28... ultrasonic transmitting/receiving unit, 31... ultrasonic transmitter, 41-44... ultrasonic receiver, 50... amplifier device, 60... microcomputer, 68... microprocessor, 69... memory, 80... processing device, 81... arithmetic device, 82... storage device, 83... input/output device, 84... communication device, 86... display, 91... communication cable, 100... material identification device

Claims (11)

an ultrasonic transmitter that generates an ultrasonic transmission wave;
an ultrasonic receiver that detects a reflected wave of the ultrasonic wave from the ultrasonic transmitter;
a processor that analyzes and processes the waveform of the reflected wave;
The ultrasonic transmitter periodically generates the transmission wave whose frequency changes according to a predetermined rule,
The processing device determines a material identification index reflecting a degree of hardness of the object based on a waveform of the reflected wave;
The processing device acquires one short-time spectrum having a time width equal to or less than twice the time width of the transmission wave, or acquires a plurality of short-time spectra having a time width narrower than the time width of the transmission wave at a predetermined time difference, and extracts a feature common to the plurality of short-time spectra.
Material Identification Device. The material identification device according to claim 1 , wherein the processing device grasps a change over time in a spatial arrangement pattern of objects characterized by the material identification indicators and sizes corresponding to the common features. 3. The material identification device according to claim 1, wherein the material identification index is information reflecting at least one of the hardness of the object, the denseness, and the thickness. The material identification device according to any one of claims 1 to 3, wherein the processing device compares a target waveform pattern obtained for a target with a plurality of reference waveform patterns obtained for a plurality of known types of materials, and identifies the material type as the material identification index of the target based on the degree of similarity. The material identification device according to claim 4, wherein the processing device compares the frequency spectrum of the reflected wave as the waveform pattern of the reflected wave. 6. The material identifying device according to claim 1, wherein the processing device detects a distance and a direction to an object corresponding to the material identifying indicator. The material identifying device according to claim 6 , wherein the processing device detects the material identifying indicator for at least one of a plurality of targets that have overlapping orientations but different distances. 8. The material identifying device according to claim 7, wherein the processing device detects the material identifying indicator for an object through which ultrasonic waves pass and an object behind the object. The ultrasonic receiver is disposed at a distance from the ultrasonic transmitter.
9. The material identifying device according to claim 6, wherein the processing device extracts reflected waves having a high correlation with the transmitted wave from the plurality of reflected waves detected by the plurality of ultrasonic receivers. The material identifying device according to any one of claims 6 to 9, wherein the processing device determines a spatial arrangement pattern by mapping an object corresponding to the material identifying indicator. The material identification device of claim 10, wherein the processing device tracks the movement of the spatial arrangement pattern.

Images