JP7629342B2 - Road surface condition determination device, road surface condition determination system, vehicle, road surface condition determination method, and program - Google Patents

Description

This disclosure relates to a road surface condition determination device, a road surface condition determination system, a vehicle, a road surface condition determination method, and a program.

Patent Document 1 discloses a device that is mounted on a vehicle and determines road surface conditions. This device detects the pavement type of the road surface based on the vehicle position and map information, determines a first road surface condition that corresponds to the pavement type and the driving sound, determines a second road surface condition that corresponds to the pavement type and the road surface image, and outputs the first road surface condition or the second road surface condition to another device.

Patent Document 2 discloses a device that evaluates sounds emitted from a moving vehicle. This device acquires time waveform data based on the sound, and generates an evaluation image showing a time axis, a frequency axis, and signal strength based on a portion of the time waveform data. This device then refers to a reference database that stores the associations between past evaluation images and reference data linked to the past evaluation images, and derives an evaluation result for the generated evaluation image.

JP 2007-309832 A JP 2020-101447 A

Conventional devices are unable to accurately determine road surface conditions using sound.

The purpose of this disclosure is to improve the accuracy of determining road surface conditions using sound.

A road surface condition determination device according to one aspect of the present disclosure includes a control unit that analyzes time series data obtained by measuring sounds generated when a vehicle is traveling on a road using a sound sensor attached to the vehicle, generates analysis data indicating changes in signal strength over time for each frequency component, inputs the generated analysis data into an identification model for identifying the road surface condition of the road, obtains an identification result of the road surface condition from the identification model, and determines the road surface condition by referring to the obtained identification result.
According to this aspect, the accuracy of judging the road surface condition from sound is improved.

As one embodiment of the present disclosure, the control unit acquires a second identification result, which is an identification result of the road surface condition, based on image data obtained by capturing an image of the road by an image sensor attached to the vehicle, separately from a first identification result, which is an identification result obtained from the identification model, and determines the road surface condition by referring to the acquired first identification result and second identification result.
According to this embodiment, the accuracy of determining the road surface condition is further improved.

In one embodiment of the present disclosure, the image data includes road surface images from a first point to a second point on the road, and the control unit extracts data from the time series data from a first time point when the vehicle passes the first point to a second time point when the vehicle passes the second point, and analyzes the extracted data to generate the analysis data.
According to this embodiment, the analysis data can be synchronized with the image data.

In one embodiment of the present disclosure, the first identification result and the second identification result each include a probability that the road surface condition falls into each of a plurality of classes, and the control unit determines the class to which the road surface condition falls by comparing the probabilities included in the first identification result and the second identification result.
According to this embodiment, it is possible to adopt the result that is more likely to be reliable, between the result of classification based on sound and the result of classification based on image.

As one embodiment of the present disclosure, the control unit acquires meteorological data indicating the weather at the time the sound was measured, and weights the probability that the sound will be included in each of the first classification result and the second classification result for each class based on the acquired meteorological data.
According to this embodiment, the accuracy of determining the road surface condition is further improved.

As an embodiment of the present disclosure, the control unit weights each of the first identification result and the second identification result depending on a time period during which the sound was measured.
According to this embodiment, it is possible to adopt the result that is more suitable for the time period, between the result of classification by sound and the result of classification by image.

In one embodiment of the present disclosure, the control unit weights each of the first identification result and the second identification result depending on an outside air temperature at the time when the sound is measured.
According to this embodiment, it is possible to adopt the result that is more suitable for the outside air temperature, between the result of classification by sound and the result of classification by image.

As an embodiment of the present disclosure, the control unit weights the first classification result and the second classification result in accordance with information caused by an external environment.
According to this embodiment, the accuracy of determining the road surface condition is further improved.

As one embodiment of the present disclosure, the control unit extracts data of a certain range of frequencies from the time-series data, and analyzes the extracted data to generate the analysis data.
According to this embodiment, the accuracy of judging the road surface condition from sound is further improved.

As one embodiment of the present disclosure, the control unit extracts data on a certain range of signal strength from the time-series data, and analyzes the extracted data to generate the analysis data.
According to this embodiment, the accuracy of judging the road surface condition from sound is further improved.

In one embodiment of the present disclosure, the discriminative model includes a trained model that has been subjected to machine learning.
According to this embodiment, the accuracy of judging the road surface condition from sound can be improved by having the discrimination model learn a large amount of data.

In one embodiment of the present disclosure, the analysis data includes a spectrogram or a scalogram.
According to this embodiment, the efficiency of judging the road surface condition from sound is improved.

A road surface condition determination system according to one aspect of the present disclosure includes the road surface condition determination device and the sound sensor.
According to this aspect, the accuracy of judging the road surface condition from sound is improved.

A vehicle according to one aspect of the present disclosure includes the road surface condition determination system.
According to this aspect, road surface conditions can be determined from sounds in a vehicle.

A road surface condition determination method according to one aspect of the present disclosure includes measuring sounds generated when a vehicle is traveling on a road using a sound sensor attached to the vehicle, analyzing time-series data obtained as the measurement results using a computer to generate analysis data indicating changes in signal strength over time for each frequency component, inputting the analysis data into an identification model for identifying the road surface condition of the road by the computer, obtaining an identification result of the road surface condition from the identification model, and referring to the identification result by the computer to determine the road surface condition.
According to this aspect, the accuracy of judging the road surface condition from sound is improved.

A program as one aspect of the present disclosure causes a computer to perform the following processes: analyzing time series data obtained by measuring sounds generated when a vehicle is traveling on a road using a sound sensor attached to the vehicle, and generating analysis data indicating changes in signal strength over time for each frequency component; inputting the analysis data into an identification model for identifying the road surface condition of the road, thereby obtaining an identification result of the road surface condition from the identification model; and determining the road surface condition by referring to the identification result.
According to this aspect, the accuracy of judging the road surface condition from sound is improved.

This disclosure improves the accuracy of determining road surface conditions using sound.

1 is a diagram illustrating a configuration of a road surface condition determination system according to an embodiment of the present disclosure. FIG. 4 is a diagram showing a procedure for determining a road surface condition according to an embodiment of the present disclosure. FIG. 13 is a diagram illustrating an example of analysis data according to an embodiment of the present disclosure. 1 is a block diagram showing a configuration of a road surface condition determination device according to an embodiment of the present disclosure. 4 is a flowchart showing an operation of the road surface condition determination system according to the embodiment of the present disclosure. 4 is a flowchart showing an operation of the road surface condition determination device according to the embodiment of the present disclosure. 4 is a flowchart showing an operation of the road surface condition determination device according to the embodiment of the present disclosure.

One embodiment of the present disclosure will be described below with reference to the drawings.

In each figure, the same or corresponding parts are given the same reference numerals. In the description of this embodiment, the description of the same or corresponding parts will be omitted or simplified as appropriate.

The configuration of the road surface condition determination system 10 according to this embodiment will be described with reference to FIG. 1.

The road surface condition determination system 10 includes a road surface condition determination device 20, an image sensor 30, and a sound sensor 40.

The road surface condition determination device 20 is a computer. In this embodiment, the road surface condition determination device 20 is a PC, but it may also be a mobile device such as a smartphone or tablet, a server device belonging to a cloud computing system or other computing system, or a dedicated device. "PC" is an abbreviation for personal computer.

The image sensor 30 is attached to the vehicle 50 traveling on the road 11. Specifically, the image sensor 30 is attached on the dashboard or near the rearview mirror facing the front of the vehicle 50 so as to be able to capture an image of the area in front of the vehicle 50. The image sensor 30 is, for example, a CMOS sensor, a CCD sensor, or a camera incorporating either of these. "CMOS" is an abbreviation for complementary metal oxide semiconductor. "CCD" is an abbreviation for charge coupled device. The image sensor 30 may be able to capture an image of the rear.

The sound sensor 40 is also attached to the vehicle 50. Specifically, the sound sensor 40 is attached near the tires of the vehicle 50 so that the sound generated from the tires can be measured. In this embodiment, the sound sensor 40 is attached near the front wheels, but it may also be attached near the rear wheels. The sound sensor 40 is, for example, a microphone.

Vehicle 50 is any type of automobile, such as a gasoline vehicle, diesel vehicle, HV, PHV, EV, or FCV. "HV" is an abbreviation for hybrid vehicle. "PHV" is an abbreviation for plug-in hybrid vehicle. "EV" is an abbreviation for electric vehicle. "FCV" is an abbreviation for fuel cell vehicle.

An overview of this embodiment will be explained with reference to Figures 1, 2, and 3.

The sound sensor 40 measures the sound generated when the vehicle 50 is traveling on the road 11. The road surface condition determination device 20 analyzes the time series data 61 obtained as the measurement result to generate analysis data 63 as shown in FIG. 3. The analysis data 63 is data showing the change over time in the signal strength for each frequency component. The road surface condition determination device 20 obtains a first identification result, which is an identification result of the road surface condition of the road 11, based on the generated analysis data 63. Specifically, the road surface condition determination device 20 inputs the analysis data 63 to the first identification model 71 to obtain the first identification result from the first identification model 71. The road surface condition determination device 20 determines the road surface condition by referring to the obtained first identification result. Therefore, according to this embodiment, the accuracy of road surface condition determination by sound is improved. Since the analysis data 63 in FIG. 3 changes depending on the traveling speed even on the same road surface, it is preferable to unify the traveling speed within a certain range or to perform correction according to the traveling speed when obtaining the identification result.

The first identification model 71 may be any identification model for identifying road surface conditions, but in this embodiment, it includes a trained model that has undergone machine learning. For example, time-series data previously obtained by the sound sensor 40 is analyzed to generate past analysis data, and the actual road surface conditions are linked to this analysis data as labels, thereby creating training data for machine learning. Then, using this training data, machine learning can be performed using a known machine learning algorithm, such as a neural network or deep learning, to generate a trained model as the first identification model 71. According to this embodiment, the accuracy of road surface condition determination from sound can be improved by having the first identification model 71 learn a large amount of data.

In this embodiment, the first identification model 71 outputs a probability for each label as the first identification result. The probability for each label is the probability that the road surface condition corresponds to the class corresponding to each label. In other words, in this embodiment, the first identification result includes the probability that the road surface condition corresponds to each of multiple classes.

In this embodiment, seven classes are defined as multiple classes: "dry", "semi-humid", "moist and wet", "sherbet", "accumulated snow", "packed snow", and "frozen". However, instead of these seven classes, any two or more classes may be defined, such as the five classes of "dry", "semi-humid", "moist and wet", "sherbet", and "snow".

In this embodiment, the analysis data 63 includes a spectrogram, but may include an image generated using another frequency analysis method, such as a scalogram, or data in a format other than an image generated using any frequency analysis method. FIG. 3 shows a first spectrogram 64, a second spectrogram 65, and a third spectrogram 66 as examples of the analysis data 63. In each spectrogram, the horizontal axis represents the time axis, the vertical axis represents the frequency axis, and the color shade represents the signal intensity. The first spectrogram 64 is an example of a spectrogram of a sound generated when the vehicle 50 is traveling on a semi-humid road surface. The second spectrogram 65 is an example of a spectrogram of a sound generated when the vehicle 50 is traveling on a wet road surface. The third spectrogram 66 is an example of a spectrogram of a sound generated when the vehicle 50 is traveling on a slush road surface.

The image sensor 30 captures an image of the road 11. The road surface condition determination device 20 acquires a second identification result, which is an identification result of the road surface condition of the road 11, based on image data 62 obtained as a captured image, separately from the first identification result. Specifically, the road surface condition determination device 20 inputs the image data 62 to the second identification model 72, thereby acquiring the second identification result from the second identification model 72. The road surface condition determination device 20 judges the road surface condition by referring to not only the first identification result but also the acquired second identification result. Therefore, according to this embodiment, the accuracy of road surface condition determination is further improved.

The second identification model 72 may be any identification model for identifying road surface conditions, but in this embodiment, it includes a trained model that has undergone machine learning. For example, training data for machine learning can be created by linking actual road surface conditions as labels to image data previously obtained by the image sensor 30. Then, this training data can be used to perform machine learning using a known machine learning algorithm, such as a neural network or deep learning, to generate a trained model as the second identification model 72. According to this embodiment, the accuracy of road surface condition determination from images can be improved by having the second identification model 72 train a large amount of data.

In this embodiment, the second classification model 72 outputs a probability for each label as the second classification result. That is, in this embodiment, the second classification result includes the probability that the road surface condition falls into each of the multiple classes, just like the first classification result.

The configuration of the road surface condition determination device 20 according to this embodiment will be described with reference to FIG.

The road surface condition determination device 20 includes a control unit 21, a memory unit 22, a communication unit 23, an input unit 24, and an output unit 25.

The control unit 21 includes at least one processor, at least one programmable circuit, at least one dedicated circuit, or any combination of these. The processor is a general-purpose processor such as a CPU or GPU, or a dedicated processor specialized for specific processing. "CPU" is an abbreviation for central processing unit. "GPU" is an abbreviation for graphics processing unit. The programmable circuit is, for example, an FPGA. "FPGA" is an abbreviation for field-programmable gate array. The dedicated circuit is, for example, an ASIC. "ASIC" is an abbreviation for application specific integrated circuit. The control unit 21 executes processing related to the operation of the road surface condition determination device 20 while controlling each part of the road surface condition determination device 20.

The storage unit 22 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or any combination thereof. The semiconductor memory is, for example, a RAM or a ROM. "RAM" is an abbreviation for random access memory. "ROM" is an abbreviation for read only memory. The RAM is, for example, an SRAM or a DRAM. "SRAM" is an abbreviation for static random access memory. "DRAM" is an abbreviation for dynamic random access memory. The ROM is, for example, an EEPROM. "EEPROM" is an abbreviation for electrically erasable programmable read only memory. The storage unit 22 functions, for example, as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 22 stores data used in the operation of the road surface condition determination device 20 and data obtained by the operation of the road surface condition determination device 20.

The communication unit 23 includes at least one communication interface. The communication interface is, for example, a LAN interface, an interface compatible with a mobile communication standard such as LTE, the 4G standard, or the 5G standard, or an interface compatible with a short-range wireless communication standard such as Bluetooth (registered trademark). "LAN" is an abbreviation for local area network. "LTE" is an abbreviation for Long Term Evolution. "4G" is an abbreviation for 4th generation. "5G" is an abbreviation for 5th generation. The communication unit 23 receives data used in the operation of the road surface condition determination device 20, and transmits data obtained by the operation of the road surface condition determination device 20.

The input unit 24 includes at least one input interface. The input interface is, for example, a physical key, a capacitive key, a pointing device, a touch screen integral with a display, a camera, or a microphone. The input unit 24 accepts an operation to input data used in the operation of the road surface condition determination device 20. The input unit 24 may be connected to the road surface condition determination device 20 as an external input device instead of being provided in the road surface condition determination device 20. As the connection interface, for example, an interface compatible with a standard such as USB, HDMI (registered trademark), or Bluetooth (registered trademark) can be used. "USB" is an abbreviation for Universal Serial Bus. "HDMI (registered trademark)" is an abbreviation for High-Definition Multimedia Interface.

The output unit 25 includes at least one output interface. The output interface is, for example, a display or a speaker. The display is, for example, an LCD or an organic EL display. "LCD" is an abbreviation for liquid crystal display. "EL" is an abbreviation for electro luminescence. The output unit 25 outputs data obtained by the operation of the road surface condition determination device 20. The output unit 25 may be connected to the road surface condition determination device 20 as an external output device instead of being provided in the road surface condition determination device 20. As the connection interface, for example, an interface compatible with standards such as USB, HDMI (registered trademark), or Bluetooth (registered trademark) can be used.

The functions of the road surface condition determination device 20 are realized by executing a program according to this embodiment in a processor serving as the control unit 21. That is, the functions of the road surface condition determination device 20 are realized by software. The program causes a computer to execute the operations of the road surface condition determination device 20, thereby causing the computer to function as the road surface condition determination device 20. That is, the computer functions as the road surface condition determination device 20 by executing the operations of the road surface condition determination device 20 in accordance with the program.

The program may be stored in a non-transitory computer-readable medium. Examples of the non-transitory computer-readable medium include flash memory, a magnetic recording device, an optical disk, a magneto-optical recording medium, or a ROM. The program may be distributed, for example, by selling, transferring, or lending portable media such as SD cards, DVDs, or CD-ROMs on which the program is stored. "SD" is an abbreviation for Secure Digital. "DVD" is an abbreviation for digital versatile disc. "CD-ROM" is an abbreviation for compact disc read only memory. The program may be distributed by storing the program in the storage of a server and transferring the program from the server to another computer. The program may be provided as a program product.

For example, a computer temporarily stores a program stored in a portable medium or a program transferred from a server in a main storage device. The computer then reads the program stored in the main storage device with a processor and executes processing according to the read program with the processor. The computer may read the program directly from the portable medium and execute processing according to the program. The computer may execute processing according to the received program each time a program is transferred from the server to the computer. Processing may be executed by a so-called ASP-type service that does not transfer a program from the server to the computer and achieves functions only by issuing execution instructions and obtaining results. "ASP" is an abbreviation for application service provider. A program is information used for processing by a computer and includes those equivalent to a program. For example, data that is not a direct command to a computer but has properties that define computer processing falls under "those equivalent to a program."

Some or all of the functions of the road surface condition determination device 20 may be realized by a programmable circuit or a dedicated circuit as the control unit 21. In other words, some or all of the functions of the road surface condition determination device 20 may be realized by hardware.

The operation of the road surface condition determination system 10 according to this embodiment will be described with reference to FIG. 5. This operation corresponds to the road surface condition determination method according to this embodiment.

In step S1, the image sensor 30 captures an image of the road 11. Image data 62 obtained as the captured image is stored in a medium such as a flash memory or a magnetic recording device mounted on the vehicle 50, or is transmitted to the outside via a network such as the Internet by a communication device mounted on the vehicle 50. At approximately the same time that the image of the road 11 is captured, the sound sensor 40 measures the sound generated when the vehicle 50 is traveling on the road 11. Time series data 61 obtained as a result of the measurement is stored in a medium mounted on the vehicle 50, like the image data 62, or is transmitted to the outside via a network by a communication device mounted on the vehicle 50.

In step S2, the control unit 21 of the road surface condition determination device 20 analyzes the time series data 61 obtained in step S1 to generate analysis data 63 indicating the change over time in the signal strength for each frequency component. This process may be performed in any order, but in this embodiment, it is performed in the order shown in FIG. 6.

In step S201 of FIG. 6, the control unit 21 acquires the time series data 61 stored in a medium mounted on the vehicle 50 in step S1 via the communication unit 23 or another interface compatible with a standard such as USB. Alternatively, the control unit 21 acquires the time series data 61 by receiving the time series data 61 transmitted by a communication device mounted on the vehicle 50 in step S1 via the communication unit 23.

In step S202 of FIG. 6, the control unit 21 extracts data of a certain range of frequencies from the time series data 61 acquired in step S201, and also extracts data of a certain range of signal strength from the time series data 61. That is, the control unit 21 extracts data of a certain range of frequencies and a certain range of signal strength from the time series data 61. The frequency range is preferably 1 kHz to 20 kHz. The signal strength range is preferably 40 dB to 60 dB. That is, the control unit 21 preferably extracts data of frequencies from 1 kHz to 20 kHz and signal strengths from 40 dB to 60 dB from the time series data 61.

In step S203 of FIG. 6, the control unit 21 analyzes the data extracted in step S202 to generate analysis data 63. Specifically, the control unit 21 performs frequency analysis of the data extracted in step S202 to generate a spectrogram.

In step S3, the control unit 21 of the road surface condition determination device 20 obtains a first identification result based on the analysis data 63 generated in step S2. Specifically, the control unit 21 inputs the analysis data 63 to a first identification model 71 for identifying the road surface condition of the road 11, and obtains the road surface condition identification result as the first identification result from the first identification model 71. In this embodiment, the first identification result includes the probability that the road surface condition falls into each of the seven classes of "dry", "semi-humid", "wet and moist", "sherbet", "snow-covered", "packed snow", and "frozen".

In step S4, the control unit 21 of the road surface condition determination device 20 acquires a second identification result based on the image data 62 acquired in step S1. Specifically, the control unit 21 acquires the image data 62 stored in the medium mounted on the vehicle 50 in step S1 via the communication unit 23 or another interface corresponding to a standard such as USB. Alternatively, the control unit 21 acquires the image data 62 by receiving the image data 62 transmitted by the communication device mounted on the vehicle 50 in step S1 via the communication unit 23. The control unit 21 extracts a road surface image from the acquired image data 62. As a method for extracting the road surface image, any method may be used, such as a method of cutting out an area of a fixed number of pixels as a road surface image, a method of cutting out a road surface area recognized by performing image analysis as a road surface image, or a method of cutting out a road surface area identified using machine learning as a road surface image. The road surface image may be cut out for each lane. The control unit 21 inputs the extracted road surface image into a second classification model 72 for classifying the road surface condition of the road 11, and obtains the classification result of the road surface condition as the second classification result from the second classification model 72. In this embodiment, the second classification result, like the first classification result, includes the probability that the road surface condition falls into each of the seven classes of "dry," "semi-humid," "wet and moist," "sherbet," "snow-covered," "packed snow," and "frozen."

In step S5, the control unit 21 of the road surface condition determination device 20 determines the road surface condition by referring to the first identification result and the second identification result acquired in step S3 and step S4, respectively. This process may be performed in any order, but in this embodiment, it is performed in the order shown in FIG. 7.

In step S501 of FIG. 7, the control unit 21 weights each of the first and second identification results according to the time period in which the sound was measured in step S1. For example, if the time period is at night, when the accuracy of road surface condition determination using images is likely to decrease, the control unit 21 weights the probability included in the first identification result more heavily than the probability included in the second identification result. If the time period is during the day, when the accuracy of road surface condition determination using images is likely to increase, the control unit 21 weights the probability included in the second identification result the same as the weighting of the probability included in the first identification result or more heavily than the weighting of the probability included in the first identification result.

In step S502 of FIG. 7, the control unit 21 weights each of the first and second identification results according to the outside air temperature when the sound was measured in step S1. For example, if the outside air temperature is low enough to cause black ice, the control unit 21 weights the probability included in the first identification result more than the probability included in the second identification result. If the outside air temperature is high enough not to cause black ice, the control unit 21 weights the probability included in the second identification result the same as the weighting of the probability included in the first identification result or more than the weighting of the probability included in the first identification result.

In step S503 of FIG. 7, the control unit 21 determines the class to which the road surface condition corresponds by comparing the probabilities included in the first and second identification results weighted in steps S501 and S502, respectively. For example, the control unit 21 determines the class with the highest overall probability in the first and second identification results as the class to which the road surface condition corresponds. Suppose that the probabilities of "dry", "semi-humid", and "moist and moist" included in the first identification result are 10%, 80%, and 10%, respectively, and the probabilities of "dry", "semi-humid", and "moist and moist" included in the second identification result are 50%, 30%, and 20%, respectively. In that case, since the probability of "semi-humid" included in the first identification result is the highest overall, the control unit 21 determines that the road surface condition is a semi-humid state. Alternatively, the control unit 21 may determine that the class with the highest average probability in the first and second identification results is the class to which the road surface condition corresponds. Let us assume that the probabilities of "dry", "semi-humid", and "moist" included in the first identification result are 30%, 40%, and 30%, respectively, and that the probabilities of "dry", "semi-humid", and "moist" included in the second identification result are 50%, 30%, and 20%, respectively. In this case, since the average probability of "dry" is the highest, the control unit 21 determines that the road surface condition is dry.

As described above, in this embodiment, the control unit 21 of the road surface condition determination device 20 analyzes the time series data 61 obtained by measuring the sound generated when the vehicle 50 is traveling on the road 11 using the sound sensor 40 attached to the vehicle 50, and generates analysis data 63 indicating the change over time in the signal strength for each frequency component. The control unit 21 inputs the generated analysis data 63 to a first identification model 71 for identifying the road surface condition of the road 11, thereby acquiring an identification result of the road surface condition from the first identification model 71. The control unit 21 judges the road surface condition by referring to the acquired identification result. Therefore, according to this embodiment, the accuracy of road surface condition judgment from sound is improved.

In this embodiment, the control unit 21 of the road surface condition determination device 20 obtains a second identification result, which is an identification result of the road surface condition, based on image data 62 obtained by capturing an image of the road 11 by the image sensor 30 attached to the vehicle 50, in addition to the first identification result, which is an identification result obtained from the first identification model 71. The control unit 21 determines the road surface condition by referring to the obtained first identification result and second identification result. Therefore, according to this embodiment, the accuracy of the road surface condition determination is further improved.

As a modified example of this embodiment, the analysis data 63 may be synchronized with the image data 62. In such a modified example, the image data 62 includes a road surface image from a first point to a second point on the road 11. In step S202 of FIG. 6, the control unit 21 of the road surface condition determination device 20 extracts data from the time series data 61, the data being from a first time point when the vehicle 50 passes the first point to a second time point when the vehicle 50 passes the second point. According to this modified example, the width of the analysis data 63, which is data having a width before and after a certain point in the time direction, can be matched to the length of the section of the road 11 included in the road surface image. In other words, synchronization on the time axis can be achieved.

As a modification of this embodiment, meteorological information such as a weather forecast may be reflected. In such a modification, the control unit 21 of the road surface condition determination device 20 acquires meteorological data indicating the weather at the time the sound was measured. Based on the acquired meteorological data, the control unit 21 weights the probability of being included in each of the first and second identification results for each class. According to this modification, the accuracy of road surface condition determination is further improved.

Weather information is an example of information caused by the external environment. The time zone information referenced in step S501 of FIG. 7 and the outside temperature information referenced in step S502 of FIG. 7 are also examples of information caused by the external environment. Information other than the weather information, time zone information, and outside temperature information may be reflected as information caused by the external environment. In other words, the control unit 21 of the road surface condition determination device 20 may perform weighting according to information caused by the external environment other than the weather information, time zone information, and outside temperature information.

The present disclosure is not limited to the above-described embodiments. For example, two or more blocks shown in the block diagram may be integrated, or one block may be divided. Two or more steps shown in the flowchart may be executed in parallel or in a different order, instead of being executed in chronological order as described, depending on the processing capabilities of the device executing each step, or as needed. Other modifications are possible without departing from the spirit of the present disclosure.

For example, the road surface condition determination device 20 may be an in-vehicle device. That is, the vehicle 50 may be equipped with the road surface condition determination system 10. According to this example, the road surface condition determination based on sound can be performed by the vehicle 50.

For example, the image sensor 30 may be omitted. In such an example, the control unit 21 of the road surface condition determination device 20 determines the road surface condition without referring to the second identification result. Alternatively, instead of the image sensor 30, an acceleration sensor or an infrared sensor may be combined with the sound sensor 40 to determine the road surface condition.

For example, the series of processes from step S1 to step S5 may be executed for each section of a fixed distance, such as 100 meters, that the vehicle 50 travels. Alternatively, the process of step S1 may be executed for each section, and then the series of processes from step S2 to step S5 may be executed for the data obtained in each section. Alternatively, the series of processes from step S1 to step S5 may be executed at regular intervals.

For example, step S501, step S502, or both of these processes in FIG. 7 may be omitted.

This disclosure relates to a road surface condition determination device, a road surface condition determination system, a vehicle, a road surface condition determination method, and a program.

10: Road surface condition determination system, 11: Road, 20: Road surface condition determination device, 21: Control unit, 22: Memory unit, 23: Communication unit, 24: Input unit, 25: Output unit, 30: Image sensor, 40: Sound sensor, 50: Vehicle, 61: Time series data, 62: Image data, 63: Analysis data, 64: First spectrogram, 65: Second spectrogram, 66: Third spectrogram, 71: First identification model, 72: Second identification model

Claims (16)

a control unit that analyzes time-series data obtained by measuring sounds generated when a vehicle is traveling on a road with a sound sensor attached to the vehicle, generates analysis data indicating changes in signal strength for each frequency component over time, and inputs the generated analysis data into an identification model for identifying a road surface condition of the road, thereby obtaining an identification result of the road surface condition from the identification model, and obtains, separately from a first identification result which is the identification result obtained from the identification model, a second identification result which is the identification result of the road surface condition based on image data obtained by capturing an image of the road with an image sensor attached to the vehicle , and judges the road surface condition by referring to the obtained first identification result and second identification result ,
the first classification result and the second classification result each include a probability that the road surface condition corresponds to each of a plurality of classes,
the control unit determines a class to which the road surface condition corresponds by comparing a probability included in the first classification result and a probability included in the second classification result,
The control unit acquires meteorological data indicating the weather at the time the sound was measured, and weights the probability that the first classification result and the second classification result are included for each class based on the acquired meteorological data . The image data includes a road surface image from a first point to a second point of the road,
2. The road surface condition determination device according to claim 1, wherein the control unit extracts data from the time series data from a first time point when the vehicle passes the first point to a second time point when the vehicle passes the second point, and analyzes the extracted data to generate the analysis data. a control unit that analyzes time-series data obtained by measuring sounds generated when a vehicle is traveling on a road with a sound sensor attached to the vehicle, generates analysis data indicating changes in signal strength for each frequency component over time, and inputs the generated analysis data into an identification model for identifying a road surface condition of the road, thereby obtaining an identification result of the road surface condition from the identification model, and obtains, separately from a first identification result which is the identification result obtained from the identification model, a second identification result which is the identification result of the road surface condition based on image data obtained by capturing an image of the road with an image sensor attached to the vehicle , and judges the road surface condition by referring to the obtained first identification result and second identification result ,
The control unit weights each of the first identification result and the second identification result depending on the time period during which the sound was measured . a control unit that analyzes time-series data obtained by measuring sounds generated when a vehicle is traveling on a road with a sound sensor attached to the vehicle, generates analysis data indicating changes in signal strength for each frequency component over time, and inputs the generated analysis data into an identification model for identifying a road surface condition of the road, thereby obtaining an identification result of the road surface condition from the identification model, and obtains, separately from a first identification result which is the identification result obtained from the identification model, a second identification result which is the identification result of the road surface condition based on image data obtained by capturing an image of the road with an image sensor attached to the vehicle , and judges the road surface condition by referring to the obtained first identification result and second identification result ,
The control unit weights each of the first identification result and the second identification result in accordance with an outside air temperature when the sound is measured . a control unit that analyzes time-series data obtained by measuring sounds generated when a vehicle is traveling on a road with a sound sensor attached to the vehicle, generates analysis data indicating changes in signal strength for each frequency component over time, and inputs the generated analysis data into an identification model for identifying a road surface condition of the road, thereby obtaining an identification result of the road surface condition from the identification model, and obtains, separately from a first identification result which is the identification result obtained from the identification model, a second identification result which is the identification result of the road surface condition based on image data obtained by capturing an image of the road with an image sensor attached to the vehicle , and judges the road surface condition by referring to the obtained first identification result and second identification result ,
The control unit weights the first identification result and the second identification result in accordance with information caused by an external environment . The road surface condition determination device according to claim 1 , wherein the control unit extracts data of a certain range of frequencies from the time series data, and analyzes the extracted data to generate the analysis data. 7. The road surface condition determination device according to claim 1, wherein the control unit extracts data on a certain range of signal strength from the time series data, and analyzes the extracted data to generate the analysis data. The road surface condition determination device according to claim 1 , wherein the identification model includes a trained model that has been subjected to machine learning. The road surface condition determination device according to claim 1 , wherein the analysis data includes a spectrogram or a scalogram. The road surface condition determination device according to any one of claims 1 to 9 ,
A road surface condition determination system comprising the sound sensor. A vehicle equipped with the road surface condition determination system according to claim 10 . A sound generated when a vehicle is traveling on a road is measured by a sound sensor attached to the vehicle;
The time series data obtained as the measurement result is analyzed by a computer to generate analysis data showing the change over time in the signal strength for each frequency component.
inputting the analysis data into an identification model for identifying a road surface condition of the road by the computer, thereby obtaining an identification result of the road surface condition from the identification model;
acquiring, by the computer, a second classification result which is a classification result of the road surface condition, based on image data obtained by capturing an image of the road by an image sensor attached to the vehicle, separately from a first classification result which is a classification result obtained from the classification model;
A road surface condition determination method for determining the road surface condition by referring to the first identification result and the second identification result by the computer,
the first classification result and the second classification result each include a probability that the road surface condition corresponds to each of a plurality of classes,
the computer determines a class to which the road surface condition belongs by comparing a probability included in the first classification result and a probability included in the second classification result;
The computer acquires meteorological data indicating the weather at the time the sound was measured, and weights the probability that the first classification result and the second classification result are included for each class based on the acquired meteorological data . A sound generated when a vehicle is traveling on a road is measured by a sound sensor attached to the vehicle;
The time series data obtained as the measurement result is analyzed by a computer to generate analysis data showing the change over time in the signal strength for each frequency component.
inputting the analysis data into an identification model for identifying a road surface condition of the road by the computer, thereby obtaining an identification result of the road surface condition from the identification model;
a second classification result, which is a classification result of the road surface condition, is obtained by the computer based on image data obtained by capturing an image of the road by an image sensor attached to the vehicle, separately from a first classification result, which is a classification result obtained from the classification model;
A road surface condition determination method for determining the road surface condition by referring to the first identification result and the second identification result by the computer,
The method of determining road surface conditions , wherein the computer weights each of the first identification result and the second identification result depending on the time period during which the sound was measured . A sound generated when a vehicle is traveling on a road is measured by a sound sensor attached to the vehicle;
The time series data obtained as the measurement result is analyzed by a computer to generate analysis data showing the change over time in the signal strength for each frequency component.
inputting the analysis data into an identification model for identifying a road surface condition of the road by the computer, thereby obtaining an identification result of the road surface condition from the identification model;
acquiring, by the computer, a second classification result which is a classification result of the road surface condition, based on image data obtained by capturing an image of the road by an image sensor attached to the vehicle, separately from a first classification result which is a classification result obtained from the classification model;
A road surface condition determination method for determining the road surface condition by referring to the first identification result and the second identification result by the computer,
The computer weights each of the first identification result and the second identification result in accordance with an outside air temperature when the sound is measured . A sound generated when a vehicle is traveling on a road is measured by a sound sensor attached to the vehicle;
The time series data obtained as the measurement result is analyzed by a computer to generate analysis data showing the change over time in the signal strength for each frequency component.
inputting the analysis data into an identification model for identifying a road surface condition of the road by the computer, thereby obtaining an identification result of the road surface condition from the identification model;
acquiring, by the computer, a second classification result which is a classification result of the road surface condition, based on image data obtained by capturing an image of the road by an image sensor attached to the vehicle, separately from a first classification result which is a classification result obtained from the classification model;
A road surface condition determination method for determining the road surface condition by referring to the first identification result and the second identification result by the computer,
The method of determining road surface conditions , wherein the computer weights the first identification result and the second identification result in accordance with information caused by an external environment . A program that causes a computer to execute the operation of the road surface condition determination device according to any one of claims 1 to 9 .

Images