JP7629343B2 - Road surface condition determination device, road surface condition determination system, vehicle, road surface condition determination method, and program - Google Patents

Description

This disclosure relates to a road surface condition determination device, a road surface condition determination system, a vehicle, a road surface condition determination method, and a program.

Patent Document 1 discloses a device that is mounted on a vehicle and determines road surface conditions. This device detects the pavement type of the road surface based on the vehicle position and map information, determines a first road surface condition that corresponds to the pavement type and the driving sound, determines a second road surface condition that corresponds to the pavement type and the road surface image, and outputs the first road surface condition or the second road surface condition to another device.

JP 2007-309832 A

When judging road surface conditions from images, depending on the image, factors such as the amount of light or objects on the road being reflected in the image may make it difficult to make an accurate judgment.

The purpose of this disclosure is to improve the accuracy of determining road surface conditions from images.

A road surface condition determination device as one aspect of the present disclosure is provided with a control unit that acquires identification results for the road surface condition for each section based on image data for each section obtained by capturing images of multiple sections of a road using an image sensor attached to a vehicle traveling on the road, and when each of the multiple sections is defined as a first section, and at least one section before, after, or before and after the first section is defined as a second section, determines the road surface condition of the first section by referring to a first identification result that is the identification result of the road surface condition of the first section and a second identification result that is the identification result of the road surface condition of the second section.
According to this aspect, the accuracy of determining the road surface condition from an image is improved.

In one embodiment of the present disclosure, the control unit inputs image data for each section into an identification model for identifying road surface conditions, and obtains an identification result of the road surface condition for each section from the identification model.
According to this embodiment, the efficiency of judging the road surface condition from an image is improved.

In one embodiment of the present disclosure, the identification result of the road surface condition for each section includes a probability that the road surface condition for each section falls into each of a plurality of classes, and the control unit determines the class to which the road surface condition of the first section falls by comparing the probabilities contained in the first identification result and the second identification result.
According to this embodiment, it becomes easier to prevent erroneous determination of the road surface condition of the first section.

In one embodiment of the present disclosure, the control unit determines a class having the highest probability in the greatest number of identification results, out of the first identification result and the second identification result, as a class to which the road surface condition of the first section corresponds.
According to this embodiment, the identification result of the road surface condition of the first section can be corrected.

In one embodiment of the present disclosure, the control unit determines a class having the highest average probability in the first classification result and the second classification result as a class to which the road surface condition of the first section corresponds.
According to this embodiment, the identification result of the road surface condition of the first section can be corrected.

As one embodiment of the present disclosure, the control unit acquires weather data indicating the weather at the time the road was imaged, and weights the probability that the first classification result and the second classification result will be included for each class based on the acquired weather data.
According to this embodiment, the accuracy of determining the road surface condition from an image is further improved.

In one embodiment of the present disclosure, the discriminative model includes a trained model that has been subjected to machine learning.
According to this embodiment, the accuracy of determining road surface conditions from images can be improved by training a discrimination model using a large amount of data.

A road surface condition determination system according to one aspect of the present disclosure includes the road surface condition determination device and the image sensor.
According to this aspect, the accuracy of determining the road surface condition from an image is improved.

A vehicle according to one aspect of the present disclosure includes the road surface condition determination system.
According to this aspect, road surface conditions can be determined in a vehicle using images.

A road surface condition determination method as one aspect of the present disclosure involves capturing images of multiple sections of a road using an image sensor attached to a vehicle traveling on the road, obtaining an identification result of the road surface condition for each section by a computer based on image data for each section obtained as the captured image, and when each of the multiple sections is designated a first section and at least one section before, after, or before and after the first section is designated a second section, the computer refers to a first identification result which is the identification result of the road surface condition of the first section and a second identification result which is the identification result of the road surface condition of the second section to determine the road surface condition of the first section.
According to this aspect, the accuracy of determining the road surface condition from an image is improved.

A program as one aspect of the present disclosure causes a computer to perform a process of obtaining an identification result of the road surface condition for each section based on image data for each section obtained by capturing images of the multiple sections of a road using an image sensor attached to a vehicle traveling on the road, and a process of determining the road surface condition of the first section by referring to a first identification result that is the identification result of the road surface condition of the first section and a second identification result that is the identification result of the road surface condition of the second section, when each of the multiple sections is defined as a first section and at least one section before, after, or before and after the first section is defined as a second section.
According to this aspect, the accuracy of determining the road surface condition from an image is improved.

This disclosure improves the accuracy of determining road surface conditions from images.

1 is a diagram illustrating a configuration of a road surface condition determination system according to an embodiment of the present disclosure. FIG. 4 is a diagram showing a procedure for determining a road surface condition according to an embodiment of the present disclosure. 11A to 11C are diagrams illustrating an example of image data for each section and an example of a result of identifying a road surface condition for each section according to an embodiment of the present disclosure. 1 is a block diagram showing a configuration of a road surface condition determination device according to an embodiment of the present disclosure. 4 is a flowchart showing an operation of the road surface condition determination system according to the embodiment of the present disclosure. 4 is a flowchart showing an example of an operation of the road surface condition determination device according to the embodiment of the present disclosure. 10 is a flowchart showing another example of the operation of the road surface condition determination device according to the embodiment of the present disclosure.

One embodiment of the present disclosure will be described below with reference to the drawings.

In each figure, the same or corresponding parts are given the same reference numerals. In the description of this embodiment, the description of the same or corresponding parts will be omitted or simplified as appropriate.

The configuration of the road surface condition determination system 10 according to this embodiment will be described with reference to FIG. 1.

The road surface condition determination system 10 includes a road surface condition determination device 20 and an image sensor 30.

The road surface condition determination device 20 is a computer. In this embodiment, the road surface condition determination device 20 is a PC, but it may also be a mobile device such as a smartphone or tablet, a server device belonging to a cloud computing system or other computing system, or a dedicated device. "PC" is an abbreviation for personal computer.

The image sensor 30 is attached to the vehicle 50 traveling on the road 11. Specifically, the image sensor 30 is attached on the dashboard or near the rearview mirror facing the front of the vehicle 50 so as to be able to capture an image of the area in front of the vehicle 50. The image sensor 30 is, for example, a CMOS sensor, a CCD sensor, or a camera incorporating either of these. "CMOS" is an abbreviation for complementary metal oxide semiconductor. "CCD" is an abbreviation for charge coupled device. The image sensor 30 may be able to capture an image of the rear.

Vehicle 50 is any type of automobile, such as a gasoline vehicle, diesel vehicle, HV, PHV, EV, or FCV. "HV" is an abbreviation for hybrid vehicle. "PHV" is an abbreviation for plug-in hybrid vehicle. "EV" is an abbreviation for electric vehicle. "FCV" is an abbreviation for fuel cell vehicle.

An overview of this embodiment will be explained with reference to Figures 1 and 2.

The image sensor 30 captures images of multiple sections of the road 11. The road surface condition determination device 20 obtains an identification result of the road surface condition for each section based on image data 60 for each section obtained as a captured image. Specifically, the road surface condition determination device 20 inputs the image data 60 for each section to the identification model 70, thereby obtaining an identification result of the road surface condition for each section from the identification model 70. When each of the multiple sections is a first section, and at least one section before, after, or before and after the first section is a second section, the road surface condition determination device 20 determines the road surface condition of the first section by referring to a first identification result that is an identification result of the road surface condition of the first section and a second identification result that is an identification result of the road surface condition of the second section. Therefore, according to this embodiment, the accuracy of road surface condition determination using images is improved.

The identification model 70 may be any identification model for identifying road surface conditions, but in this embodiment, it includes a trained model that has undergone machine learning. For example, training data for machine learning can be created by linking the actual road surface conditions as labels to image data previously obtained by the image sensor 30. Then, this training data can be used to perform machine learning using a known machine learning algorithm, such as a neural network or deep learning, to generate a trained model as the identification model 70. According to this embodiment, the accuracy of road surface condition determination from images can be improved by having the identification model 70 learn a large amount of data.

In this embodiment, the identification model 70 outputs a probability for each label as the identification result of the road surface condition for each section. The probability for each label is the probability that the road surface condition corresponds to the class corresponding to each label. In other words, in this embodiment, the identification result of the road surface condition for each section includes the probability that the road surface condition for each section corresponds to each of multiple classes.

In this embodiment, seven classes are defined as multiple classes: "dry", "semi-humid", "moist and wet", "sherbet", "accumulated snow", "packed snow", and "frozen". However, instead of these seven classes, any two or more classes may be defined, such as the five classes of "dry", "semi-humid", "moist and wet", "sherbet", and "snow".

Figure 3 shows a first section image 61, a previous section image 62, and a subsequent section image 63 as examples of image data 60 for each section. The length of each section is set to a fixed distance, such as 10 meters. When a certain section is the first section and the sections before and after the first section are the second section, the first section image 61 is an example of an image obtained by imaging the first section. The previous section image 62 is an example of an image obtained by imaging the section immediately before the first section. The subsequent section image 63 is an example of an image obtained by imaging the section immediately after the first section.

Figure 3 further shows, as an example of the classification result of the road surface condition for each section, the classification results output from the classification model 70 when the first section image 61, the previous section image 62, and the next section image 63 are input to the classification model 70. The probabilities of "moist" and "dry" included in the classification result corresponding to the first section image 61 are 30% and 50%, respectively. If the probabilities of other labels included in the classification result corresponding to the first section image 61 are all lower than 30%, it is considered that the road surface condition of the first section is likely to be dry. However, since the probability of "moist" included in the classification result corresponding to the previous section image 62 is 90%, it is very likely that the road surface condition of the section immediately preceding the first section is in a moist state. Since the probability of "moist" included in the classification result corresponding to the next section image 63 is also 90%, it is very likely that the road surface condition of the section immediately following the first section is also in a moist state. Therefore, it is highly likely that the road surface condition of the first section is also in a moist state. In fact, the wipers are visible in the first section image 61 shown in Figure 3, which is thought to have adversely affected the results of identifying the road surface conditions in the first section.

In this embodiment, when determining the road surface condition of the first section, not only the identification result corresponding to the first section image 61 but also the identification result corresponding to the previous section image 62, the identification result corresponding to the next section image 63, or both of these are referenced. This makes it easier to prevent erroneous determination of the road surface condition of the first section.

The configuration of the road surface condition determination device 20 according to this embodiment will be described with reference to FIG.

The road surface condition determination device 20 includes a control unit 21, a memory unit 22, a communication unit 23, an input unit 24, and an output unit 25.

The control unit 21 includes at least one processor, at least one programmable circuit, at least one dedicated circuit, or any combination of these. The processor is a general-purpose processor such as a CPU or GPU, or a dedicated processor specialized for specific processing. "CPU" is an abbreviation for central processing unit. "GPU" is an abbreviation for graphics processing unit. The programmable circuit is, for example, an FPGA. "FPGA" is an abbreviation for field-programmable gate array. The dedicated circuit is, for example, an ASIC. "ASIC" is an abbreviation for application specific integrated circuit. The control unit 21 executes processing related to the operation of the road surface condition determination device 20 while controlling each part of the road surface condition determination device 20.

The storage unit 22 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or any combination thereof. The semiconductor memory is, for example, a RAM or a ROM. "RAM" is an abbreviation for random access memory. "ROM" is an abbreviation for read only memory. The RAM is, for example, an SRAM or a DRAM. "SRAM" is an abbreviation for static random access memory. "DRAM" is an abbreviation for dynamic random access memory. The ROM is, for example, an EEPROM. "EEPROM" is an abbreviation for electrically erasable programmable read only memory. The storage unit 22 functions, for example, as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 22 stores data used in the operation of the road surface condition determination device 20 and data obtained by the operation of the road surface condition determination device 20.

The communication unit 23 includes at least one communication interface. The communication interface is, for example, a LAN interface, an interface compatible with a mobile communication standard such as LTE, the 4G standard, or the 5G standard, or an interface compatible with a short-range wireless communication standard such as Bluetooth (registered trademark). "LAN" is an abbreviation for local area network. "LTE" is an abbreviation for Long Term Evolution. "4G" is an abbreviation for 4th generation. "5G" is an abbreviation for 5th generation. The communication unit 23 receives data used in the operation of the road surface condition determination device 20, and transmits data obtained by the operation of the road surface condition determination device 20.

The input unit 24 includes at least one input interface. The input interface is, for example, a physical key, a capacitive key, a pointing device, a touch screen integral with a display, a camera, or a microphone. The input unit 24 accepts an operation to input data used in the operation of the road surface condition determination device 20. The input unit 24 may be connected to the road surface condition determination device 20 as an external input device instead of being provided in the road surface condition determination device 20. As the connection interface, for example, an interface compatible with a standard such as USB, HDMI (registered trademark), or Bluetooth (registered trademark) can be used. "USB" is an abbreviation for Universal Serial Bus. "HDMI (registered trademark)" is an abbreviation for High-Definition Multimedia Interface.

The output unit 25 includes at least one output interface. The output interface is, for example, a display or a speaker. The display is, for example, an LCD or an organic EL display. "LCD" is an abbreviation for liquid crystal display. "EL" is an abbreviation for electro luminescence. The output unit 25 outputs data obtained by the operation of the road surface condition determination device 20. The output unit 25 may be connected to the road surface condition determination device 20 as an external output device instead of being provided in the road surface condition determination device 20. As the connection interface, for example, an interface compatible with standards such as USB, HDMI (registered trademark), or Bluetooth (registered trademark) can be used.

The functions of the road surface condition determination device 20 are realized by executing a program according to this embodiment in a processor serving as the control unit 21. That is, the functions of the road surface condition determination device 20 are realized by software. The program causes a computer to execute the operations of the road surface condition determination device 20, thereby causing the computer to function as the road surface condition determination device 20. That is, the computer functions as the road surface condition determination device 20 by executing the operations of the road surface condition determination device 20 in accordance with the program.

The program may be stored in a non-transitory computer-readable medium. Examples of the non-transitory computer-readable medium include flash memory, a magnetic recording device, an optical disk, a magneto-optical recording medium, or a ROM. The program may be distributed, for example, by selling, transferring, or lending portable media such as SD cards, DVDs, or CD-ROMs on which the program is stored. "SD" is an abbreviation for Secure Digital. "DVD" is an abbreviation for digital versatile disc. "CD-ROM" is an abbreviation for compact disc read only memory. The program may be distributed by storing the program in the storage of a server and transferring the program from the server to another computer. The program may be provided as a program product.

For example, a computer temporarily stores a program stored in a portable medium or a program transferred from a server in a main storage device. The computer then reads the program stored in the main storage device with a processor and executes processing according to the read program with the processor. The computer may read the program directly from the portable medium and execute processing according to the program. The computer may execute processing according to the received program each time a program is transferred from the server to the computer. Processing may be executed by a so-called ASP-type service that does not transfer a program from the server to the computer and achieves functions only by issuing execution instructions and obtaining results. "ASP" is an abbreviation for application service provider. A program is information used for processing by a computer and includes those equivalent to a program. For example, data that is not a direct command to a computer but has properties that define computer processing falls under "those equivalent to a program."

Some or all of the functions of the road surface condition determination device 20 may be realized by a programmable circuit or a dedicated circuit as the control unit 21. In other words, some or all of the functions of the road surface condition determination device 20 may be realized by hardware.

The operation of the road surface condition determination system 10 according to this embodiment will be described with reference to FIG. 5. This operation corresponds to the road surface condition determination method according to this embodiment.

The process of step S1 is performed for each section of a fixed distance, such as 10 meters, that the vehicle 50 travels. The processes of steps S2 to S4 are also performed for each section. In this embodiment, the processes of steps S2 to S4 are performed after the process of step S1 is performed for all of the multiple sections of the road 11, but after the process of step S1 is performed for one or more sections, they may be performed in parallel with the process of step S1 being performed for the remaining sections. As a variation of this embodiment, the process of step S1 may be performed at regular intervals.

In step S1, the image sensor 30 captures images of each section of the road 11. Image data 60 obtained as the captured image is stored in a medium such as a flash memory or a magnetic recording device mounted on the vehicle 50, or is transmitted to the outside via a network such as the Internet by a communication device mounted on the vehicle 50. The control unit 21 of the road surface condition determination device 20 acquires the image data 60 stored in the medium mounted on the vehicle 50 via the communication unit 23 or another interface corresponding to a standard such as USB. Alternatively, the control unit 21 acquires the image data 60 by receiving the image data 60 transmitted by the communication device mounted on the vehicle 50 via the communication unit 23.

Each of the multiple sections is designated as a first section, and at least one section before, after, or before and after the first section is designated as a second section. Which sections are designated as second sections and how many sections are designated as second sections may be determined arbitrarily, but in this embodiment, if the section imaged first is the first section, the section immediately following the first section is designated as the second section. If the section imaged last is the first section, the section immediately preceding the first section is designated as the second section. In all other cases, the section immediately preceding the first section and the section immediately following the first section are designated as the second section.

In step S2, the control unit 21 of the road surface condition determination device 20 obtains a first identification result based on the image data 60 of the first section obtained in step S1. Specifically, the control unit 21 extracts the road surface image of the first section from the image data 60 of the first section obtained in step S1. Any method may be used to extract the road surface image, such as a method of cutting out an area of a fixed number of pixels as a road surface image, a method of cutting out a road surface area recognized by performing image analysis as a road surface image, or a method of cutting out a road surface area identified using machine learning as a road surface image. The road surface image may be cut out for each lane. The control unit 21 inputs the extracted road surface image of the first section into a discrimination model 70 for discriminating the road surface condition, thereby obtaining the discrimination result of the road surface condition of the first section from the discrimination model 70 as the first discrimination result. In this embodiment, the first discrimination result includes the probability that the road surface condition of the first section corresponds to each of the seven classes of "dry", "semi-humid", "wet and moist", "sherbet", "snowfall", "compacted snow", and "frozen".

In step S3, the control unit 21 of the road surface condition determination device 20 obtains a second identification result based on the image data 60 of the second section obtained in step S1. Specifically, the same process as that performed in step S2 on the image data 60 of the first section is performed on the image data 60 of the second section. That is, a road surface image of the second section is extracted from the image data 60 of the second section obtained in step S1. The control unit 21 inputs the extracted road surface image of the second section to the identification model 70, and obtains the identification result of the road surface condition of the second section as the second identification result from the identification model 70. In this embodiment, the second identification result includes the probability that the road surface condition of the second section corresponds to each of the seven classes of "dry", "semi-humid", "wet and moist", "sherbet", "snowfall", "packed snow", and "frozen".

In this embodiment, except when the section imaged first is the first section, the identification result of the road surface condition of the first section has already been acquired as the second identification result in the processing of step S3 executed when the section immediately preceding the first section was set as the first section. Therefore, in step S2, the control unit 21 of the road surface condition determination device 20 may simply acquire that identification result as the first identification result.

In this embodiment, if the section imaged second is the first section, the identification result of the road surface condition of the section immediately preceding the first section has already been acquired as the first identification result in the process of step S2 executed when the section immediately preceding the first section was the first section. Therefore, in step S3, the control unit 21 of the road surface condition determination device 20 may simply acquire the identification result as the second identification result. Except when the section imaged first is the first section and when the section imaged second is the first section, the identification result of the road surface condition of the section immediately preceding the first section has already been acquired as the second identification result in the process of step S3 executed when the section two sections before the first section was the first section. Therefore, in step S3, the control unit 21 of the road surface condition determination device 20 may simply acquire the identification result as the second identification result.

In step S4, the control unit 21 of the road surface condition determination device 20 determines the road surface condition by referring to the first and second identification results acquired in steps S2 and S3, respectively. Specifically, the control unit 21 determines the class to which the road surface condition of the first section falls by comparing the probability of being included in the first and second identification results.

As an example of the process of step S4, the control unit 21 of the road surface condition determination device 20 determines the class with the highest probability in the greatest number of the first and second identification results as the class to which the road surface condition of the first section corresponds. This process may be performed in any order, but is, for example, performed in the order shown in FIG. 6.

In step S401 of FIG. 6, the control unit 21 sets counter i to 1. In step S402, the control unit 21 sets Vi, which records the number of votes for label Li, to 0. In step S403, the control unit 21 sets counter j to 1.

In step S404 of FIG. 6, the control unit 21 determines whether the probability of label Li is highest in the classification result of the jth image of the section among the first and second classification results. In this embodiment, label L1 is "dry", label L2 is "semi-humid", label L3 is "moist", label L4 is "sherbet", label L5 is "accumulated snow", label L6 is "packed snow", and label L7 is "frozen". If the probability of label Li is highest in the classification result of the jth image of the section, in step S405, the control unit 21 increments Vi by 1. If the probability of a label other than label Li is highest in the classification result of the jth image of the section, the processing of step S405 is skipped.

In step S406 of FIG. 6, the control unit 21 determines whether the counter j is smaller than the number of identification results obtained in steps S2 and S3. In this embodiment, the number of identification results is 3. If the counter j is smaller than the number of identification results, the control unit 21 increments the counter j by 1 in step S407. Then, the process of step S404 is executed again. If the counter j is the same as the number of identification results, the process of step S408 is executed.

In step S408 of FIG. 6, the control unit 21 determines whether the counter i is smaller than the number of classes. In this embodiment, the number of classes is 7. If the counter i is smaller than the number of classes, in step S409, the control unit 21 increments the counter i by 1. Then, the process of step S402 is executed again. If the counter i is the same as the number of classes, the process of step S410 is executed.

In step S410 of FIG. 6, the control unit 21 determines the class corresponding to the label with the most votes as the class to which the road surface condition corresponds. Suppose that the probabilities of "moist/wet" and "dry" included in the first identification result are 30% and 50%, respectively, and the probabilities of all other labels are lower than 30%. Suppose that the probabilities of "moist/wet" and "dry" included in the second identification result for the section immediately preceding the first section are 90% and 5%, respectively. Suppose that the probabilities of "moist/wet" and "dry" included in the second identification result for the section immediately following the first section are also 90% and 5%, respectively. In this case, V1, which records the number of votes for label L1 "dry", is 1, and V3, which records the number of votes for label L3 "moist/wet", is 2. Since "moist/wet" has the most votes, the control unit 21 determines that the road surface condition is moist/wet.

As another example of the process of step S4, the control unit 21 of the road surface condition determination device 20 may determine that the class with the highest average probability in the first identification result and the second identification result is the class to which the road surface condition of the first section corresponds. This process may be performed in any procedure, but is, for example, performed in the procedure shown in FIG. 7.

In step S411 of FIG. 7, the control unit 21 sets a counter i to 1. In step S412, the control unit 21 calculates the average Ai of the probability of the label Li included in the classification results obtained in steps S2 and S3.

In step S413 of FIG. 7, the control unit 21 determines whether the counter i is smaller than the number of classes. In this embodiment, the number of classes is 7. If the counter i is smaller than the number of classes, the control unit 21 increments the counter i by 1 in step S414. Then, the process of step S412 is executed again. If the counter i is the same as the number of classes, the process of step S415 is executed.

In step S415 of FIG. 7, the control unit 21 determines the class corresponding to the label with the highest average probability as the class to which the road surface condition corresponds. Suppose that the probabilities of "moist" and "dry" included in the first identification result are 30% and 50%, respectively, and the probabilities of all other labels are lower than 30%. Suppose that the probabilities of "moist" and "dry" included in the second identification result for the section immediately preceding the first section are 90% and 5%, respectively. Suppose that the probabilities of "moist" and "dry" included in the second identification result for the section immediately following the first section are also 90% and 5%, respectively. In this case, the average A1 of the probability of label L1 "dry" is 20%, and the average A3 of the probability of label L3 "moist" is 70%. Since the average probability of "moist" is the highest, the control unit 21 determines that the road surface condition is moist.

In step S410 of FIG. 6, the road surface condition is determined to fall into the class corresponding to the label with the most votes, and in step S415 of FIG. 7, the road surface condition is determined to fall into the class corresponding to the label with the highest probability. As another example, it is also possible to differentiate the priority of the determination depending on the class. For example, for classes corresponding to dangerous road surfaces such as packed snow or frozen roads, or classes corresponding to locally occurring semi-wet road surfaces, it is easier to realize road surface management that ensures safety by giving priority to the determination result even if the number of votes is small or the probability is low.

As described above, in this embodiment, the control unit 21 of the road surface condition determination device 20 acquires an identification result of the road surface condition for each section based on image data 60 for each section obtained by capturing images of multiple sections of the road 11 using an image sensor 30 attached to a vehicle 50 traveling on the road 11. When each of the multiple sections is defined as a first section, and at least one section before, after, or before and after the first section is defined as a second section, the control unit 21 determines the road surface condition of the first section by referring to a first identification result that is an identification result of the road surface condition of the first section, and a second identification result that is an identification result of the road surface condition of the second section. Therefore, according to this embodiment, the accuracy of road surface condition determination from images is improved.

In this embodiment, the control unit 21 of the road surface condition determination device 20 inputs image data 60 for each section into a discrimination model 70 for identifying road surface conditions, and obtains the discrimination results of the road surface conditions for each section from the discrimination model 70. Therefore, according to this embodiment, the efficiency of road surface condition determination from images is improved.

In this embodiment, when the control unit 21 of the road surface condition determination device 20 performs a road surface condition determination based on images obtained by capturing images of the road 11 at regular intervals from a traveling vehicle 50, the control unit 21 can correct the determination result at a certain time using the determination results before, after, or both before and after the time in the time direction. Specifically, the control unit 21 can correct the determination result by executing a weighted voting process in the time direction as shown in FIG. 6. This can reduce erroneous determination of the road surface condition.

As a modification of this embodiment, meteorological information such as a weather forecast may be reflected. In such a modification, the control unit 21 of the road surface condition determination device 20 acquires meteorological data indicating the weather at the time the road 11 was imaged. Based on the acquired meteorological data, the control unit 21 weights the probability of being included in each of the first and second classification results for each class. This modification further improves the accuracy of road surface condition determination from images.

The present disclosure is not limited to the above-described embodiments. For example, two or more blocks shown in the block diagram may be integrated, or one block may be divided. Two or more steps shown in the flowchart may be executed in parallel or in a different order, instead of being executed in chronological order as described, depending on the processing capabilities of the device executing each step, or as needed. Other modifications are possible without departing from the spirit of the present disclosure.

For example, the road surface condition determination device 20 may be an in-vehicle device. That is, the vehicle 50 may be equipped with the road surface condition determination system 10. According to this example, the road surface condition determination based on an image can be performed by the vehicle 50.

For example, the probability of being included in each of the first and second identification results may be weighted so that the road surface condition is more likely to be determined to be wet before and after a section where the road surface condition is determined to be slush, snow, or packed snow.

This disclosure relates to a road surface condition determination device, a road surface condition determination system, a vehicle, a road surface condition determination method, and a program.

10: Road surface condition determination system, 11: Road, 20: Road surface condition determination device, 21: Control unit, 22: Memory unit, 23: Communication unit, 24: Input unit, 25: Output unit, 30: Image sensor, 50: Vehicle, 60: Image data, 61: First section image, 62: Previous section image, 63: Next section image, 70: Recognition model

Claims (10)

A road surface condition determination device comprising a control unit that obtains a road surface condition identification result for each section, including a probability that the road surface condition for each section falls into each of a plurality of classes, based on image data for each section obtained by capturing images of multiple sections of a road with an image sensor attached to a vehicle traveling on the road, and determines the class to which the road surface condition of the first section falls by comparing the probability included in a first identification result, which is the identification result of the road surface condition of the first section, with a second identification result, which is the identification result of the road surface condition of the second section, when each of the multiple sections is a first section and at least one section before, after, or before and after the first section is a second section. The road surface condition determination device according to claim 1, wherein the control unit inputs image data for each section into a discrimination model for discriminating road surface conditions, and obtains discrimination results for the road surface conditions for each section from the discrimination model. The road surface condition determination device according to claim 2 , wherein the identification model includes a trained model that has been subjected to machine learning. 4. The road surface condition determination device according to claim 1 , wherein the control unit determines a class with the highest probability in a majority of the first and second identification results as a class to which the road surface condition of the first section corresponds. 4. The road surface condition determination device according to claim 1 , wherein the control unit determines a class having the highest average probability in the first identification result and the second identification result as a class to which the road surface condition of the first section corresponds. 6. The road surface condition determination device according to claim 1, wherein the control unit acquires weather data indicating the weather at the time the road was imaged, and weights the probability of each of the first classification result and the second classification result being included for each class based on the acquired weather data. The road surface condition determination device according to any one of claims 1 to 6 ,
A road surface condition determination system comprising the image sensor. A vehicle comprising the road surface condition determination system according to claim 7 . capturing images of a plurality of sections of a road using an image sensor attached to a vehicle traveling on the road;
acquiring, by a computer, a classification result of the road surface condition for each section , including a probability that the road surface condition for each section corresponds to each of a plurality of classes, based on image data for each section obtained as a captured image;
A road surface condition determination method in which, when each of the multiple sections is defined as a first section, and at least one section before, after, or before and after the first section is defined as a second section, the computer compares the probability that the first identification result, which is an identification result of the road surface condition of the first section, will be included in a first identification result, which is an identification result of the road surface condition of the second section, to determine the class to which the road surface condition of the first section falls . A process of acquiring a classification result of a road surface condition for each section, including a probability that the road surface condition for each section falls into each of a plurality of classes, based on image data for each section obtained by capturing images of the plurality of sections of the road using an image sensor attached to a vehicle traveling on the road;
a program that causes a computer to execute a process of determining the class to which the road surface condition of the first section falls by comparing the probability that a first identification result, which is an identification result of the road surface condition of the first section, will be included in a second identification result, which is an identification result of the road surface condition of the second section, when each of the multiple sections is a first section and at least one section before, after, or before and after the first section is a second section.

Images