JP7827527B2 - Information processing device, method, and program - Google Patents

Description

This disclosure relates to an information processing device, method, and program.

There has long been a demand for technology related to the evaluation and storage of vehicle driving conditions. For example, Patent Document 1 discloses technology for evaluating driving conditions. Furthermore, technology related to the storage of driving images in the event of an accident is also known.

JP 2017-187816 A

Conventionally, driving image storage only saves images when an accident is detected (when a large acceleration is detected), and there is room for improvement in the technology for saving driving images.

The purpose of this disclosure, made in light of these circumstances, is to improve technology related to the storage of driving images.

An information processing device according to an embodiment of the present disclosure includes:
An information processing device,
Save the captured image of the vehicle's surroundings,
The control unit is provided with a control unit that decides to keep the image stored if it is determined that there is an abnormality in the behavior of the vehicle based on at least one indicator other than acceleration that indicates the behavior of the vehicle, and decides to delete the image if it is not determined that there is an abnormality in the behavior of the vehicle.

According to one embodiment of the present disclosure, a method comprises:
A method executed by an information processing device,
The method includes storing an image of the surroundings of the vehicle, and deciding to keep storing the image if it is determined that there is an abnormality in the behavior of the vehicle based on at least one indicator other than acceleration that indicates the behavior of the vehicle, and deciding to delete the image if it is not determined that there is an abnormality in the behavior of the vehicle.

A program according to an embodiment of the present disclosure includes:
In the information processing device,
The image capturing device stores an image of the surroundings of the vehicle, and when it is determined that there is an abnormality in the behavior of the vehicle based on at least one indicator other than acceleration that indicates the behavior of the vehicle, it decides to keep the image stored, and when it is not determined that there is an abnormality in the behavior of the vehicle, it decides to delete the image.

One embodiment of the present disclosure improves technology for saving driving images.

1 is a block diagram illustrating a schematic configuration of a system according to an embodiment of the present disclosure. 1 is a block diagram showing a schematic configuration of a vehicle. FIG. 1 is a block diagram showing a schematic configuration of an information processing device. 10 is a flowchart illustrating an operation of the information processing device.

Embodiments of the present disclosure are described below.

<Overview of the embodiment>
An overview of a system 1 according to an embodiment of the present disclosure will be described with reference to Fig. 1. The system 1 includes a vehicle 10 and an information processing device 20. The vehicle 10 and the information processing device 20 are communicably connected to a network 30 including, for example, the Internet and a mobile communication network.

The vehicle 10 is, for example, an automobile, but is not limited to this and may be any vehicle. The automobile may be any type of automobile, such as a gasoline vehicle, diesel vehicle, HEV (hybrid electric vehicle), PHEV (plug-in hybrid electric vehicle), BEV (battery electric vehicle), or FCEV (fuel cell electric vehicle). In this embodiment, the vehicle 10 is driven by a driver, but the driving may be automated at any level. The level of automation is, for example, any of levels 1 to 5 in the SAE (Society of Automotive Engineers) level classification. The vehicle 10 may also be a dedicated vehicle for MaaS (Mobility as a Service). The number of vehicles 10 included in the system 1 may be determined arbitrarily.

In this embodiment, the vehicle 10 has the function of acquiring, storing, and/or transmitting images of the area around the vehicle 10. The vehicle 10 also acquires and transmits information (indicators) that indicate the behavior of the vehicle 10.

The information processing device 20 is, for example, a computer such as a server device. The information processing device 20 is capable of communicating with the vehicle 10 via the network 30.

First, an overview of this embodiment will be described, and details will be provided later. The information processing device 20 saves images of the area around the vehicle 10. The information processing device 20 determines whether there is an abnormality in the behavior of the vehicle 10 based on at least one indicator other than acceleration that indicates the behavior of the vehicle 10. If it is determined that there is an abnormality in the behavior of the vehicle 10, the information processing device 20 decides to continue saving the images. If it is not determined that there is an abnormality in the behavior of the vehicle 10, the information processing device 20 decides to delete the images.

As described above, according to this embodiment, when it is determined that the behavior of the vehicle 10 is abnormal based on at least one indicator other than acceleration, images of the area around the vehicle 10 are kept stored. Here, the behavior of the vehicle 10 includes not only behaviors determined to be dangerous based on acceleration (G-value), but also behaviors that may lead to dangerous driving even if they are not determined to be dangerous based on acceleration (e.g., the vehicle 10 making an abnormal sound, or the user uttering a surprised voice). Both types of behavior are equally likely to require the user to save images. However, with conventional cameras such as dashcams, images of behaviors that are not determined to be dangerous based on acceleration are deleted. This makes it difficult to keep all images of behaviors that are likely to require the user to save. On the other hand, according to this embodiment, it is easier to keep images of behaviors that may lead to dangerous driving, regardless of acceleration. Therefore, technology related to driving image storage is improved in that it is easier to keep all images of behaviors that are likely to require the user to save.

Next, we will explain each component of System 1 in detail.

<Vehicle configuration>
As shown in FIG. 2 , the vehicle 10 includes a communication unit 11 , an acquisition unit 12 , an imaging unit 13 , an output unit 14 , an input unit 15 , a storage unit 16 , and a control unit 17 .

The communication unit 11 includes one or more communication interfaces that connect to the network 30. The communication interfaces are compatible with mobile communication standards such as, but not limited to, 4G (4th Generation) or 5G (5th Generation). In this embodiment, the vehicle 10 communicates with the information processing device 20 via the communication unit 11 and the network 30.

The acquisition unit 12 includes one or more devices that acquire location information of the vehicle 10. Specifically, the acquisition unit 12 includes, for example, a receiver compatible with the Global Positioning System (GPS), but is not limited to this and may include a receiver compatible with any satellite positioning system. The acquisition unit 12 also includes any sensor module capable of acquiring information indicating the state of the vehicle 10. For example, the sensor module may include a vibration sensor, an infrared sensor, a speed sensor, an angular velocity sensor, an acceleration sensor, a geomagnetic sensor, a distance sensor such as LiDAR (light detection and ranging), or a combination of these.

The imaging unit 13 includes one or more cameras. Each camera included in the imaging unit 13 may be provided on the vehicle 10 so as to be able to capture images of subjects outside or inside the vehicle. In this embodiment, the imaging unit 13 includes a front camera capable of capturing images of subjects in front of the vehicle 10, a rear camera capable of capturing images of subjects behind the vehicle 10, and an interior camera capable of capturing images of subjects inside the vehicle 10. Alternatively, the imaging unit 13 may include an omnidirectional camera capable of capturing images of subjects around the vehicle 10. The imaging unit 13 may also include an input interface capable of communicating with cameras such as a drive recorder installed on the vehicle 10 so as to be able to capture images of subjects outside or inside the vehicle, and may be able to acquire images of multiple users riding in the vehicle 10 from the cameras.

The output unit 14 may include one or more output devices that output information to notify the user. Examples of such output devices include, but are not limited to, a display that outputs information visually or a speaker that outputs information audibly. The output unit 14 may also include an interface for connecting an external output device.

The input unit 15 includes one or more input devices that detect user input. In this embodiment, the input unit 15 includes a microphone that receives input of the user's voice and sounds emitted by the vehicle 10. However, the input device is not limited to these examples and may include, for example, physical keys, capacitive keys, or a touchscreen integrated with a display. The input unit 15 may also include an interface for connecting an external input device.

The storage unit 16 includes one or more memories. Examples of memories include, but are not limited to, semiconductor memory, magnetic memory, or optical memory. Each memory included in the storage unit 16 may function as, for example, a main memory, an auxiliary memory, or a cache memory. The storage unit 16 stores any information used in the operation of the vehicle 10. For example, the storage unit 16 may store system programs, application programs, embedded software, etc. The information stored in the storage unit 16 may be updateable with information obtained from the network 30 via the communication unit 11, for example. In this embodiment, the storage unit 16 may store images (or videos) of the area around the vehicle 10.

The control unit 17 includes one or more processors, one or more programmable circuits, one or more dedicated circuits, or a combination of these. The processor may be, for example, a general-purpose processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or a dedicated processor specialized for specific processing, but is not limited to these. The programmable circuit may be, for example, but is not limited to, an FPGA (Field-Programmable Gate Array). The dedicated circuit may be, for example, but is not limited to, an ASIC (Application Specific Integrated Circuit). The control unit 17 controls the overall operation of the vehicle 10.

<Configuration of information processing device>
As shown in FIG. 3, the information processing device 20 includes a communication unit 21, a storage unit 22, and a control unit 23.

The communication unit 21 includes one or more communication interfaces that connect to the network 30. The communication interfaces are compatible with, for example, a mobile communication standard, a wired LAN (Local Area Network) standard, or a wireless LAN standard, but are not limited to these and may be compatible with any communication standard. In this embodiment, the information processing device 20 communicates with the vehicle 10 via the communication unit 21 and the network 30.

The storage unit 22 includes one or more memories. Each memory included in the storage unit 22 may function, for example, as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 22 stores any information used in the operation of the information processing device 20. For example, the storage unit 22 may store system programs, application programs, databases, map information, etc. The information stored in the storage unit 22 may be updateable with information obtained from the network 30 via the communication unit 21, for example. In this embodiment, the storage unit 22 can store images (or videos) of the area around the vehicle 10 that are received from the vehicle 10.

The control unit 23 includes one or more processors, one or more programmable circuits, one or more dedicated circuits, or a combination of these. The control unit 23 controls the overall operation of the information processing device 20.

<Operation flow of information processing device>
The operation of the information processing device 20 according to this embodiment will be described with reference to Fig. 4. The operation of Fig. 4 corresponds to the method according to this embodiment. The operation of Fig. 4 is repeatedly executed, for example, at a predetermined cycle. The predetermined cycle can be determined arbitrarily.

Step S100: The control unit 23 of the information processing device 20 saves the captured image of the area around the vehicle 10.

Specifically, the control unit 23 of the information processing device 20 receives images of the surroundings of the vehicle 10 (hereinafter also referred to as "captured images") captured at a given frame rate by the imaging unit 13 of the vehicle 10 from the vehicle 10 via the communication unit 21 and the network 30. The control unit 23 stores the received captured images in the memory unit 22, thereby preserving the captured images of the surroundings of the vehicle 10. However, this is not a limited method for storing captured images, and any method can be used. Furthermore, although the captured images are stored in the memory unit 22 in this embodiment, this is not a limitation. For example, the captured images may be stored in external storage connected to the information processing device 20, or may be stored in the memory unit 16 of the vehicle 10.

Furthermore, in this embodiment, simultaneously with the imaging unit 13 of the vehicle 10 capturing an image, the input unit 15 records the behavior of the user aboard the vehicle 10 and sounds emitted by the vehicle 10, and the acquisition unit 12 measures vibrations emitted by the vehicle 10 and the distance between the vehicle 10 and objects located around the vehicle 10. Then, the control unit 23 of the information processing device 20 receives the above-described captured image, as well as the following information (1) to (4) accompanying the captured image, from the vehicle 10 via the communication unit 21 and the network 30.
(1) Information indicating the behavior of the user while riding in the vehicle 10 (hereinafter also referred to as “first information”)
(2) Information indicating whether or not abnormal sound is being generated in the vehicle 10 (hereinafter also referred to as “second information”)
(3) Information indicating the presence or absence of abnormal vibration of the vehicle 10 (hereinafter also referred to as “third information”)
(4) Information indicating the distance between the vehicle 10 and an object located around the vehicle 10 (hereinafter also referred to as “fourth information”)

The first information in (1) above may include any information indicating the user's behavior, such as the user's vocalizations or body movements while aboard the vehicle 10. There may be one or more users aboard the vehicle 10, but in the following description, it is assumed that only one user, i.e., the driver of the vehicle 10, is aboard the vehicle 10. The user's behavior may include any behavior, but in this embodiment, it is the user's utterance of a first voice, which is the user's calm voice, or a second voice different from the first voice. Therefore, the first information in this embodiment is information indicating the user's voice recorded via the input unit 15 of the vehicle 10. The user's voice recorded via the input unit 15 includes the first voice or the second voice. The second voice, which is different from the user's first voice, which is the user's calm voice, may be any voice suggesting a critical situation for the user, but in this embodiment, it is the user's surprised voice (typically, a scream).

The second information (2) above may include any information indicating the sound emitted by the vehicle 10, but in this embodiment, it is information indicating the sound emitted by the vehicle 10 recorded via the input unit 15 of the vehicle 10. The sound emitted by the vehicle 10 recorded via the input unit 15 includes normal sounds and abnormal sounds. Normal sounds are sounds emitted by components of the vehicle 10 when there is no accident or malfunction of the vehicle 10. Abnormal sounds are sounds emitted by components of the vehicle 10 when there is an accident or malfunction of the vehicle 10. For example, abnormal sounds include sounds caused by a collision or contact between the vehicle 10 and another object, the sound of tires rubbing during sudden braking, or sounds caused by a vehicle malfunction such as engine trouble.

The third information (3) above may include any information indicating vibrations emitted by the vehicle 10, but in this embodiment, it is a measured value of vibrations emitted by the vehicle 10 measured via the acquisition unit 12 of the vehicle 10. The vibrations emitted by the vehicle 10 measured via the acquisition unit 12 include normal vibrations and abnormal vibrations. Normal vibrations refer to vibrations emitted by components of the vehicle 10 when there is no accident or malfunction of the vehicle 10. Abnormal vibrations refer to vibrations emitted by components of the vehicle 10 when there is an accident or malfunction of the vehicle 10. For example, abnormal vibrations include vibrations caused by a collision or contact between the vehicle 10 and another object, vibrations of the vehicle body during sudden braking, or vibrations caused by a vehicle malfunction such as engine trouble.

The fourth information (4) above may include any information indicating the distance between the vehicle 10 and an object located in the vicinity of the vehicle 10, but in this embodiment it is a measurement value of the distance between the vehicle 10 and an object located in the vicinity of the vehicle 10 (hereinafter also referred to as "distance D") measured via the acquisition unit 12 of the vehicle 10. The object may be, for example, another vehicle, an obstacle, a person, an animal, or a combination thereof. Any method can be used to calculate the distance D. For example, the control unit 17 of the vehicle 10 may calculate the distance D by analyzing the position information of the vehicle 10 acquired via the acquisition unit 12, distance information to surrounding objects acquired from a distance sensor or the like, and/or captured images acquired via the imaging unit 13 of the vehicle 10.

The control unit 23 stores the received first information to fourth information in the memory unit 22. Then, each time the control unit 23 performs the operation of FIG. 4, it acquires the first information, second information, third information, or fourth information associated with any number of frames of captured images by reading it from the memory unit 22. Note that the first information to fourth information acquired here can be used as at least one index other than acceleration in the vehicle behavior determination described below. However, acquisition of the first information to fourth information is not limited to these examples, and any method can be used.

Step S101: The control unit 23 acquires at least one indicator other than acceleration that indicates the behavior of the vehicle 10. The at least one indicator includes any information other than acceleration that can identify the behavior of the vehicle 10. The at least one indicator may be an indicator acquired from outside the vehicle 10, an indicator internal to the vehicle 10, or both.

Specifically, the control unit 23 acquires at least one indicator by reading from the memory unit 22 at least one of the first information to the fourth information stored in the memory unit 22 in step S100. As will be understood from the following explanation, the more indicators acquired here, the more likely it is that images of behavior that may lead to dangerous driving, regardless of acceleration, will be saved. This makes it easier to save all images of behavior that the user is likely to need to save. Therefore, the more indicators acquired, the better. In the following explanation, it is assumed that the at least one indicator includes all of the first information to the fourth information.

Next, the control unit 23 of the information processing device 20 determines whether or not there is an abnormality in the behavior of the vehicle 10 based on at least one indicator acquired in step S101 (hereinafter also referred to as "vehicle behavior determination"). Specifically, the control unit 23 performs the operations of steps S102 to S105, which will be described later, to perform the vehicle behavior determination. The vehicle behavior determination is outlined as follows: If the determination result in any of steps S102 to S105 is positive, the control unit 23 determines that there is an abnormality in the behavior of the vehicle 10. In this case, the captured image stored in the memory unit 22 in step S100 is maintained (step S107, which will be described later). On the other hand, if the determination results in all steps S102 to S105 are negative, the control unit 23 does not determine that there is an abnormality in the behavior of the vehicle 10. In this case, the captured image stored in the memory unit 22 in step S100 is deleted (step S106, which will be described later).

The vehicle behavior determination according to this embodiment will be described in detail below. In order to focus on making it easier to maintain the storage of captured images during periods where it was previously difficult to do so, the following description will be based on the premise that vehicle behavior determination is performed during periods when the acceleration of the vehicle 10 is below a threshold. For periods when the acceleration of the vehicle 10 exceeds the threshold, the same processing as in the past may be performed. For example, when the control unit 23 detects that the acceleration of the vehicle 10 has exceeded the threshold, it may decide to maintain the storage of a predetermined number of frames of captured images in the memory unit 22 (e.g., several seconds before and after the detection point). On the other hand, when the acceleration of the vehicle 10 is below the threshold and the determination results in steps S102 to S105 are negative, the control unit 23 may decide to delete the captured images stored in the memory unit 22.

Step S102: The control unit 23 determines whether or not there is an abnormality in the user's behavior based on the first information acquired in step S101 (hereinafter also referred to as "user behavior determination"). If it is determined that there is an abnormality in the user's behavior (step S102-Yes), the process proceeds to step S107. On the other hand, if it is not determined that there is an abnormality in the user's behavior (step S102-No), the process proceeds to step S103.

Specifically, the control unit 23 determines whether the user's voice acquired in step S101 is a first voice or a second voice (hereinafter also referred to as "voice determination"). Any method can be used for voice determination. For example, the control unit 23 of the information processing device 20 may construct a model that classifies neutral voices and voices suggesting a critical situation. Any method can be used for constructing the model. For example, a model can be constructed by collecting a large number of samples by recording the neutral voices of multiple men and women and the voices suggesting a critical situation, extracting physical features such as spectrograms or cepstrograms of each voice, and performing machine learning such as deep learning. The control unit 23 may perform voice analysis of the received user's voice based on the constructed model, and based on the results of the voice analysis, determine whether the voice is the user's first voice or the second voice. If the result of the voice analysis indicates that the received user's voice is the second voice, the control unit 23 determines that there is an abnormality in the user's behavior.

Step S103: If it is determined that the user's behavior is not abnormal (Step S102—No), the control unit 23 determines whether or not an abnormal sound from the vehicle 10 has been detected, based on the second information acquired in Step S101. If an abnormal sound from the vehicle 10 has been detected (Step S103—Yes), the process proceeds to Step S107. On the other hand, if an abnormal sound from the vehicle 10 has not been detected (Step S103—No), the process proceeds to Step S104.

Specifically, the control unit 23 determines whether the sound emitted by the vehicle 10 acquired in step S101 is a normal sound or an abnormal sound (hereinafter also referred to as "vehicle sound determination"). If the sound is an abnormal sound, the control unit 23 determines that an abnormal sound from the vehicle 10 has been detected. Any method can be used for vehicle sound determination. For example, the control unit 23 of the information processing device 20 may determine that the sound emitted by the vehicle 10 is a normal sound if a value (e.g., in dB) indicating the loudness of the sound emitted by the vehicle 10 is equal to or less than a predetermined threshold, and may determine that the sound emitted by the vehicle 10 is an abnormal sound if the value exceeds the predetermined threshold. The predetermined threshold can be set arbitrarily, and for example, a threshold may be set for each frequency band, such as high and low tones.

Step S104: If no abnormal sound of the vehicle 10 is detected (Step S103—No), the control unit 23 determines whether or not abnormal vibration of the vehicle 10 has been detected based on the third information acquired in Step S101. If abnormal vibration of the vehicle 10 is detected (Step S104—Yes), the process proceeds to Step S107. On the other hand, if abnormal vibration of the vehicle 10 is not detected (Step S104—No), the process proceeds to Step S105.

Specifically, the control unit 23 determines whether the vibrations emitted by the vehicle 10 acquired in step S101 are normal or abnormal (hereinafter also referred to as "vibration determination"). If the vibrations are abnormal, the control unit 23 determines that abnormal vibrations of the vehicle 10 have been detected. Any method can be used for vibration determination. For example, the control unit 23 of the information processing device 20 may determine that the vibrations emitted by the vehicle 10 are normal if a value (e.g., in Hz) indicating the frequency of the elastic vibrations emitted by the vehicle 10 is within a predetermined threshold range, and may determine that the vibrations emitted by the vehicle 10 are abnormal if the value exceeds the predetermined threshold range. The predetermined threshold range can be set arbitrarily. Alternatively, the control unit 23 of the information processing device 20 may determine that the vibrations emitted by the vehicle 10 are normal if a value indicating the vibration intensity (e.g., vibration amplitude) of the elastic vibrations emitted by the vehicle 10 is equal to or less than a predetermined threshold, and may determine that the vibrations emitted by the vehicle 10 are abnormal if the value exceeds the predetermined threshold. The predetermined threshold can be set arbitrarily.

Step S105: If abnormal vibration of the vehicle 10 is not detected (Step S104—No), the control unit 23 determines, based on the fourth information acquired in Step S101, whether an object approaching at a distance equal to or less than a predetermined threshold (hereinafter also referred to as an "approaching object") has been detected. If an approaching object is detected (Step S105—Yes), the process proceeds to Step S107. On the other hand, if an approaching object is not detected (Step S105—No), the process proceeds to Step S106.

Specifically, the control unit 23 determines whether the distance D acquired in step S101 is equal to or less than a predetermined threshold (hereinafter also referred to as "distance determination"). If the distance D is equal to or less than the predetermined threshold, the control unit 23 determines that an approaching object has been detected. As described above, the expected approaching object may be another vehicle, an obstacle, a person, an animal, or a combination of these. The predetermined threshold can be set arbitrarily, and, for example, a different value may be set for each expected approaching object or for each traveling speed of the vehicle 10. For example, the predetermined threshold for another vehicle may be set to any distance that does not violate the obligation to maintain a safe distance between vehicles as stipulated in Japan's Road Traffic Act.

Step S106: If an approaching object is not detected (Step S105—No), the control unit 23 decides to delete the captured image stored in the memory unit 22 in Step S100. The process then ends.

Specifically, the control unit 23 deletes the captured image stored in the storage unit 22 in step S100 from the storage unit 22. Any method can be used to delete the captured image. For example, the control unit 23 may delete the captured image from the storage unit 22 after a predetermined time has elapsed since the captured image was saved in the storage unit 22.

Step S107: If it is determined that the user's behavior is abnormal (Step S102—Yes), if an abnormal sound is detected from the vehicle 10 (Step S103—Yes), if abnormal vibration is detected from the vehicle 10 (Step S104—Yes), or if an approaching object is detected (Step S105—Yes), the control unit 23 decides to maintain the storage of the captured image stored in the memory unit 22 in Step S100. The process then ends.

Specifically, if the determination result of any of steps S102 to S105 is affirmative, the control unit 23 determines that there is an abnormality in the behavior of the vehicle 10 and decides to maintain the storage of the captured image stored in the memory unit 22 in step S100. The control unit 23 does not delete the captured image from the memory unit 22, but maintains the storage of the captured image in the memory unit 22. Any method can be used to maintain the storage of the captured image. For example, the control unit 23 may maintain the storage of a predetermined number of frames of captured images in the memory unit 22 (e.g., several seconds before and after the point in time when it is determined that there is an abnormality in the behavior of the vehicle 10). Note that, although the determination is performed in the order of steps S102 to S105 in the flowchart of FIG. 4, steps S102 to S105 may be arranged in any order, and the determination of each step may be performed in any order. Furthermore, each step may be performed in parallel.

In this way, through steps S101 to S107, if the control unit 23 determines that there is an abnormality in the behavior of the vehicle 10 based on at least one indicator other than acceleration that indicates the behavior of the vehicle 10, it decides to maintain the storage of the captured image stored in the memory unit 22 in step S100, and if it does not determine that there is an abnormality in the behavior of the vehicle 10, it decides to delete the captured image.

As described above, the information processing device 20 according to this embodiment stores images of the area around the vehicle 10. If the information processing device 20 determines that there is an abnormality in the behavior of the vehicle 10 based on at least one indicator other than acceleration that indicates the behavior of the vehicle 10, it decides to keep the images stored, and if it does not determine that there is an abnormality in the behavior of the vehicle 10, it decides to delete the images.

Unlike conventional methods, this configuration makes it easier to maintain the storage of images of behaviors that could lead to dangerous driving, regardless of acceleration. Therefore, technology related to the storage of driving images is improved in that it makes it easier to store all images of behaviors that are likely to require the user to save.

While the present disclosure has been described based on various drawings and examples, it should be noted that those skilled in the art may make various modifications and alterations based on the present disclosure. Therefore, it should be noted that these modifications and alterations are included within the scope of the present disclosure. For example, the functions included in each component or step may be rearranged so as not to result in logical inconsistencies, and multiple components or steps may be combined or divided into one.

For example, in one variation of this embodiment, the control unit 23 of the information processing device 20 may determine whether or not there is an abnormality in the behavior of the vehicle 10 based on the acceleration of the vehicle 10 in addition to at least one indicator. That is, the control unit 23 may perform parallel processing of saving captured images based on the vehicle behavior determination according to this embodiment and saving captured images based on conventional acceleration. In this case, in step S101, the control unit 23 receives, from the vehicle 10 via the communication unit 21 and the network 30, information indicating the acceleration of the vehicle 10 measured via the acquisition unit 12 of the vehicle 10 (e.g., a measured acceleration value) in addition to at least one indicator. The control unit 23 may determine whether or not there is an abnormality in the behavior of the vehicle 10 based on the received information indicating the acceleration. Any method can be used for determining vehicle behavior based on acceleration. For example, if acceleration exceeding a predetermined threshold is detected in the vehicle 10 (e.g., during sudden braking), the control unit 23 may determine that there is an abnormality in the behavior of the vehicle 10.

For example, in the above embodiment, voice judgment is performed as the user behavior judgment. However, the control unit 23 may perform another process to judge whether there is anything abnormal in the user's behavior, in addition to or instead of voice judgment. For example, the control unit 23 may receive images showing the user's body movements captured at an arbitrary frame rate by an in-vehicle camera of the vehicle 10 from the vehicle 10 via the communication unit 21 and the network 30. The control unit 23 may construct a model that classifies calm body movements from body movements that suggest a critical situation by performing machine learning such as deep learning described in step S102. The control unit 23 may perform image analysis of the user's body movements shown in the received images based on the constructed model, and perform user behavior judgment based on the results of the image analysis.

Furthermore, for example, in the above-described embodiment, the configuration and operation of the information processing device 20 may be distributed among multiple computers that can communicate with each other. Furthermore, for example, an embodiment in which some or all of the components of the information processing device 20 are provided in the vehicle 10 is also possible. For example, a drive recorder installed in the vehicle 10 may include some or all of the components of the information processing device 20.

Furthermore, an embodiment is also possible in which, for example, a general-purpose computer functions as the information processing device 20 according to the above-described embodiment. Specifically, a program describing the processing content that realizes each function of the information processing device 20 according to the above-described embodiment is stored in the memory of the general-purpose computer, and the program is read and executed by a processor. Therefore, the present disclosure can also be realized as a program executable by a processor, or a non-transitory computer-readable medium that stores the program.

Some embodiments of the present disclosure will be described below as examples, however, it should be noted that the embodiments of the present disclosure are not limited to these examples.
[Appendix 1]
An information processing device,
Save the captured image of the vehicle's surroundings,
An information processing device comprising a control unit that decides to keep the image stored if it is determined that there is an abnormality in the behavior of the vehicle based on at least one indicator other than acceleration that indicates the behavior of the vehicle, and decides to delete the image if it is not determined that there is an abnormality in the behavior of the vehicle.
[Appendix 2]
10. The information processing device according to claim 1,
the at least one indicator includes information indicative of a behavior of a user while riding in the vehicle;
The information processing device, wherein the control unit determines that there is an abnormality in the behavior of the vehicle when it is determined that there is an abnormality in the behavior of the user.
[Appendix 3]
3. The information processing device according to claim 2,
the user behavior is that the user utters a first voice that is a calm voice of the user or a second voice that is different from the first voice;
The information processing device, wherein the control unit determines that there is something abnormal in the behavior of the user when the utterance of the second voice by the user is detected.
[Appendix 4]
An information processing device according to any one of Supplementary Notes 1 to 3,
the at least one indicator includes information indicating whether or not an abnormal sound is present in the vehicle;
The control unit determines that there is an abnormality in the behavior of the vehicle when the abnormal sound is detected.
[Appendix 5]
5. The information processing device according to claim 1,
the at least one indicator includes information indicating whether or not abnormal vibration of the vehicle is occurring,
The control unit determines that there is an abnormality in the behavior of the vehicle when the abnormal vibration is detected.
[Appendix 6]
6. The information processing device according to any one of Supplementary Notes 1 to 5,
the at least one indicator includes a distance between the vehicle and an object located around the vehicle;
The information processing device, wherein the control unit determines that there is an abnormality in the behavior of the vehicle when an object approaching at a distance equal to or less than a predetermined threshold is detected.
[Appendix 7]
7. The information processing device according to claim 1,
The control unit determines whether or not there is an abnormality in the behavior of the vehicle based on the at least one indicator and also on the acceleration of the vehicle.
[Appendix 8]
A method executed by an information processing device,
A method comprising: storing an image of the surroundings of a vehicle; and deciding to keep the image stored if it is determined that there is an abnormality in the behavior of the vehicle based on at least one indicator other than acceleration that indicates the behavior of the vehicle, and deciding to delete the image if it is not determined that there is an abnormality in the behavior of the vehicle.
[Appendix 9]
9. The method of claim 8, further comprising:
the at least one indicator includes information indicative of a behavior of a user while riding in the vehicle;
The method further includes determining that a behavior of a front vehicle is abnormal when it is determined that a behavior of the user is abnormal.
[Supplementary Note 10]
10. The method of claim 9,
the user's behavior is the user uttering a first voice that is a calm voice of the user or a second voice that is different from the first voice;
The method further includes determining that the user's behavior is abnormal if the user's utterance of the second voice is detected.
[Appendix 11]
11. The method of any one of appendices 8 to 10, comprising:
the at least one indicator includes information indicating whether or not an abnormal sound is present in the vehicle;
The method further includes determining that there is an abnormality in the behavior of the vehicle if the abnormal sound is detected.
[Appendix 12]
12. The method of any one of appendices 8 to 11, comprising:
the at least one indicator includes information indicating whether or not abnormal vibration of the vehicle is occurring,
The method further includes determining that there is an abnormality in the behavior of the vehicle if the abnormal vibration is detected.
[Appendix 13]
13. The method of any one of appendices 8 to 12, comprising:
the at least one indicator includes a distance between the vehicle and an object located around the vehicle;
The method further includes determining that there is an abnormality in the behavior of the vehicle when an object is detected approaching at a distance equal to or less than a predetermined threshold.
[Appendix 14]
14. The method of any one of appendices 8 to 13, comprising:
The method further includes determining whether there is an abnormality in the behavior of the vehicle based on the acceleration of the vehicle in addition to the at least one indicator.
[Appendix 15]
In the information processing device,
A program that executes the following: saving an image of the periphery of a vehicle; and, if it is determined that there is an abnormality in the behavior of the vehicle based on at least one indicator other than acceleration that indicates the behavior of the vehicle, deciding to keep the image saved, and, if it is not determined that there is an abnormality in the behavior of the vehicle, deciding to delete the image.
[Appendix 16]
16. The program of claim 15,
the at least one indicator includes information indicative of a behavior of a user while riding in the vehicle;
The program further causes the information processing device to determine that there is an abnormality in behavior of a vehicle in front of the user when it is determined that there is an abnormality in behavior of the user.
[Appendix 17]
17. The program of claim 16,
the user's behavior is the user uttering a first voice that is a calm voice of the user or a second voice that is different from the first voice;
The program further causes the information processing device to determine that there is an abnormality in the behavior of the user when the utterance of the second voice by the user is detected.
[Appendix 18]
18. The program according to any one of appendices 15 to 17,
the at least one indicator includes information indicating whether or not an abnormal sound is present in the vehicle;
The program further causes the information processing device to determine that there is an abnormality in the behavior of the vehicle when the abnormal sound is detected.
[Appendix 19]
19. The program according to any one of appendices 15 to 18,
the at least one indicator includes information indicating whether or not abnormal vibration of the vehicle is occurring,
The program further causes the information processing device to determine that there is an abnormality in the behavior of the vehicle when the abnormal vibration is detected.
[Appendix 20]
20. The program according to any one of appendices 15 to 19,
the at least one indicator includes a distance between the vehicle and an object located around the vehicle;
The program further causes the information processing device to determine that there is an abnormality in the behavior of the vehicle when an object approaching at a distance equal to or less than a predetermined threshold is detected.

REFERENCE SIGNS LIST 1 System 10 Vehicle 11 Communication unit 12 Acquisition unit 13 Imaging unit 14 Output unit 15 Input unit 16 Storage unit 17 Control unit 20 Information processing device 21 Communication unit 22 Storage unit 23 Control unit 30 Network

Claims (12)

An information processing device,
Save the captured image of the vehicle's surroundings,
a plurality of pieces of information accompanying the image, which can be used as at least one indicator other than acceleration that indicates the behavior of the vehicle, including first information indicating the behavior of the user while riding in the vehicle, second information indicating whether or not there is an abnormal sound from the vehicle, third information indicating whether or not there is an abnormal vibration from the vehicle, and fourth information indicating a distance between the vehicle and an object located around the vehicle ;
reading at least one piece of information from the plurality of pieces of information to obtain the at least one index;
An information processing device comprising a control unit that decides to keep the image stored if it is determined that there is an abnormality in the vehicle's behavior based on the at least one acquired indicator, and decides to delete the image if it is not determined that there is an abnormality in the vehicle's behavior. 2. The information processing device according to claim 1,
the user behavior of the first information is a vocalization or a body movement of the user ,
The information processing device, wherein the control unit determines that there is an abnormality in the user 's behavior when it is determined that there is an abnormality in the user's vocalization or the user's body movement . 2. The information processing device according to claim 1 ,
the user behavior of the first information is that the user uttered a first voice that is a calm voice of the user or a second voice that is different from the first voice,
The information processing device, wherein the control unit determines that there is something abnormal in the behavior of the user when the utterance of the second voice by the user is detected. 4. The information processing device according to claim 1,
The control unit determines whether or not there is an abnormality in the behavior of the vehicle based on the at least one indicator and also on the acceleration of the vehicle. A method executed by an information processing device,
Storing images of the surroundings of the vehicle;
storing a plurality of pieces of information accompanying the image, which can be used as at least one indicator other than acceleration that indicates the behavior of the vehicle, including first information indicating the behavior of the user while riding in the vehicle, second information indicating the presence or absence of abnormal noise from the vehicle, third information indicating the presence or absence of abnormal vibration from the vehicle, and fourth information indicating the distance between the vehicle and an object located around the vehicle ;
The method includes: reading at least one piece of information from the plurality of pieces of information to obtain the at least one indicator; and deciding to keep the image stored if it is determined that there is an abnormality in the behavior of the vehicle based on the obtained at least one indicator, and deciding to delete the image if it is not determined that there is an abnormality in the behavior of the vehicle. 6. The method of claim 5 ,
the user behavior of the first information is a vocalization or a body movement of the user ,
The method further includes determining that there is an abnormality in the behavior of a previous previous user when it is determined that there is an abnormality in the user 's vocalization or body movement . 6. The method of claim 5 ,
The user behavior of the first information is that the user utters a first voice that is a calm voice of the user or a second voice that is different from the first voice,
The method further includes determining that the user's behavior is abnormal if the user's utterance of the second voice is detected. 8. The method according to any one of claims 5 to 7 ,
The method further includes determining whether there is an abnormality in the behavior of the vehicle based on the acceleration of the vehicle in addition to the at least one indicator. In the information processing device,
Storing images of the surroundings of the vehicle;
storing a plurality of pieces of information accompanying the image, which can be used as at least one indicator other than acceleration that indicates the behavior of the vehicle, including first information indicating the behavior of the user while riding in the vehicle, second information indicating the presence or absence of abnormal noise from the vehicle, third information indicating the presence or absence of abnormal vibration from the vehicle, and fourth information indicating the distance between the vehicle and an object located around the vehicle ;
A program that reads out at least one piece of information from the plurality of pieces of information to acquire the at least one indicator; and if it is determined that there is an abnormality in the behavior of the vehicle based on the acquired at least one indicator, decides to keep the image stored, and if it is not determined that there is an abnormality in the behavior of the vehicle, decides to delete the image. 10. The program according to claim 9 ,
the user behavior of the first information is a vocalization or a body movement of the user ,
The program further causes the information processing device to determine that there is an abnormality in the behavior of a previous previous user when it is determined that there is an abnormality in the user 's vocalization or body movement . 10. The program according to claim 9 ,
The user behavior of the first information is that the user utters a first voice that is a calm voice of the user or a second voice that is different from the first voice,
The program further causes the information processing device to determine that there is something abnormal in the behavior of the user when the utterance of the second voice by the user is detected. 12. The program according to claim 9,
The program further causes the information processing device to determine whether or not there is an abnormality in the behavior of the vehicle based on the acceleration of the vehicle in addition to the at least one indicator.