JP7643053B2 - Recording system and recording method - Google Patents

Description

The present invention relates to a recording system and a recording method.

In recent years, the number of vehicles equipped with drive recorders has been increasing. Drive recorders are used as recording devices that capture images of the inside and outside of the vehicle and record the captured data. The data recorded in the drive recorder is used to investigate the cause of an accident when one occurs, or to confirm the facts when an incident occurs.

For example, Patent Document 1 discloses a recording device that collects audio around the vehicle and records images and audio around the vehicle in a recording unit if it determines that the vehicle is being tailgated.

JP 2020-160630 A

In the event of an abnormal event unrelated to the vehicle, such as an accident or incident involving another vehicle, data from the drive recorder may be provided as witness information to the police, insurance companies, the media, the cloud, etc. Drive recorder data may contain audio of personal conversations inside the vehicle, etc., which may be information related to the privacy of the witnesses themselves. When providing data as witness information, there is a need to protect the privacy of such witnesses.

The present invention has been made to solve the above-mentioned problems, and provides a recording system and recording method that optimally protects the privacy of witnesses when an abnormality occurs that is not related to the vehicle.

A recording system according to one aspect of the present invention includes an acquisition unit, an event detection unit, and a recording control unit. The acquisition unit acquires image data of an area including at least the periphery of a vehicle, and audio data of audio of an area including at least the periphery of the vehicle. The event detection unit detects the occurrence of an event in which an abnormality is detected around the vehicle, but an abnormality in the vehicle is not detected. When the event occurs, the recording control unit records restricted audio data, in which a predetermined audio is restricted from the audio data for a first predetermined period corresponding to the occurrence time of the event, in the recording unit together with the image data for the first predetermined period.

A recording method according to one aspect of the present invention includes an acquisition step, an event detection step, and a recording control step. The acquisition step is a step of acquiring image data of an area including at least the periphery of a vehicle, and audio data of audio of an area including at least the periphery of the vehicle. The event detection step is a step of detecting the occurrence of an event in which an abnormality is detected around the vehicle, but an abnormality in the vehicle is not detected. The recording control step is a step of recording, when the event occurs, restricted audio data in which a predetermined audio is restricted from the audio data for a first predetermined period corresponding to the occurrence time of the event, together with the image data for the first predetermined period in a recording unit.

The present invention provides a recording system and recording method that optimally protects the privacy of witnesses when an abnormality occurs that is not related to the vehicle.

FIG. 1 is a diagram illustrating the configuration of a recording apparatus according to a first embodiment. 5 is a diagram showing an example of a relationship between data recorded in a recording section and data recorded in a buffer memory according to the first embodiment; FIG. 5 is a diagram showing an example of a relationship between data recorded in a recording section and data recorded in a buffer memory according to the first embodiment; FIG. FIG. 11 is a diagram illustrating an example of the flow of an event detection process according to the first embodiment. FIG. 4 is a diagram showing a flow of a recording process of media data according to the first embodiment. 13 is a diagram showing an example of a relationship between data recorded in a recording section and data recorded in a buffer memory according to the second embodiment. FIG. FIG. 11 is a configuration diagram of a recording apparatus according to a third embodiment. FIG. 11 is a diagram showing a flow of a recording process of media data according to a third embodiment. FIG. 13 is a configuration diagram of a recording apparatus according to a fourth embodiment.

The present invention will be described below through embodiments of the invention, but the invention according to the claims is not limited to the following embodiments. Furthermore, not all of the configurations described in the embodiments are necessarily essential as means for solving the problems. For clarity of explanation, the following description and drawings have been omitted and simplified as appropriate. Note that the same reference numerals are used for the same elements in each drawing.

<Embodiment 1>
Hereinafter, an embodiment of the present invention will be described with reference to the drawings. A configuration example of a recording system (hereinafter, referred to as a recording device) 10 according to the first embodiment will be described with reference to FIG. 1. FIG. 1 is a configuration diagram of the recording device 10 according to the first embodiment. The recording device 10 may be used, for example, as a drive recorder mounted on a vehicle. The recording device 10 may be a computer device that operates by a processor executing a program stored in a memory. The processor may be, for example, a microprocessor, an MPU (Micro Processing Unit), or a CPU (Central Processing Unit). The memory may be a volatile memory or a non-volatile memory, or may be configured by a combination of a volatile memory and a non-volatile memory. The processor executes one or more programs including a group of instructions for causing a computer to perform an algorithm described with reference to the following drawings.

The recording device 10 has a control unit 20, a camera 31, a microphone 32, a recording unit 33, a sensor 34, a display unit 35, and a speaker 36. The control unit 20, the camera 31, the microphone 32, the recording unit 33, the sensor 34, the display unit 35, and the speaker 36 may be directly connected to each other using a predetermined cable or the like, or may be connected to each other via a CAN (Controller Area Network), which is an in-vehicle network. The recording device 10 may also be configured to include components realized by the control unit 20. Specifically, the recording device 10 mainly includes an acquisition unit 21, an event detection unit 22, and a recording control unit 23.

The camera 31 generates image data that captures at least an area around the vehicle. In the first embodiment, the camera 31 generates image data that captures the outside of the vehicle, but may generate image data that captures the inside of the vehicle. The microphone 32 generates audio data that captures audio from an area that captures at least the area around the vehicle. In the first embodiment, the microphone 32 captures audio from inside and outside the vehicle to generate audio data. The recording unit 33 may be a hard disk, a card-type memory, etc., and records the image data and audio data.

The sensor 34 may be, for example, an acceleration sensor, and may detect impacts and the like that occur when a vehicle equipped with the recording device 10 has an accident or is involved in an accident. Furthermore, the sensor 34 may detect the amount of operation of the brake, accelerator, or steering wheel, and generate vehicle speed and acceleration information. The sensor 34 may also be, for example, a distance sensor or millimeter wave radar, and may detect the proximity of other vehicles or other objects. The display unit 35 may be a display that plays back image data recorded in the recording unit 33. The speaker 36 plays back audio data recorded in the recording unit 33. The speaker 36 includes an audio amplifier unit and audio processor (not shown) before the input to the speaker 36, and these are also included when the term speaker 36 is written.

Next, an example of the configuration of the control unit 20 will be described. The control unit 20 has an acquisition unit 21, an event detection unit 22, a recording control unit 23, a buffer memory 24, and a playback control unit 25. Each of the components constituting the control unit 20, such as the acquisition unit 21, the event detection unit 22, the recording control unit 23, and the playback control unit 25, may be software or a module in which processing is performed by a processor executing a program stored in a memory. Alternatively, each of the components constituting the control unit 20 may be hardware, such as a circuit or a chip.

The acquisition unit 21 acquires image data generated by the camera 31 and audio data generated by the microphone 32. Data including image data and audio data may be referred to as media data. In other words, the acquisition unit 21 may be said to acquire media data from the camera 31 and the microphone 32. The image data is still image data or video data. The image data may be generated using a method such as H.264 or H.265. The audio data may be generated using a method such as PCM (Pulse Code Modulation), and the media data, which is video data including image data and audio data, may be generated using MPEG (Moving Picture Experts Group) 2-TS (Transport Stream) or AVI (Audio Video Interleave), etc.

The acquisition unit 21 outputs media data including image data and audio data to the buffer memory 24. The buffer memory 24 temporarily stores the output media data. The buffer memory 24 may be referred to as an internal memory built into the control unit 20. Temporarily storing may mean, for example, that the buffer memory 24 erases media data after a predetermined period of time has elapsed. Alternatively, temporarily storing may mean that the data output from the acquisition unit 21 is overwritten onto the data in the buffer memory 24 and stored. Alternatively, temporarily storing may mean that when the buffer memory 24 outputs the media data output from the acquisition unit 21 to the recording control unit 23, it erases the output media data.

The acquisition unit 21 may also receive vehicle position information from a positioning system (not shown) such as the Global Navigation Satellite System (GNSS), and supply the position information to the buffer memory 24 together with the media data.

The event detection unit 22 detects the occurrence of an event using at least one of the image data acquired from the camera 31 via the buffer memory 24, the audio data acquired from the microphone 32, and the measurement information output from the sensor 34. Specifically, the event detection unit 22 has a surrounding abnormality detection unit 220, a vehicle abnormality detection unit 221, and a judgment unit 223.

The surrounding abnormality detection unit 220 detects the occurrence of an abnormality around the vehicle from the media data. Specifically, the surrounding abnormality detection unit 220 detects the occurrence of an abnormality around the vehicle from the voice data. An abnormality around the vehicle may include an abnormality inside the vehicle in addition to an abnormality outside the vehicle. For example, the surrounding abnormality detection unit 220 analyzes the voice data and detects a sound related to an abnormality around the vehicle (called an abnormal sound). The abnormal sound may include a non-verbal sound and a linguistic sound (utterance). The non-verbal sound may be, for example, a collision sound or a braking sound. The utterance may be, for example, a scream or an utterance such as "There's an accident" or "He's tailgating."

As an example, the surrounding abnormality detection unit 220 extracts features of the audio signal from the waveform of the audio signal, specifically, from the frequency, the elapsed time from the start of vocalization, or the volume, etc. The surrounding abnormality detection unit 220 detects the occurrence of an abnormality around the vehicle when the similarity between the extracted features and the features of sample data of a pre-registered abnormal sound is equal to or greater than a predetermined threshold. Existing acoustic analysis technology or voiceprint analysis technology can be used to calculate the features.

As another example, the surrounding abnormality detection unit 220 uses an acoustic model trained using training data including normal sound data and abnormal sound data to determine whether the voice data includes an abnormal sound. The acoustic model may be an existing acoustic model. For example, the acoustic model outputs a value indicating whether the voice data corresponding to the input includes an abnormal sound. Alternatively, the acoustic model outputs the probability that the voice data corresponding to the input includes an abnormal sound. The surrounding abnormality detection unit 220 detects the occurrence of an abnormality around the vehicle when the probability that the voice data corresponding to the input includes an abnormal sound is equal to or greater than a predetermined value. The surrounding abnormality detection unit 220 may use features obtained from the waveform of the voice signal as input for the acoustic model.

When a specific speech content, such as a scream or an utterance such as "There's an accident" or "He's tailgating" is defined as an abnormal sound, the surrounding abnormality detection unit 220 may further estimate the speech content by natural language processing and determine whether it is an abnormal sound. Specifically, the surrounding abnormality detection unit 220 first extracts phonemes from the voice data based on a predetermined acoustic model. The surrounding abnormality detection unit 220 then determines whether a speech component is included by comparing the extracted phonemes with a predetermined language model. If a speech component is included, the surrounding abnormality detection unit 220 estimates the speech content using a predetermined language model and determines whether it includes content defined as an abnormal sound. The language model includes a trained voice recognition dictionary. If the surrounding abnormality detection unit 220 determines that the estimated speech content includes an abnormal sound, it detects the occurrence of an abnormality around the vehicle.

In addition to or instead of the voice data, the surrounding abnormality detection unit 220 may analyze image data to detect abnormalities around the vehicle, such as collisions between other vehicles or between other vehicles and other objects, rollovers or fires of other vehicles, and abnormal proximity between other vehicles, such as tailgating. In this case, the surrounding abnormality detection unit 220 may detect abnormalities around the vehicle using an image recognition model that has been trained using image data from normal times, when other vehicles are abnormal, and when the vehicle itself is abnormal, as training data. For example, the image recognition model may have a convolutional neural network that receives image data as input and outputs a value indicating whether the surroundings of the vehicle are normal or abnormal. In addition to or instead of analyzing image data, the surrounding abnormality detection unit 220 may detect the occurrence of an abnormality around the vehicle by detecting the proximity of other vehicles or between other vehicles and other objects using a sensor 34, such as a distance sensor or millimeter wave radar.

When the surrounding abnormality detection unit 220 detects the occurrence of an abnormality around the vehicle, it supplies information indicating that an abnormality has occurred around the vehicle (surrounding abnormality information) to the determination unit 223. The surrounding abnormality information includes the time when the abnormality around the vehicle occurred. Furthermore, the surrounding abnormality information may also include the content of the abnormality that has occurred.

The vehicle abnormality detection unit 221 detects the occurrence of an abnormality in the host vehicle, mainly an abnormality related to the running of the host vehicle, from the measurement information output from the sensor 34. For example, when the impact level detected by the sensor 34 operating as an acceleration sensor exceeds a predetermined value, the vehicle abnormality detection unit 221 detects that an abnormality related to the host vehicle has occurred. The impact level exceeding a predetermined value may be rephrased as the acceleration exceeding a predetermined value. Also, for example, when the distance between the host vehicle and another vehicle detected by the sensor 34 operating as a distance sensor is less than a predetermined value, the vehicle abnormality detection unit 221 detects that an abnormality related to the host vehicle has occurred. When the vehicle abnormality detection unit 221 detects the occurrence of an abnormality in the host vehicle, it supplies information indicating that an abnormality in the host vehicle has occurred (host vehicle abnormality information) to the determination unit 223. The host vehicle abnormality information includes the time when the abnormality in the host vehicle has occurred. Furthermore, the host vehicle abnormality information may include the content of the abnormality that has occurred. The vehicle abnormality detection unit 221 may analyze image data in addition to or instead of the measurement information output from the sensor 34, and detect the occurrence of an abnormality in the host vehicle, such as abnormal proximity between the host vehicle and another vehicle.

The determination unit 223 determines whether an event has occurred based on the detection result of the surrounding abnormality detection unit 220 and the detection result of the vehicle abnormality detection unit 221. In this embodiment 1, a case where an abnormality is detected around the vehicle and an abnormality is not detected in the vehicle (host vehicle) is called a first event. Examples of the first event include a collision between other vehicles in the vicinity of the vehicle and a tailgating of another vehicle in the vicinity of the vehicle. The determination unit 223 determines that a first event has occurred when it acquires information indicating that an abnormality has occurred around the vehicle and does not acquire information indicating that an abnormality has occurred in the host vehicle. In addition to the first event, the determination unit 223 may detect a second event in which an abnormality in the vehicle is detected.

In response to detecting the occurrence of an event, the determination unit 223 supplies information indicating that an event has occurred (event occurrence information) to the recording control unit 23. The information indicating that an event has occurred includes the time at which the event has occurred.

The recording control unit 23 records the media data stored in the buffer memory 24 in the recording unit 33 in a predetermined file format. The media data acquired from the buffer memory 24 includes image data and audio data. In other words, the recording control unit 23 transfers the media data from the buffer memory 24 to the recording unit 33. Specifically, the recording control unit 23 generates a video file including image data and audio data from the media data stored in the buffer memory 24, and records the video file in the recording unit 33 as media data for recording.

When the occurrence of the first event is detected, the recording control unit 23 generates restricted audio data from the audio data of the occurrence period of the first event stored in the buffer memory 24. Then, the recording control unit 23 records the restricted audio data in the recording unit 33 together with image data of the occurrence period of the first event. The occurrence period of the first event may be a predetermined period corresponding to the time when the first event occurs. For example, the occurrence period of the first event may be a predetermined period (first predetermined period) including the time when the first event occurs and the period before and after the time when the first event occurs. As an example, in the occurrence period of the first event, the period before the occurrence time of the first event may be the same as the period after the occurrence time of the first event, or may be longer or shorter than the period after the occurrence time of the first event. The occurrence period of the first event may only include either the period before or after the time when the first event occurs. In addition, the restricted audio data may be silent data, or may be audio data with the overall volume reduced. Specifically, when the occurrence of a first event is detected, the recording control unit 23 generates restricted audio data by restricting the audio contained in the audio data from the audio data during the occurrence of the first event stored in the buffer memory 24. The recording control unit 23 then generates a video file containing the restricted audio data in place of the audio data during the occurrence of the first event and image data during the occurrence of the first event, and records the video file in the recording unit 33 as media data for recording.

When the recording control unit 23 records the generated media data for recording in the recording unit 33, the recording control unit 23 may record the media data for recording caused by the first event, i.e., the media data for recording including image data and restricted audio data, in the recording unit 33 as a record related to the event, as data that cannot be overwritten. For example, the recording control unit 23 may specify an address of a memory area in the recording unit 33 that prohibits overwriting of data, and record the media data for recording in the recording unit 33. Alternatively, the recording control unit 23 may set flag information indicating prohibition of overwriting in the header or payload of the media data for recording. The recording control unit 23 may record the media data for recording including image data and audio data caused by the second event in the recording unit 33 as data that cannot be overwritten, as a record related to the event.

The recording control unit 23 may discard recording media data that is not attributable to an event, that is, recording media data that includes image data and audio data, without recording it in the recording unit 33, or may record it in the recording unit 33 as a normal recording that is not related to an event. When recording it in the recording unit 33 as a normal recording, the recording media data may be recorded in the recording unit 33 as overwritable data. For example, the recording control unit 23 may specify an address of a memory area in the recording unit 33 that allows data to be overwritten, and record the recording media data in the recording unit 33. Alternatively, the recording control unit 23 may set flag information indicating overwritability in the header or payload of the recording media data.

The recording unit 33 records the data output from the recording control unit 23, in other words, the media data transferred from the buffer memory 24 via the recording control unit 23. The recording unit 33 is, for example, a recording medium such as an optical disc or a card-type recording medium.

The playback control unit 25 acquires the recording media data recorded in the recording unit 33, outputs image data contained in the acquired recording media data to the display unit 35, and outputs audio data contained in the acquired recording media data to the speaker 36, etc.

1 shows a configuration in which the recording device 10 includes a playback control unit 25, a display unit 35, and a speaker 36, but a playback device or the like different from the recording device 10 may include the playback control unit 25, the display unit 35, and the speaker 36. In other words, the recording device 10 does not need to include the playback control unit 25, the display unit 35, and the speaker 36. That is, a playback device or the like including the playback control unit 25, the display unit 35, and the speaker 36 may play back data recorded in the recording unit 33 of the recording device 10. For example, the playback device or the like may be a device that plays back data recorded on a portable recording medium such as an optical disc or a card-type recording medium. The data recorded in the recording unit 33 may be transferred to the portable recording medium.

Here, the relationship between the data recorded in the recording unit 33 and the data recorded in the buffer memory 24 will be described with reference to FIG. 2. FIG. 2 is a diagram showing an example of the relationship between the data recorded in the recording unit 33 according to the first embodiment and the data recorded in the buffer memory 24. The horizontal axis shown in FIG. 2 indicates the passage of time. Each of D-3 to D+4 indicates media data divided into predetermined periods that were taken during the period in which the first event occurred. The predetermined period may be called a time frame, and may be, for example, several seconds, tens of seconds, or several minutes. In FIG. 2, D-2 is the oldest media data taken, and D+4 is the most recently taken media data. Note that the middle part of FIG. 2 shows data that has been newly added to the buffer memory 24 during the predetermined period. In other words, the buffer memory 24 may store data other than the data newly added to the buffer memory 24 in the predetermined time frame, for example, unerased data added before the time frame.

The media data acquired by the acquisition unit 21 is immediately output to the buffer memory 24. Therefore, FIG. 2 shows that the media data acquired by the acquisition unit 21 is stored in the buffer memory 24 as it is. In other words, FIG. 2 shows that the timing at which the acquisition unit 21 acquires the media data and the timing at which the media data is stored in the buffer memory 24 are substantially the same. Substantially the same also includes cases where the timing at which the acquisition unit 21 acquires the media data and the timing at which the media data is stored in the buffer memory 24 do not completely match. For example, substantially the same also includes cases where the timing at which the media data is stored in the buffer memory 24 is later than the timing at which the acquisition unit 21 acquires the media data. On the other hand, the media data recorded in the recording unit 33 is temporarily stored in the buffer memory 24 and then recorded in the recording unit 33 by the recording control unit 23. Therefore, the timing at which the media data acquired by the acquisition unit 21 is stored in the buffer memory 24 is different from the timing at which the media data is recorded in the recording unit 33. Specifically, the timing at which the media data is recorded in the recording unit 33 is later than the timing at which the media data is stored in the buffer memory 24.

t-3 to t+3 and ti indicate times. The event detection unit 22 determines whether an event occurred during a predetermined period in the past, from time t-3 to t+3. For example, at time t, the event detection unit 22 determines whether an event occurred between time t-1 and time t. The event detection unit 22 similarly determines whether an event occurred at other times.

For example, at time t-2, the event detection unit 22 determines that the first event has not occurred between time t-3 and time t-2. In this case, the recording control unit 23 may discard the media data D-2 stored in the buffer memory 24 between time t-2 and time t-1 without recording it in the recording unit 33, or may record it in the recording unit 33 as a normal recording that is not related to an event.

On the other hand, the event detection unit 22 determines that the first event occurred at time t, which is between time t-1 and time t. In this case, the recording control unit 23 replaces the audio data of the media data D stored in the buffer memory 24 during a predetermined period corresponding to time ti, for example, between time t and time t+1, with restricted audio data, and records the image data and the restricted audio data in the recording unit 33 as media data D' in file form.

2 shows that the media data that has been completely stored in the buffer memory 24 is recorded in the recording unit 33. That is, the timing at which the media data is recorded in the recording unit 33 is one time frame later than the timing at which the media data is stored in the buffer memory 24. However, the timing at which the media data stored in the buffer memory 24 is recorded in the recording unit 33 is not limited to this. The difference between the two timings may be less than one time frame or more than two time frames. For example, the recording control unit 23 may acquire the media data that is being stored in the buffer memory 24 and record the acquired media data in the recording unit 33. Alternatively, when the storage of the media data in the buffer memory 24 is completed, the recording control unit 23 may record the media data in the recording unit 33 after a predetermined period has elapsed from the point at which the storage of the media data is completed, regardless of the time frame.

2 shows that the recording control unit 23 records only the media data acquired by the acquisition unit 21 during the period including the first event occurrence time in the recording unit 33 with audio restriction. In other words, the shooting period (event occurrence period) of the media data subject to audio restriction is limited to the time frame including the first event occurrence time. However, the recording control unit 23 is not limited to this, and may also record media data acquired by the acquisition unit 21 during the period before or after the period including the first event occurrence time in the recording unit 33 with audio restriction, in addition to the media data acquired by the acquisition unit 21 during the period including the first event occurrence time. In other words, the shooting period of the media data subject to audio restriction may include time frames before or after the time frame in addition to the time frame including the first event occurrence time, and the period of the time frame including the first event occurrence time may be set arbitrarily. For example, in addition to the data D acquired by the acquisition unit 21, the recording control unit 23 may also record data D-1 in the recording unit 33 as data D' and data D-1' with audio restriction.

The relationship between the data recorded in the recording unit 33 and the data recorded in the buffer memory 24 in this case is shown in FIG. 3. FIG. 3 is a diagram showing another example of the relationship between the data recorded in the recording unit 33 and the data recorded in the buffer memory 24 according to the first embodiment. When a first event is detected in either the time frame corresponding to the shooting period of each media data in the buffer memory 24 or the next time frame, the recording control unit 23 restricts the audio of the media data and records it in the recording unit 33.

Specifically, at time t-2, the event detection unit 22 determines that the first event did not occur between time t-3 and time t-2, and at time t-1, determines that the first event did not occur between time t-2 and time t-1. At this time, the recording control unit 23 converts the media data D-2 stored in the buffer memory 24 between time t-1 and time t into a file without restricting the audio data, and records it in the recording unit 33.

Then, at time t, the event detection unit 22 determines that the first event occurred at time ti, which is between time t-1 and time t. At this time, even if the first event did not occur in the previous time frame, the recording control unit 23 replaces the audio data of the media data D-1 stored in the buffer memory 24 during a predetermined period corresponding to time ti, for example, between time t and time t+1, with restricted audio data, and records this together with the image data in the recording unit 33 as filed data D-1'.

Next, the recording control unit 23 replaces the audio data of the media data D stored in the buffer memory 24 with restricted audio data, and records the data together with the image data in the recording unit 33 as filed data D'.

Then, at time t+1, the event detection unit 22 determines that the first event has not occurred between time t and time t+1. At this time, the recording control unit 23 may directly convert the media data D+1 stored in the buffer memory 24 into a file and record it in the recording unit 33 without restricting the audio data.

Next, the processing of the control unit 20 of the recording device 10 will be described with reference to Figures 4 and 5. Figures 4 and 5 show an example in which the recording period of the media data subject to audio restriction is limited to a time frame including the time when an event occurs, as in Figure 2, but this is not limiting.

Figure 4 is a diagram showing the flow of the event detection process according to the first embodiment. First, the acquisition unit 21 acquires media data including image data and audio data and outputs it to the buffer memory 24, and the buffer memory 24 stores the media data (step S10). Next, the surrounding abnormality detection unit 220 of the event detection unit 22 determines whether or not an abnormal sound in the surroundings has been detected from the audio data included in the media data stored in the buffer memory 24 (step S11). If the surrounding abnormality detection unit 220 detects an abnormal sound in the surroundings from the audio data (Yes in step S11), it supplies the surrounding abnormality information to the determination unit 223 and proceeds to the process in step S12. On the other hand, if the surrounding abnormality detection unit 220 does not detect an abnormal sound in the surroundings from the audio data (No in step S11), it returns the process to step S10.

In step S12, the vehicle abnormality detection unit 221 of the event detection unit 22 determines whether or not an abnormality has been detected in the host vehicle from the measurement information output from the sensor 34. If an abnormality has been detected in the host vehicle (Yes in step S12), the vehicle abnormality detection unit 221 supplies the host vehicle abnormality information to the determination unit 223 and returns the process to step S10. On the other hand, if an abnormality has not been detected in the host vehicle (No in step S12), the vehicle abnormality detection unit 221 does not supply the host vehicle abnormality information to the determination unit 223 and proceeds to step S13.

In step S13, the determination unit 223 of the event detection unit 22 determines that a first event has occurred in response to the fact that the surrounding abnormality information has been acquired but the subject vehicle abnormality information has not been acquired. The determination unit 223 then generates event occurrence information and supplies the event occurrence information to the recording control unit 23 (step S14). After that, the process returns to step S10, and the processes from step S10 onwards are repeatedly executed.

Next, FIG. 5 is a diagram showing the flow of the media data recording process according to the first embodiment. First, the acquisition unit 21 acquires media data including image data and audio data and outputs it to the buffer memory 24, and the buffer memory 24 stores the media data (step S20). Next, the recording control unit 23 determines whether or not a first event has occurred before starting to record the media data stored in the buffer memory 24 to the recording unit 33 (step S21). That is, the recording control unit 23 determines whether or not event occurrence information has been acquired from the determination unit 223 of the event detection unit 22.

When the recording control unit 23 determines that the first event has occurred (Yes in step S21), it extracts audio data from the media data including the timing when the first event occurred during the shooting period, and generates restricted audio data by restricting the audio (step S22). The recording control unit 23 then generates a video file including the image data and the restricted audio data, and records it in the recording unit 33 (step S23). After that, it returns to step S20, and repeats the processes from step S20 onwards.

On the other hand, if the recording control unit 23 determines that the first event has not occurred (No in step S21), it directly files the media data captured during the period when the first event has not occurred and records it in the recording unit 33 (step S24). That is, the recording control unit 23 files the media data stored in the buffer memory 24 without restricting the sound and records it in the recording unit 33. If the recording control unit 23 determines that the first event has not occurred, it may be configured not to record in the recording unit 33. Thereafter, it returns to step S20 and repeatedly executes the processes from step S20 onwards.

In the above description, it has been described that when the occurrence of the first event is detected, the recording control unit 23 records the media data with audio restriction in the recording unit 33. However, in addition to this, when the occurrence of the first event is detected, the recording control unit 23 may also record the media data with no audio restriction in the recording unit 33. For example, the recording unit 33 has a first recording area and a second recording area, and the media data stored in the buffer memory 24 is recorded as is in the first recording area as a normal recording that is not related to the event detection result. Meanwhile, in the second recording area, data with audio restriction from the media data whose shooting period includes the timing when the first event occurred may be recorded as an event record related to the detection of the first event.

As described above, the recording device 10 according to the first embodiment can replace the audio data of media data whose shooting period includes the timing when the first event occurred with restricted audio data that restricts the audio and record the media data in the recording unit 33. In this way, when an abnormality unrelated to the vehicle occurs, the recording device 10 does not record audio that may include the speech of the driver of the vehicle or conversations between occupants, etc., so that the privacy of a witness can be appropriately protected when providing information as a witness. Note that the recording device 10 can record image data and audio data in the recording unit 33 for media data shot during a period when the first event does not occur. Therefore, when an abnormality related to the vehicle occurs, the recorded media data can be used to investigate the cause of the accident, etc.

Furthermore, when the recording device 10 erases the audio data and records it in the recording unit 33 when the first event occurs, the amount of data to be recorded can be reduced compared to when recording media data including audio data. This reduces the number of times that the recording unit 33 overwrites and records media data, allowing the recording unit 33 to record media data for a longer period of time.

<Embodiment 2>
Next, a second embodiment will be described. In the second embodiment, when a first event occurs, the recording control unit 23 stores media data for a period during which the first event occurs in the recording unit 33, distinguishing it from cases in which the event does not occur. However, the recording control unit 23 restricts only audio for a specific period from the media data stored distinguishing it from cases in which the event does not occur.

For example, the specific period during which the sound is restricted and recorded may be a specific period (second specific period) before the occurrence of the first event during the first event occurrence period (first specific period). During and immediately after the accident, there is a high possibility that sound that may be important evidence of the accident will be picked up, so it is preferable to record the sound without restricting it. Therefore, when the first event occurs, the recording control unit 23 may generate restricted sound data in which the sound is restricted from the sound data of the specific period before the occurrence of the first event. Then, the recording control unit 23 may record the restricted sound data together with the image data of the specific period (second specific period) before the occurrence of the first event in a specific storage area of the recording unit 33. Then, the recording control unit 23 may record the sound data of the specific period (third specific period) after the occurrence of the first event as it is, together with the image data of the specific period (third specific period) after the occurrence of the first event in a specific storage area of the recording unit 33. The second predetermined period may include a period before the first predetermined period, and may not include a portion of the period before the occurrence of the first event that is included in the first predetermined period. The second predetermined period is not strictly limited to a form that does not include a portion during the occurrence of the accident and immediately after the occurrence, and may include a portion during the occurrence of the accident and immediately after the occurrence. Similarly, the third predetermined period may include a period after the first predetermined period, and may not include a portion of the period after the occurrence of the first event that is included in the first predetermined period. The third predetermined period is not necessarily limited to a form that includes a portion during the occurrence of the accident and immediately after the occurrence, and may include a form that does not include a portion during the occurrence of the accident and immediately after the occurrence.

Figure 6 is a diagram showing an example of the relationship between data recorded in the recording unit 33 and data recorded in the buffer memory 24 according to the second embodiment. In this example, the recording unit 33 has a first recording area which is a normal recording area not related to the first event detection result, and a second storage area which is an event recording area related to the first event detection result. The timing at which the media data is recorded in the recording unit 33 is two time frames later than the timing at which the media data is stored in the buffer memory 24.

Regardless of the result of the first event detection, the recording control unit 23 records the media data stored in the buffer memory 24 in the first recording area of the recording unit 33.

When the occurrence of a first event is detected for each media data in the buffer memory 24, the recording control unit 23 restricts the audio for a specific period of the media data and records it in the second storage area of the recording unit 33.

Specifically, the event detection unit 22 determines that the first event occurred between time t-1 and time t. At this time, the recording control unit 23 replaces the audio data of the media data D-1 stored in the buffer memory 24, which corresponds to the time frame immediately preceding time t-1 to time t when the first event occurred, from time t-2 to time t-1, with restricted audio data, and records this together with the image data in the second storage area of the recording unit 33. Alternatively or in addition to the above, the recording control unit 23 may replace the audio data of the media data corresponding to time t-1 to time t1 when the first event occurred with restricted audio data, and record this together with the image data in the second storage area of the recording unit 33.

At this time, the recording control unit 23 also records the audio data of the media data D stored in the buffer memory 24, which corresponds to the period from time t-1 to time t when the first event occurred, together with the image data, in the second storage area of the recording unit 33 as is. Alternatively or in addition to the above, the recording control unit 23 may replace the audio data of the media data corresponding to the period from time t-1 to time t when the first event occurred with restricted audio data, and record this together with the image data in the second storage area of the recording unit 33.

At this time, the recording control unit 23 also records the audio data of media data D+1, which is stored in the buffer memory 24 and corresponds to the time frame from time t to time t+1, which is one time frame after time t-1 to time t when the first event occurred, together with the image data in the second storage area of the recording unit 33.

In addition to or instead of the above-mentioned recording area, the recording unit 33 may have a recording area for recording only the audio data during the first event occurrence period, and a recording area for recording only the image data during the first event occurrence period. The recording device 10 may then receive from the user a specification of a period during which audio is restricted, for example after driving, generate media data in which audio is restricted for the specified period, and record the data in the recording unit 33.

In addition to or instead of the above-mentioned recording area, the recording unit 33 may have a recording area for recording audio-restricted media data during the first event occurrence period, and a recording area for recording audio-unrestricted media data during the first event occurrence period. The recording device 10 may then accept a designation from the user of a period during which audio is restricted, for example after driving, and record a combination of audio-restricted media data and audio-unrestricted media data for the designated period in the recording unit 33.

In this way, the recording device 10 of embodiment 2 can achieve the same effect as embodiment 1 while recording the audio inside the vehicle cabin at the time the first event occurs and after the first event occurs, and can reliably record the audio necessary to provide information while protecting the privacy of witnesses.

<Embodiment 3>
Next, a third embodiment will be described. The third embodiment is characterized in that, when a first event occurs, the recording control unit 23 stores in the recording unit 33 media data in which only a predetermined sound is restricted during the period in which the first event occurs. The predetermined sound to be restricted is, for example, the voice of an occupant in the vehicle cabin. In the following description, the voice to be restricted is the voice of an occupant in the vehicle cabin, but is not limited thereto.

Figure 7 is a block diagram of a recording device 10a according to embodiment 3. Recording device 10a differs from recording device 10 in that it includes a control unit 20a instead of control unit 20. Control unit 20a also differs from control unit 20 in that it includes a restricted audio detection unit 26 in addition to the components of control unit 20.

When the occurrence of the first event is detected, the restricted voice detection unit 26 functions as an occupant voice detection unit that detects the voice of an occupant in the vehicle cabin from the voice data included in the media data stored in the buffer memory 24. For example, the restricted voice detection unit 26 may perform a voiceprint analysis of the voice data during the occurrence period of the first event and determine whether or not the voice data includes the voice of a pre-registered occupant. The restricted voice detection unit 26 may also detect the voice from the direction in which the occupant sits as the voice of the occupant by beamforming processing. In this case, the recording device 10 may have multiple microphones 32. When the restricted voice detection unit 26 determines that the voice data includes the voice of a pre-registered occupant, it supplies information indicating the voice of the occupant to the recording control unit 23. The restricted voice detection unit 26 may also detect the voice of the occupant in the vehicle cabin by comparing the voice of a microphone installed inside the vehicle cabin with the voice of a microphone installed outside the vehicle cabin or by performing a cancellation process.

When the occurrence of the first event is detected, the recording control unit 23 generates restricted voice data from the voice data during the occurrence of the first event based on information indicating the occupant's vocalization. The restricted voice data may be data in which the voice signal corresponding to the occupant's vocalization is erased from the original voice signal, or may be data in which only the voice signal corresponding to the occupant's vocalization is attenuated. The recording control unit 23 records the restricted voice data in the recording unit 33 together with image data during the occurrence of the first event.

Figure 8 is a diagram showing the flow of media data recording processing in embodiment 3. In this figure, steps that are the same as those in Figure 5 are given the same symbols.

In response to receiving event occurrence information from the determination unit 223 (Yes in step S21), the restricted voice detection unit 26 determines whether or not a passenger's voice is detected from the voice data (step S30).

If the restricted audio detection unit 26 detects the occupant's vocalization (Yes in step S30), it supplies information indicating the occupant's vocalization to the recording control unit 23 and proceeds to step S31. In step S31, in response to acquiring the information indicating the occupant's vocalization, the recording control unit 23 extracts audio data from the media data including the timing when the first event occurred during the shooting period, and generates restricted audio data from the audio data by restricting the audio corresponding to the occupant's vocalization. The recording control unit 23 then generates a video file including the image data and the restricted audio data, and records it in the recording unit 33 (step S32).

On the other hand, if the restricted voice detection unit 26 does not detect the occupant's voice (No in step S30), the process proceeds to step S33. In step S33, the recording control unit 23 directly files the media data including the timing when the first event occurred during the shooting period and records it in the recording unit 33. However, the media data filed in this case may be recorded separately from the media data filed when the occurrence of the first event is not detected.

The restricted audio detection unit 26 may be configured integrally with the surrounding abnormality detection unit 220.

In this way, when a first event occurs, the recording device 10a according to the third embodiment records media data in which only a specific sound is restricted during the period in which the first event occurs. Therefore, for example, when an accident occurs, only private conversations in the vehicle cabin can be erased from the sound, while the impact sound of the accident and the screams and chatter of people in the vicinity can be recorded. This makes it possible to reliably record the sound necessary for providing information while protecting the privacy of witnesses.

In the above description, when the first event occurs, the voice data corresponding to the voice of the passenger in the vehicle cabin is uniformly restricted. However, this is not limited to the above, and when the first event occurs, the voice data corresponding to the voice of the passenger in the vehicle cabin related to the abnormality related to the first event may not be subject to restriction. In other words, the restricted voice detection unit 26 may detect, from the voice data included in the media data stored in the buffer memory 24, the voice data of the passenger in the vehicle cabin other than the voice related to the abnormality. In this case, the restricted voice detection unit 26 may use a predetermined language model to determine whether the detected voice of the passenger in the vehicle cabin is related to a predetermined abnormality, such as a scream, "It's an accident," or "They're tailgating." This makes it possible to more appropriately protect the privacy of witnesses and more reliably record voice necessary for providing other information.

<Embodiment 4>
Next, a fourth embodiment will be described. The fourth embodiment is characterized in that, when a first event occurs, the recording control unit 23 stores in the recording unit 33 media data in which sounds other than sounds corresponding to the type of abnormality are restricted during the period in which the first event occurs.

Figure 9 is a configuration diagram of a recording device 10b according to the fourth embodiment. The recording device 10b differs from the recording device 10a in that it includes a control unit 20b instead of the control unit 20a. The control unit 20b also differs from the control unit 20a in that it includes an event detection unit 22b instead of the event detection unit 22.

Event detection unit 22b differs from event detection unit 22 in that it includes surrounding anomaly detection unit 220b instead of surrounding anomaly detection unit 220.

The surrounding anomaly detection unit 220b basically has the same functions as the surrounding anomaly detection unit 220, but includes an anomaly type determination unit 225.

The abnormality type determination unit 225 determines the type of abnormality (abnormality type) around the vehicle from the voice data in which an abnormality around the vehicle is detected. Here, the abnormality types may be classified into, for example, "accident," "near miss," and "tailgating." For example, if the voice data contains an impact sound such as "crash" or an utterance of "accident," the abnormality type determination unit 225 determines the abnormality type as "accident." Also, if the voice data contains a conversation such as "there is a car tailgating," the abnormality type determination unit 225 determines the abnormality type as "tailgating." Then, the abnormality type determination unit 225 supplies information indicating the abnormality type to the determination unit 223.

When the determination unit 223 determines that a first event has occurred, it adds information indicating the type of abnormality to the event occurrence information and supplies the event occurrence information to the restricted audio detection unit 26.

When the restricted audio detection unit 26 acquires event occurrence information, it detects audio corresponding to the abnormality type from the audio data as audio to be restricted. For example, when the abnormality type is "accident", the restricted audio detection unit 26 detects conversational audio within the vehicle cabin such as "accident" or "crash" from the audio data. When the abnormality type is "tailgating", the restricted audio detection unit 26 detects conversational audio within the vehicle cabin such as "tailgating", "dangerous", and "nearby" from the audio data. The restricted audio detection unit 26 then determines audio other than the detected audio, that is, audio not related to the abnormality type, as audio to be restricted, and supplies information indicating the audio to be restricted to the recording control unit 23.

When a first event occurs, the recording control unit 23 generates restricted audio data by restricting audio corresponding to the type of abnormality from the audio data during the first event occurrence period based on information indicating the audio to be restricted. In other words, when a first event occurs, the recording control unit 23 generates restricted audio data from the audio data during the first event occurrence period in which audio related to the type of abnormality is not restricted and audio unrelated to the type of abnormality is restricted. The recording control unit 23 then records in the recording unit 33 the media data in which audio corresponding to the type of abnormality is restricted from the audio during the event occurrence period.

In this way, according to the recording device 10b of embodiment 4, when an abnormality occurs around the vehicle, audio unrelated to the abnormality is restricted to more appropriately protect the privacy of witnesses, while audio related to the abnormality is not restricted, so that audio necessary for providing information can be more reliably recorded.

The present invention is not limited to the above embodiment, and can be modified as appropriate without departing from the spirit of the present invention. For example, the surrounding abnormality detection process by the surrounding abnormality detection units 220 and 220b, the vehicle abnormality detection process by the vehicle abnormality detection unit 221, and the abnormality type determination process by the abnormality type determination unit 225 may be performed by an external device (not shown) instead of the recording device. In other words, the surrounding abnormality detection units 220 and 220b, the vehicle abnormality detection unit 221, and the abnormality type determination unit 225 may not be included in the recording device. In this case, the recording device acquires various detection results or determination results from the external device via communication means (not shown) and supplies them to the determination unit 223. Similarly, the restricted audio detection unit 26 may not be included in the recording device. The recording device acquires information indicating the audio to be restricted from the external device via communication means (not shown) and supplies it to the recording control unit 23.

In the first to fourth embodiments, the surrounding abnormality detection unit 220 determines whether the surroundings of the vehicle are normal or abnormal (i.e., abnormality related to at least one of the vehicle itself and the other vehicle) from at least one of the voice data and image data, and detects the occurrence of an abnormality around the vehicle. However, instead of this, the surrounding abnormality detection unit 220 may determine whether the surroundings are normal, abnormal in the vehicle itself, or abnormal in another vehicle not related to the vehicle from at least one of the voice data and image data. In this case, instead of step S12 in FIG. 4, the surrounding abnormality detection unit 220 may determine whether an abnormality in another vehicle not related to the vehicle has been detected from the media data stored in the buffer memory 24. Then, when an abnormality in the other vehicle is detected, the surrounding abnormality detection unit 220 may supply other vehicle abnormality information indicating that an abnormality in the other vehicle has been detected to the determination unit 223. Then, instead of step S13 in FIG. 4, the determination unit 223 may determine that a first event has occurred in response to acquiring the other vehicle abnormality information. That is, step S12 in FIG. 4 may be omitted.

Furthermore, some or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes.
(Appendix 1)
An acquisition unit that acquires image data of an area including at least the periphery of the vehicle and audio data of an area including at least the periphery of the vehicle;
an event detection unit that detects the occurrence of an event in which an abnormality is detected around the vehicle and an abnormality is not detected in the vehicle;
and a recording control unit that, when the event occurs, records restricted audio data in which a specified audio is restricted from the audio data for a first specified period corresponding to the time when the event occurs, together with the image data for the first specified period in a recording unit.
(Appendix 2)
The event detection unit
a surrounding abnormality detection unit that detects an abnormality around the vehicle from the voice data;
and a vehicle abnormality detection unit that detects an abnormality in the vehicle from measurement information of a sensor of the vehicle.
(Appendix 3)
When the event occurs, the recording control unit
recording, in the recording unit, restricted voice data in which a predetermined voice is restricted from the voice data for a second predetermined period including a period before the occurrence of the event, together with the image data for the second predetermined period;
The recording system according to claim 1 or 2, wherein the audio data for a third predetermined period including a period following occurrence of the event is recorded in the recording unit together with the image data for the second predetermined period.
(Appendix 4)
an occupant speech detection unit that detects speech of an occupant in a cabin of the vehicle from the voice data;
The recording control unit, when the event occurs, restricts a voice corresponding to the detected speech of the occupant from the voice data of the first predetermined period or the voice data of a second predetermined period including a period before the occurrence of the event, and generates the restricted voice data.
(Appendix 5)
The occupant utterance detection unit further includes an abnormality-related utterance detection unit that detects an utterance related to the abnormality from an utterance of the occupant,
The recording control unit generates the restricted voice data from the voice data of the first predetermined period or the voice data of a second predetermined period including a period before the occurrence of the event, without restricting voice corresponding to the occupant's speech regarding the abnormality.The recording system described in Appendix 4.
(Appendix 6)
an abnormality type determination unit that determines a type of abnormality from the voice data;
The recording system according to any one of appendixes 1 to 5, wherein when the event occurs, the recording control unit restricts audio corresponding to the type of abnormality from the audio data for the first specified period or the audio data for a second specified period including a period before the event occurs, and generates the restricted audio data.
(Appendix 7)
An acquisition step of acquiring image data of an area including at least the surroundings of the vehicle and audio data of an area including at least the surroundings of the vehicle;
an event detection step of detecting an occurrence of an event in which an abnormality is detected around the vehicle and an abnormality is not detected on the vehicle;
and a recording control step of, when the event occurs, recording restricted audio data in which a specified audio is restricted from the audio data for a first predetermined period corresponding to the time when the event occurs, together with the image data for the first predetermined period in a recording unit.

10, 10a, 10b Recording system (recording device)
20, 20a, 20b Control unit 21 Acquisition unit 22, 22b Event detection unit 23 Recording control unit 24 Buffer memory 25 Playback control unit 26 Restricted audio detection unit 31 Camera 32 Microphone 33 Recording unit 34 Sensor 35 Display unit 36 Speaker 220, 220b Surrounding abnormality detection unit 221 Vehicle abnormality detection unit 223 Determination unit 225 Abnormality type determination unit

Claims (5)

An acquisition unit that acquires image data obtained by capturing an image of an area including at least the periphery of a vehicle and audio data obtained by capturing audio of an area including at least the periphery of the vehicle;
an event detection unit that detects the occurrence of an event in which an abnormality is detected around the vehicle and an event in which an abnormality is not detected around the vehicle, and determines a type of the abnormality around the vehicle ;
and a recording control unit that, when the event occurs, records restricted voice data in which voices not related to the determined abnormality type are restricted from the voice data for a first predetermined period corresponding to the time when the event occurs, in a recording unit together with the image data for the first predetermined period. An acquisition unit that acquires image data obtained by capturing an image of an area including at least the periphery of a vehicle and audio data obtained by capturing audio of an area including at least the periphery of the vehicle;
an event detection unit that detects the occurrence of an event in which an abnormality is detected around the vehicle and an abnormality is not detected in the vehicle;
a restricted voice detection unit that detects, when the event is detected, an utterance of an occupant of the vehicle by a voiceprint analysis of the voice data or by detecting a voice coming from a direction in which the occupant of the vehicle is seated by a beamforming process;
a recording control unit that, when the event occurs, records in a recording unit, restricted voice data obtained by erasing a voice signal corresponding to the voice of the occupant from the voice data for a first predetermined period corresponding to the occurrence time of the event, or by attenuating the voice signal corresponding to the voice of the occupant, together with the image data for the first predetermined period;
A recording system comprising: When the event occurs, the recording control unit
recording, in the recording unit, restricted voice data in which a predetermined voice is restricted from the voice data for a second predetermined period including a period before the occurrence of the event, together with the image data for the second predetermined period;
The recording system according to claim 1 , wherein the audio data for a third predetermined period including a period following the occurrence of the event is recorded in the recording section together with the image data for the third predetermined period. an occupant speech detection unit that detects speech of an occupant in a cabin of the vehicle from the voice data;
4. The recording system according to claim 1, wherein, when the event occurs, the recording control unit restricts a voice corresponding to the detected speech of the occupant from the voice data of the first predetermined period or the voice data of a second predetermined period including a period before the occurrence of the event, and generates the restricted voice data. An acquisition step of acquiring image data obtained by capturing an image of an area including at least the periphery of a vehicle, and audio data obtained by capturing audio of an area including at least the periphery of the vehicle;
an event detection step of detecting an occurrence of an event in which an abnormality is detected around the vehicle and an abnormality is not detected around the vehicle, and determining a type of the abnormality around the vehicle ;
and a recording control step of, when an event occurs in which an abnormality is detected around the vehicle and no abnormality is detected in the vehicle, recording restricted voice data in which voices unrelated to the determined type of abnormality are restricted from the voice data for a first predetermined period corresponding to the occurrence time of the event, together with the image data for the first predetermined period in a recording unit.

Images