JP7843521B2 - Playback device and playback method, as well as their programs, recording device and control method for the recording device, etc. - Google Patents

Description

This invention relates, for example, to a playback device and playback method, as well as a program therefor, and to a recording device and a control method for the recording device.

In-vehicle video recording devices are known that use cameras installed in the vehicle to capture images of the area around the vehicle and record surrounding images and vehicle speed when the vehicle is subjected to an impact due to events such as sudden braking, sudden steering, or collision. Playback devices for playing back the images captured by such drive recorders are also known (Patent Document 1).

Patent No. 5966131

However, the invention described in Patent Document 1 only involved playing back recorded video footage, such as images taken before and after an event occurred.

This invention aims, for example, to increase the likelihood of displaying images that are more suitable for the user than before. The purpose of this invention is not limited to this, and the applicant intends to obtain rights through divisional applications, amendments, etc., for configurations that aim to achieve the effects derived from the components disclosed in this specification and the drawings. For example, problems disclosed in this specification can be reinterpreted as "the problem is..." where "it is possible to..." is used. Each problem is described independently, and the applicant intends to obtain rights to the configurations that solve these problems individually through divisional applications, amendments, etc. Even if a problem is implicitly understood from the description in the specification, the applicant intends to include a portion of the configuration described in this specification in the claims through amendment or a divisional application.

(1) A playback device for playing back recorded video data may be provided, which processes the recorded video data based on acquired information and controls the device to display the video based on the processed video data.

In this way, users can view not the recorded video itself, but a video processed based on the acquired information. By processing the video in a way that is suitable for the user viewing it, it is possible to obtain a video that is suitable for the user. Rather than simply processing the recorded video, the recorded video data is processed based on the acquired information, so there is a higher possibility of displaying a video that is more suitable for the user viewing it. For example, in a playback device that plays back video data recorded in a vehicle, it is preferable to process the recorded video data based on information acquired in the vehicle and to control the display of a video based on the processed video data. In particular, it is preferable to process the recorded video data based on information acquired in the vehicle at the same time as the video data recorded in the vehicle and to control the display of a video based on the processed video data. The same time period may be within a predetermined time range, which may be a few seconds, for example, but it is especially preferable for the same time period to be simultaneous.
The information to be acquired should preferably be information that changes in accordance with the driver's actions, and especially information about the driver's own behavior. Information about the driver's own state should preferably be information that shows a relationship between the driver's own state and the video recorded by the vehicle. The relationship can be objective, but it is especially preferable to have a subjective relationship with the driver. A subjective relationship should preferably be one that the driver thinks they saw, for example, information necessary to allow the observer to estimate "how they saw it."
However, it is best to acquire information other than the car's speed.

The recorded video data may be, for example, video data recorded by a recording device, or video data recorded on a memory card such as an SD (Secure Digital) card, microSD card, or XQD® card. The acquired information may be, for example, information input by the user viewing the video. By having the user input information and the playback device acquire that information, the likelihood of displaying video suitable for the user's viewing increases. Furthermore, the acquired information may be predetermined information. Information may also be acquired by reading such predetermined information. The processing may simply involve changing the video data before and after the processing.

(2) The processing described above may include a processing method that displays the user's field of view in the image.

This method makes the user's field of view in the video easier to understand. Knowing the user's field of view makes it easier to distinguish between parts of the image the user can see and parts they cannot. The user's field of view can be fixed, determined according to the user's age, or measured and determined for each user. In particular, the processing should involve displaying the user's field of view in the video based on the acquired information. Specifically, it is preferable to perform processing that displays the user's field of view in the video based on the information used to adjust the field of view input from the field of view input means, and the acquired information. Displaying the field of view in the video means displaying it in a manner that allows for the distinction between what is inside and outside the field of view. For example, the field of view can be enclosed in a frame, or the area outside the field of view can be filled in or blurred. The video may be, for example, video recorded by a recording device using a wide-angle camera, or video recorded by a recording device using a 360-degree camera.

(3) When an object requiring attention is found outside the user's field of view in the aforementioned video, it is advisable to notify the user accordingly.

This approach makes it easier for the user to understand that there are objects requiring attention in areas they cannot see. Whether or not there are objects requiring attention outside the user's field of view can be determined, for example, by detecting objects in each frame of the recorded video using image recognition. If the detected object's range in that frame falls within the area outside the field of view, it is determined that the object exists; otherwise, it is determined that the object does not exist. Furthermore, the object detection process should ideally omit, and preferably avoid, at least part of, the image recognition search for the object within the field of view in that frame. Ideally, the image recognition search for the object should only be performed within the area outside the field of view in that frame. For example, if the video is acquired by a recording device, the user will know that there are objects requiring attention in areas they cannot see, increasing the likelihood that they will be able to pay attention to areas they cannot see by moving their head while driving. Examples of objects requiring attention include pedestrians, bicycles, cars, and motorcycles. This should be communicated by displaying the location of the object within the video.

(4) The processing described above may be a processing that displays the user's gaze at the time of recording the video data on the video.

This makes it easier to understand where the user was looking during the recording of video data. In particular, the processing should be a process that displays the user's gaze during video recording based on the acquired information. Specifically, it is preferable to display the user's gaze on the video based on the user's gaze input from an input means for inputting the user's gaze during video recording, and the previously acquired information. For example, the user's eye movements could be recorded during video recording, and the user's commands could be displayed on the video based on that recording. The gaze can refer to, for example, the direction the user is looking, or it could be called their line of sight. Displaying the gaze on the video means, for example, displaying the gaze sequentially as points on the video, or displaying the trajectory of the gaze on the video. The user's gaze can be detected, for example, by photographing the user with a camera during video recording and detecting the gaze from the captured image, or by having the user wear gaze detection glasses or other gaze detection devices and detecting the gaze using those devices.

(5) The processing described above may be a processing method that moves the field of view in the direction the user is facing when recording the video data and displays it in the video.

This method makes it easier to determine the direction the user is facing during video data recording, and to understand the field of view in that direction. It also makes it easier to see which parts of the video were visible during recording. In particular, it is preferable to acquire information indicating the direction detected by a detection means that detects the direction the user is facing, and then move the field of view based on this acquired information to display it on the video. The field of view can be moved by processing the frames so that the areas inside and outside the field of view can be distinguished at regular intervals. For example, the field of view can be moved relatively, or it can be moved relatively by moving the video itself.

(6) For example, the processing may be a process that displays the portion of the video that the user has not viewed during a certain period of time while recording the video data.

This method makes it easier to identify areas that the user has not seen during a certain period of time. In particular, it is preferable to acquire information from a detection means that detects areas the user has not seen during a certain period of time, and then use this information to process and display the areas the user has not seen during that period on the video. It is also preferable to detect the user's gaze and use the detected user gaze information and the information for a certain period of time to detect areas the user has not seen during that period. The user's gaze can be detected, for example, by photographing the user with a camera during video data recording and detecting the gaze from the captured image, or by having the user wear gaze detection glasses or other gaze detection devices and detecting the gaze using those devices. The "certain period of time" could be, for example, the time spent stopped at an intersection, while turning right or left at an intersection, or while changing lanes, assuming the video data is recorded by a recording device. Displaying the areas not seen on the video can be done, for example, by filling in or blurring those areas.

(7) For example, the processing may include displaying on the video whether the user's gaze at the time of recording the video data matches the direction the user should be looking.

This makes it easier to see whether the user is looking in the direction they should be looking or not. In particular, it is preferable to obtain information from an input means that inputs information indicating whether the user's gaze during video data recording coincides with the direction the user should be looking, and then, based on the obtained information, perform a processing operation to display on the video whether the user's gaze during video data recording coincides with the direction the user should be looking. Alternatively, it is preferable to obtain information input from an input means that inputs information indicating the user's gaze during video data recording and information input from an input means that inputs information the user should be looking at, and then, based on the obtained information, perform a processing operation to display on the video whether the user's gaze during video data recording coincides with the direction the user should be looking. Alternatively, it is preferable to obtain information from a detection means that detects whether the user's gaze during video data recording coincides with the direction the user should be looking, and then, based on the obtained information, perform a processing operation to display on the video whether the user's gaze during video data recording coincides with the direction the user should be looking. For example, when video data is being recorded by a recording device, the direction the user should be looking is predetermined according to the surrounding circumstances. The system could also calculate the degree of agreement between the user's gaze and the direction the user should be looking.

(8) For example, when the video data is recorded by a recording device, the direction the user should look should differ depending on the driving operation of the car at the time the video data is recorded, in determining whether the user's line of sight and the direction the user should look should coincide.

This approach makes it easier to understand that the direction the user looks differs depending on whether the car was being driven or the user was operating the car during the recording of the video data. Because the direction the user looks differs depending on whether the car was being driven or the user was operating the car during the recording of the video data, it becomes easier to see whether the user is looking in the direction they should be looking or not, depending on the driving operation. In particular, it is good to determine the direction the user should look based on information from an input device that inputs information about whether the car was being driven or the user was operating the car during the recording of the video data. For example, when the car is being driven straight, the direction the user should look should be roughly straight ahead, and when changing lanes, the direction the user should look should be the rearview mirror, the side mirror of the lane being changed, or the side window on the side being changed. Furthermore, when driving, if you turn the steering wheel relatively sharply after activating the turn signal to make a right or left turn, you should look diagonally forward and diagonally backward on the side you turned the steering wheel. If you turn the steering wheel relatively sharply after activating the turn signal to change lanes, you should look in the rearview mirror, the side mirror of the lane you are changing to, or the side window on the side you are changing lanes to.

(9) For example, the processing may involve acquiring information indicating whether the order in which the user viewed the video data matches the order in which it should be viewed, and then displaying this information on the video based on the acquired information.

This makes it easier to determine whether the order in which the user viewed the video data matches the order in which it should have been viewed. In particular, it is advisable to display information on the video indicating whether the order in which the user viewed the video data matches the order in which it should have been viewed, based on information from a detection means that detects whether the order in which the user viewed the video data matches the order in which it should have been viewed during recording. The order in which the user viewed the video data during recording can be determined, for example, by analyzing video data obtained by filming a user driving a car and detecting the user's gaze in a time series. The order in which it should have been viewed can be determined based on information input from an input means that inputs information indicating that order. For example, the recorded video data can be played back, and the order in which an instructor or other person should view the video represented by the played-back video data can be input from the input means. The order in which it should have been viewed may change depending on the circumstances at the time of recording. For example, if the video data is from when the car is starting, one example of the order in which it should have been viewed is to look ahead, look in the rearview mirror or behind, look in the right side mirror or fender mirror, and look out the right window. The degree to which the order in which the user viewed the video data matches the order in which it should have been viewed can also be calculated.

(10) For example, the processing described above may be a processing method that displays the parts that the user should see in the video.

This approach makes it easier for the user to identify areas to focus on in the video. In particular, it's beneficial to retrieve information from an input device indicating areas the user should focus on, and then process the video to display those areas based on that information. For example, if the video data represents the driver's perspective while driving, the areas the driver should focus on become clearer, which is often useful for driving instruction. The processing could involve, for instance, marking the rearview mirror if that's the area to focus on, or marking the side mirrors if that's the area to focus on.

(11) For example, if the user is not looking at a specific part of the video data during recording, the image of that scene should be displayed.

This approach makes it easier to identify situations where the user is not looking at what they should be looking at. In particular, when recording video data, it is advisable to display the screen showing the scene where the user is not looking at what they should be looking at, based on information entered through an input device that indicates the user is not looking at what they should be looking at. Whether or not the user is looking at what they should be looking can be determined using gaze-tracking information obtained from a detection device that detects the user's gaze. For example, if the video data represents footage of the user driving, it becomes easier to identify situations where the user is not looking at what they should be looking at, thus increasing the likelihood of improving driving skills. Furthermore, if the video data represents footage of the user driving, evaluation can be performed based on images of scenes where the user is not looking at what they should be looking at.

(12) For example, the display speed of the portion of the video where the subject's movement does not change may be set to be faster than the display speed of the portion of the video where the subject's movement does change.

This approach increases the likelihood of quickly viewing the entire video. In particular, to determine whether a video section shows no change in the subject's movement or does, a detection method should be used. This detection method should detect the difference between images between frames; if the difference is less than a certain threshold, it indicates no change in the subject's movement; if the difference is greater than a certain threshold, it indicates a change in the subject's movement. For example, if the video section showing no change in the subject's movement is considered less important, and the video section showing change in the subject's movement is considered more important, then the less important parts can be viewed more quickly, while the more important parts can be viewed more carefully. The video section showing no change in the subject's movement includes not only sections with no change at all, but also sections with very little change, and sections with a change of movement below a certain threshold. Furthermore, the video section showing change in the subject's movement includes not only sections with even slight movement, but also sections with a change of movement exceeding a certain threshold.

(13) For example, video data is recorded by a recording device, and the user's evaluation of the car's driving is performed on the portion of the video where the subject's movement changes.

This approach makes it easier for users to evaluate car driving in video segments where the subject's movement changes. In particular, to determine whether a video segment shows no change in subject movement or does, a detection method can be used to detect whether the video segment shows no change in subject movement or does. This detection method should detect the difference between images between frames, assuming that if the difference is less than a certain level, there is no change in subject movement, and if the difference is greater than a certain level, there is a change in subject movement. Specifically, the evaluation of car driving should be performed by analyzing the video, and an evaluation method should be used that gives a higher rating the closer the driving is to adhering to predetermined traffic rules, and a lower rating the closer it is to driving that does not adhere to predetermined traffic rules. For example, in driving evaluation, it can be considered more important to evaluate driving in situations where the subject's movement changes, such as when turning right, turning left, or changing lanes, than in situations where the vehicle is stopped at a traffic light and the subject's movement does not change. Driving evaluation can be performed in such situations.

(14) For example, the video data is recorded by a recording device, and the video is displayed based on the portion of the video that includes the customer.

This approach makes it easier to view footage that includes passengers. Whether a section of footage includes passengers can be determined by analyzing the video data and detecting individuals other than the driver inside the vehicle. For example, if recording devices are installed in commercial vehicles such as buses and taxis, and video data is recorded by such devices, footage of those vehicles containing passengers can be displayed.

(15) For example, it is advisable to find the relevant video portion from the video based on keywords that describe the situation in the video portion.

This approach makes it easier to find video segments related to the keyword within the video. In particular, it is desirable to detect the keyword entered from an input device that describes the situation of the video segment, and then use an analysis device to analyze the video and find the video segment related to the detected keyword. The situation of the video segment can be described as abstract conditions such as bright or dark, or as specific conditions such as the inclusion of a bullet train or a red car.

(16) For example, the video data is recorded by a recording device, and based on keywords that describe the condition of the car at the time the video data was recorded, it is possible to find the video portion related to those keywords from within the video.

This approach increases the likelihood of finding video segments related to keywords describing the car's condition at the time of recording. The system detects keywords entered into an input method for recording video data, and then uses an analysis tool to analyze the video and find the segments related to the detected keywords. Keywords describing the car's condition could include, for example, "the car accelerated," "the car reached a certain speed," or "the turn signal was activated."

(17) For example, the video data may be recorded by a recording device, and a warning may be issued if the user is not looking in the direction they should be looking while driving the car.

This approach makes it easier to warn the user if they are not looking in the correct direction. In particular, it is advisable to warn based on information entered by an input device that provides information indicating that the user was not looking in the correct direction. Such an input device should display the user's gaze, detected by a detection device that detects the user's gaze while driving, on the played-back video, and input information indicating that the displayed gaze was determined not to be looking in a predetermined direction. The warning could be, for example, by adding a warning mark to the displayed video, coloring part or all of the displayed video with a specific color, or outputting an alarm.

(18) For example, the video data is recorded by a recording device, and the processing is a processing method that distinguishes between the range visible to the user and the range not visible to the user while the user is driving the car, and displays the video accordingly.

This approach makes it easier to distinguish between areas visible to the user and areas not visible. In particular, it is preferable to use an input method that provides information indicating the user's visible area to process the image, distinguishing between the visible and invisible areas based on the input. For example, the areas not visible to the user could be blurred, filled with a semi-transparent color, or completely filled in.

(19) For example, the processing may be performed on the recorded video data in such a way that the display range of the video changes according to the orientation of the device that displays the video based on the processed video data.

This method makes the image easier to view depending on the orientation of the display device. It is preferable to change the display range of the image based on information from a detection means that detects the orientation of the display device based on the processed image data. For example, it is preferable to record the entire 360-degree range during recording and then display the image corresponding to the orientation of the display device.

(20) For example, the video data is recorded by a recording device, and the processing may be performed on the recorded video data in such a way that the display range of the video is changed, with the direction of travel of the vehicle as the reference direction, according to the orientation of the device that displays the video based on the processed video data.

This approach increases the likelihood of viewing the image according to the orientation of the display device, by changing the display range of the image relative to the direction of the vehicle's movement. It is preferable to change the display range of the image based on information from a detection means that detects the orientation of the display device based on the processed image data. This allows viewing not only the image in the direction of travel while the vehicle is moving, but also images in directions other than the direction of travel.

(21) For example, it is advisable to display information regarding the vehicle's condition at the time the video data was recorded.

This makes the vehicle's status easier to understand. Information regarding the vehicle's status could include, for example, the vehicle's speed, acceleration, and position. In particular, it is preferable to display information on the display screen of the display device using a display control means, based on information input from an input means for inputting information regarding the vehicle's status during video data recording. Information regarding the vehicle's status may be displayed overlaid on the video, or it may be displayed in a separate area from the area where the video is displayed. For example, information regarding the vehicle's status may be displayed in the area above, below, to the left, or above the video.

(22) For example, video data is recorded by a recording device, and a map image representing the vehicle's movement range at the time of video data recording is displayed. The video based on the video data recorded by the recording device is then displayed at a specified location within the displayed map image.

In particular, the vehicle's movement range during video data recording is detected by a detection means, image data representing a map image of the detected movement range is acquired by an acquisition means, the map image represented by the acquired image data is displayed on the display screen of the display device by a display control means, a location within the displayed map image is specified by a specification means, the video data recorded at the specified location is found within the video data by an image analysis means, and the found video is displayed on the display screen of the display device by the display control means. This makes it easier to view the video at the specified location from the map image. Even if the time of the desired video is unknown, if the location is known, the video at that location can be viewed.

(23) For example, it is preferable to display at least one of the following: an image based on video data recorded by the user's recording device at a specified location within the displayed map image, and an image based on video data recorded by a third party's recording device at a specified location within the displayed map image.

This approach increases the likelihood of viewing not only video data recorded by the user's recording device, but also video data recorded by a third party's recording device. In particular, a command is sent from the transmission means to a server or other location where video data recorded by a third party's recording device is stored at a specified location on the displayed map image. The video data sent from the server or other location in response to the command is received by the receiving means, and the video based on the video data recorded by the third party's recording device is displayed on the display screen of the display device by the display control means. Both the video based on the video data recorded by the user's recording device and the video based on the video data recorded by the third party's recording device may be displayed simultaneously, or they may be displayed in a switchable manner. The video based on the video data recorded by the third party's recording device may be from the same season as the video data recorded by the user's recording device, or from a different season. Furthermore, video footage based on video data recorded by a third-party recording device may depict the same weather conditions as when the video data was recorded by the user's recording device, or it may depict different weather conditions.

(24) For example, the name including the location where the recorded video data was filmed may be displayed as the name that identifies the recorded video data.

In this way, a name including the filming location can be used as the name to identify the video data. This makes it easier to find video data at a specific location. In particular, it is preferable to detect the filming location from the data indicating the recording position of the recorded video data using a detection means, change the name including the detected filming location using a renaming means to identify the recorded video data, and then display the changed name on the display screen of the display device using a display control means. Specifically, it is preferable to change the file name storing the recorded data to include the filming location of the recorded video data using a renaming means. For example, the file name of the video file storing the video data may include the filming location. Alternatively, the name of the filming location itself may be used as the name to identify the video data.

(25) For example, if multiple video data sets are recorded, a name that identifies the video data from the same shooting location may be displayed collectively.

This approach makes it easier to determine whether the video data is from the same filming location. In particular, it is advisable to control the display device using a display control means to group the names used to identify video data from the same filming location and display them consecutively on the screen. Filming locations with many identical shots can be considered everyday locations such as commuter or school locations, while filming locations with few identical shots can be considered non-everyday locations such as travel destinations. This increases the likelihood of easily selecting video data from everyday locations or non-everyday locations.

(26) For example, it is advisable to prioritize displaying names that appear less frequently.

This approach prioritizes the display of less frequently occurring names, making it easier to select the corresponding video data. In particular, it is desirable to control the display device using a display control means to prioritize the display of less frequently occurring names on the screen. A low frequency can be determined, for example, when the frequency of occurrence falls below a predetermined threshold. Prioritizing display means, for example, displaying less frequently occurring names at the top when multiple identifying names are displayed in a list format vertically, or displaying them earlier in the sequence when they are displayed sequentially.

(27) For example, displaying an image of a map of the shooting location would be a good idea.

This approach increases the likelihood of viewing a map of the filming location in addition to the video data. In particular, it is desirable to use a search mechanism to find an image of the filming location based on the location data, and then use a display control mechanism to display the found map image on the display screen of the device.

(28) For example, the video data is recorded by a recording device, and it is preferable to display a map image representing the range of movement of the car during the recording of the video data, and to display information regarding the operation of the car during the recording of the video data on the map image.

This approach makes it easier to understand what kind of vehicle operations were performed at the location shown in the map image. By viewing the vehicle operation information displayed on the map image, it becomes possible to understand what kind of vehicle operations are likely to be necessary when driving in the location represented by the map image. The system should work as follows: a search means finds the map image representing the vehicle's travel range from data indicating that range; a display control means controls the display device to display the found map image on the screen; vehicle operation information is input from an input means; and the display control means controls the display device to display the input information on the map image. Vehicle operations could include, for example, using the turn signal, sudden steering, or sudden braking.

(29) For example, the video data is recorded by a recording device, and the video based on the processed video data is displayed, along with information based on a comparison between the driving of a third party and the driving of the user at the location where the video data was recorded.

This approach makes it easier to compare the driving of a third party with the driving of the user. This comparison increases the likelihood that the user can objectively evaluate their own driving skills compared to those of a third party. In particular, it is preferable to input data indicating the recording location of the video data from an input means, acquire information showing the driving of a third party at the recording location indicated by the input data using an acquisition means, use the acquired information about the third party's driving to perform a comparison between the user's driving and the third party's driving using a comparison means, and then control the display device using a display control means to display information based on the comparison on the display screen. The information based on the comparison between the third party's driving and the user's driving could, for example, be the ranking of the third party's driving and the user's driving, or the average evaluation of the third party's driving and the evaluation value of the user's driving.

(30) For example, the video data is recorded by the user's recording device, and it is preferable to display the video based on the processed video data and the video based on the video data recorded in the recording device in response to the operation of a third party at the location where the video data was recorded.

This approach makes it easier to compare video footage of the user's driving with video footage of a third party's driving. By viewing the third party's driving, the user is more likely to understand the strengths and weaknesses of their own driving. In particular, it is preferable to input data indicating the location where the video data is recorded from an input means, acquire video data recorded in accordance with the third party's driving at the recording location indicated by the input data using an acquisition means, and then control the display device using a display control means to display the video based on the acquired video data recorded in accordance with the third party's driving, along with the video obtained in accordance with the user's driving, on the display screen. The location where the video data is recorded is, for example, the location where video data obtained from the user driving a vehicle equipped with a recording device is recorded.

(31) For example, information obtained through at least one of vehicle-to-infrastructure communication and vehicle-to-vehicle communication is recorded in association with video data, and control is performed to display the information obtained through at least one of vehicle-to-infrastructure communication and vehicle-to-vehicle communication in relation to the video based on the processed video data.

This approach makes the information obtained through at least one of the vehicle-to-infrastructure (Vehicle-to-Infrastructure) and vehicle-to-vehicle (Vehicle-to-Vehicle) communication easier to understand. In particular, it is preferable to control the display device using a display control means to read the information obtained through at least one of the vehicle-to-infrastructure and vehicle-to-vehicle communication using a reading means, and to display the read information on a display screen. The information obtained through at least one of the vehicle-to-infrastructure and vehicle-to-vehicle communication may be, for example, information obtained based on an ITS (Intelligent Transport Systems). The information obtained through vehicle-to-infrastructure communication may be, for example, information transmitted from a roadside unit installed on the road, and the information obtained through vehicle-to-vehicle communication may be information communicated between vehicles. Displaying in relation to the video means that the information is related to the video; it may be displayed on the video, displayed in a separate area from the video, or displayed when the video is being displayed.

(32) For example, the processing step may be a processing step that corrects the color to a memory color.

This approach makes it easier to obtain images based on memory colors. In particular, it is beneficial to use this as a correction method to adjust the image to match memory processing. For example, it increases the likelihood of obtaining an image that is relatively close to the user's memory. Memory colors can be defined as colors that the user remembers as an image, or colors associated with a specific object. Correction to memory colors can be achieved, for example, by changing brightness, hue, and saturation through user input while viewing the image.

(33) For example, the processing may be a processing that blurs at least a portion of the image based on the recorded video data.

This method increases the likelihood of seeing at least a portion of the image blurred. In particular, processing the image using a blurring method is recommended. You can blur a specified area of the image, or blur everything outside that area. You can also make it possible to switch between blurring the specified area and blurring everything outside the specified area.

(34) For example, when outputting audio for the video based on the processed video data, a communication robot may be further provided that faces the display screen showing the video, and when outputting audio to a viewer of the video based on the processed video data, it faces the viewer.

This approach increases the likelihood of obtaining a communication robot that, when outputting audio related to a video, faces the display screen showing the video, and when outputting audio to the viewer of the video, faces the viewer. In particular, the communication robot should ideally have an audio output means, a head rotation means capable of moving its neck between the display screen and the viewer, a determination means for analyzing the video displayed on the display screen and determining whether to output audio related to the video or to the viewer, and a head rotation control means for controlling the head rotation means so that when outputting audio related to a video, it faces the display screen showing the video, and when outputting audio to the viewer, it faces the viewer, increasing the likelihood that the viewer will pay attention to the video, and when outputting audio to the viewer, it faces the viewer, increasing the likelihood that the viewer will carefully listen to the audio output from the communication robot. The communication robot should, for example, be able to face both the display screen and the viewer, and also be capable of outputting audio.

(35) For example, it is preferable to control the display of the processed video data based on road information to change the manner in which the video is displayed.

This approach makes it easier to change the display mode of the video based on information about roads. In particular, it is advisable to analyze the video data, detect road-related information contained within the video, and then change the display mode of the video based on the processed video data based on the detected road-related information. Road-related information could include, for example, road width and road signs. For instance, the display mode could be changed by altering the brightness, hue, and saturation of the video based on the road-related information, or by displaying videos with different camera angles.

(36) For example, it may be a recording device that records the video data to be played back by the playback device.

This increases the likelihood of recording the video data to be played back on the playback device. For example, it could be a drive recorder that records the video data to be played back on the playback device. The recording device may be installed in four-wheeled vehicles such as automobiles, motorcycles, bicycles, etc.

(37) For example, the recording device may be controlled to record the video data in high quality based on specified conditions.

This method makes it easier to record video data in high quality based on specified conditions. In particular, it is preferable to specify the conditions using a specification means, input the specified conditions using an input means, and then control the recording device using a recording control means to record video data in high quality based on the input conditions. The specified conditions could be, for example, location and time. High quality could be achieved, for example, by increasing the frame rate above a certain threshold or setting the resolution higher than a certain level.

(38) For example, at least one of the playback device and the recording device may be equipped with a specifying means for specifying conditions, and control may be performed to record the video data in high quality based on the conditions specified by the specifying means.

This increases the likelihood of recording high-quality video data based on the conditions specified by the designation means. In particular, it is preferable to specify the conditions using the designation means, input the specified conditions using the input means, and control the recording device with the recording control means to record high-quality video data based on the input conditions. When the playback device is equipped with the designation means, for example, it is preferable to record data representing the conditions specified by the designation means on a recording medium for recording video data, load such a recording medium into the recording device, and read the data representing the specified conditions from the recording medium, thereby recording high-quality video data in the recording device based on the specified conditions.

(39) For example, the condition may be either that the speed of the vehicle on which the recording device is installed exceeds a certain level, or that the vehicle on which the recording device is installed has traveled to a designated location.

This approach makes it easier to record high-resolution video data when the vehicle's speed exceeds a certain level or when it is driving through a designated location. In particular, it is preferable to use a camera installed in the recording device to photograph the vehicle's speedometer, analyze the captured video using an analysis means, and then, when a determination means determines that the vehicle's speed has exceeded a certain level, high-resolution recording is initiated. Alternatively, the vehicle's position may be detected by a position detection means, and high-resolution recording may be initiated when a determination means determines that the vehicle is driving through a designated location.

(40) For example, when recording the video data by the recording device, it is preferable to record the information obtained through at least one of the vehicle-to-infrastructure communication and vehicle-to-vehicle communication in association with the video data.

This approach increases the likelihood of recording information obtained through at least one of the vehicle-to-infrastructure and vehicle-to-vehicle communication in association with video data. In particular, it is preferable to receive the information transmitted through at least one of the vehicle-to-infrastructure and vehicle-to-vehicle communication using a receiving means, and then record the received information in association with the video data obtained through filming. Specifically, it is preferable to record the information transmitted through at least one of the vehicle-to-infrastructure and vehicle-to-vehicle communication in the header of the video file storing the video data obtained through filming.

(41) For example, control may be performed to transmit data representing a first calculated value uniquely determined from the video data recorded by the recording device to the blockchain system.

This approach increases the likelihood of transmitting data representing a first calculated value uniquely determined from the video data to the blockchain system. It also allows the data representing the first calculated value to be recorded in the blockchain. In particular, it is preferable to control the transmission means to calculate a first calculated value uniquely determined from the video data recorded by the recording device, and then transmit the data representing the calculated first calculated value to the blockchain system. The calculation means for calculating the first calculated value may be provided in the recording device or the playback device, and the transmission means may also be provided in the recording device or the playback device. Since the blockchain is relatively unlikely to be tampered with, recording the data representing the first calculated value in the blockchain increases the likelihood that the data representing the first calculated value has not been tampered with, and thus increases the likelihood of proving that the video data has not been tampered with using the data representing the first calculated value and the video data.

(42) For example, the video data recorded by the recording device is MPEG-based video data, and it is preferable to control the recording of data representing a first calculated value, uniquely determined for each unit, in a block chain, using the area immediately preceding the P-frame as the unit for dividing the video data.

This approach increases the likelihood of recording data representing the first calculated value for each unit, separated immediately before the P-frame, in the block chain. Since a P-frame cannot be reproduced as a single image without an I-frame, calculating the first calculated value for each unit separated immediately before the P-frame increases the likelihood of the calculation becoming more difficult. In particular, it is preferable to control the transmission means to use the point immediately before the P-frame as the unit for separating the video data, calculate a uniquely determined first calculated value for each unit, and transmit the data representing the calculated first calculated value to the block chain system.

(43) For example, it is advisable to implement control to record data representing a second calculated value uniquely determined from event data into the blockchain.

This approach increases the likelihood of recording data representing a second calculated value uniquely determined by event data in the blockchain. It also makes tampering with the second calculated value more difficult. In particular, it is desirable to control the transmission means to calculate a second calculated value uniquely determined from event data using a calculation means, and then transmit data representing the calculated second calculated value to the blockchain system. By comparing the second calculated value with the event data, it is possible to determine whether the event data has been tampered with. Since tampering with the second calculated value is difficult, tampering with the event data also becomes difficult.

(44) For example, it is preferable to record data indicating whether the video data recorded in the recording device was from manual operation or automatic operation, in association with the video data.

This approach increases the likelihood of determining whether the video data was recorded during manual or autonomous driving. In particular, it is advisable to input identification data from an input device indicating whether the video data was recorded during manual or autonomous driving, and to record this input identification data in association with the video data. Sometimes, the identification data should be recorded in the header area of the video file in which the video data is recorded. Manual driving can be defined as, for example, a person operating the steering wheel, while autonomous driving is defined as a system that controls the steering wheel mechanically without human intervention.

(45) For example, the image quality of the video data recorded by the recording device may be changed depending on whether the vehicle is being driven manually or automatically.

This approach increases the likelihood of obtaining different image quality data for manual and autonomous driving. In particular, it is advisable to input identification data indicating whether the vehicle is in manual or autonomous driving mode, and then change the image quality of the recorded video data (when the identification data indicates manual driving) and the image quality (when the identification data indicates autonomous driving) using a modification device. The image quality could be increased during manual driving and decreased during autonomous driving, or vice versa.

(46) For example, a communication robot could be operated during autonomous driving.

This approach increases the likelihood of operating the communication robot during autonomous driving. During autonomous driving, for example, the user does not need to drive, allowing for communication with the communication robot without interfering with driving. In particular, it would be beneficial if the recording device's input means input identification data indicating autonomous driving, and if such identification data is input to the input means, the recording device's transmission means sends an operation command to the communication robot.

(47) For example, it is advisable to display video based on recorded video data and to control the display mode of the video in response to a determination that there is a high probability of an event occurring.

This approach makes it easier to change the video display mode in response to a determination that there is a high probability of an event occurring. In particular, it is preferable to install a display device on a recording device in the vehicle, display video on the display screen of the display device, analyze the video using an analysis means while the vehicle is in motion, and, based on the analysis results, determine that there is a high probability of an event occurring in front of the vehicle, then control the display device using a display control means to change the video display mode on the display screen. For example, by recognizing that the video display mode has changed, it becomes clear that there is a high probability of an event occurring, allowing preparation for that event to be made. For example, if video data is recorded by a drive recorder installed in a moving car, a change in the video mode indicates, for example, that there is a possibility of some kind of event occurring in a few seconds, such as a sudden pedestrian or a car suddenly appearing.

(48) For example, it is advisable to record data representing the communication status of the communication radio waves received by the communication device.

This method makes it easier to record data representing the communication status of communication radio waves. In particular, it is advisable to receive the communication radio waves using a communication device compatible with receiving communication radio waves and record data representing the communication status of the communication radio waves. Specifically, the data representing the communication status should indicate whether communication is in progress or interrupted. For example, the communication radio waves should be based on LTE (Long Term Evolution), Wi-Fi, or Bluetooth (registered trademark). When displaying video, the display mode may be changed according to the communication status.

(49) For example, the recording mode of the video data may be changed based on the communication radio waves received by the communication device.

This method makes it easier to change the video data recording mode based on communication radio waves. In particular, it is advisable to receive communication radio waves using a communication device compatible with receiving such signals, and then change the video data recording mode based on the received signals. The recording mode itself can be anything as long as the recording method changes. For example, you could set the recording mode to high-definition when receiving communication radio waves and to low-definition when not receiving them. The reverse is also possible.

(50) For example, it is preferable to record video data representing the driver who has been called in attendance in association with the video data obtained from the driver's driving.

This approach increases the likelihood of recording video data representing the driver during roll call in association with video data obtained from the driver's driving. In particular, it is desirable to input video data representing the driver during roll call using an input device, and then record this input video data in association with video data obtained from the driver's driving, captured by a camera installed in the recording device. It is especially desirable to store these video data in the same folder. For example, it becomes possible to display video obtained from the driver's driving in relation to the video of the driver during roll call. For example, it becomes possible to display video of the driver during roll call and video of the driver's driving simultaneously.

(51) For example, it is preferable to perform control to record the video data obtained by a recording device installed on a truck driven by the driver in association with the video data obtained by a recording device installed on a forklift operated by the driver.

This approach makes it easier to record and associate video data of truck driving with video data of forklift driving. In particular, it is preferable to transmit video data obtained from a camera mounted on a recording device attached to the truck, capturing the truck driver's driving, to a server, and video data obtained from a camera mounted on a recording device attached to the forklift, capturing the forklift driver's driving, to the server, and then record these video data on the server in association with each other. Specifically, it is preferable to create a video folder for each driver on the server, and store the video data of the driver driving the truck and the video data of the driver driving the forklift in the same video file. Alternatively, when driving the truck, the video data of the truck driving may be recorded on a recording medium loaded in the recording device attached to the truck, and when driving the forklift, the recording medium may be removed from the recording device attached to the truck and loaded into the recording device attached to the forklift, and the video data of the forklift driving may be stored in the same video file as the video data of the truck driving. For example, it becomes possible to compare and view video of a driver driving a truck with video of the same driver driving a forklift.

(52) For example, it would be beneficial to control the system so that information that drivers should pay attention to is displayed overlaid on the actual scenery visible from a moving vehicle.

This approach increases the likelihood of displaying information that drivers should pay attention to while driving, overlaid on the real-world scenery. This allows drivers to understand the information while actually driving, while simultaneously viewing the real-world scenery. In particular, it would be beneficial to display this information on a Mixed Reality (MR) device while the driver views the real-world scenery. Specifically, for example, the user could wear MR glasses, viewing the real-world scenery through the glasses, and simultaneously displaying the information in a semi-transparent form overlaid on the real-world scenery.

(53) For example, it is advisable to implement control that notifies the user if they have traveled to the same location as previously traveled.

This method makes it easier to determine whether a location has been traveled before. Knowing that a location has been visited previously allows for new possibilities, such as deciding whether to follow the same route or a different one. In particular, the recording medium loaded into the recording device contains data indicating previous travel locations. The recorded data shows the previous travel location, which is then compared with the current location. If they match, the system notifies the user that they are traveling in the same location as before through an audio output from the audio output device or a display on the display screen. The GPS (global positioning system) receiver installed in the recording device is ideally used to detect the current location.

(54) For example, it is preferable to control the system to photograph the car's speedometer and record the video data based on the photographed speedometer.

This approach increases the likelihood of recording video data based on the captured speedometer reading. In particular, it is desirable to position a camera within the vehicle's recording device to capture the speedometer, and then analyze the captured video using an analysis device. When the speedometer indicates a speed that triggers video data recording, recording should begin. For example, video data should be recorded when the vehicle's speedometer exceeds 80 km/h.

(55) For example, a recording device attached to an automated guided vehicle may be used to record the video data based on the detection of approaching objects around the automated guided vehicle.

This approach increases the likelihood of recording video data based on the detection of approaching objects around the automated guided vehicle (AGV). In particular, it is advisable to use a camera installed in the recording device to photograph the area near the AGV, and then, when the detection means detects the approach of an object around the AGV, record the video data obtained from the camera's image. For example, this would allow us to see what happens when the AGV comes into contact with an object.

(56) For example, a recording device attached to a forklift may be used to control the recording of the video data based on at least one of the operation of the forklift and the fact that the driver is on the forklift.

This approach makes it easier to record video data based on at least one of the forklift's movement and the driver's presence on the forklift. In particular, it is preferable to start recording video data when the detection means detects at least one of the forklift's movement or the driver's presence on the forklift. Specifically, it is preferable to use a camera installed in the recording device to film the inside of the forklift, and then analyze the video data obtained to detect whether the forklift has moved or whether the driver has entered the forklift, and start recording video data in response to this detection. Forklift movement could, for example, be defined as the forks moving. For example, video data recording could be stopped when the driver dismounts from the forklift.

(57) The trigger information for event recording may be recorded in the header of the file storing the video data.

This approach increases the likelihood of recording event trigger information in the file header. For example, it increases the possibility of using that trigger information to perform various actions.

(58) For example, a playback method for playing back recorded video data is characterized by processing the recorded video data based on acquired information and performing control to display the video based on the processed video data.

This approach increases the likelihood that users will see processed video based on acquired information, rather than the recorded video itself. By tailoring the processing to the user's preferences, a more user-friendly video can be obtained. Because the recorded video data is processed based on acquired information, rather than the recorded video itself, the likelihood of displaying a more suitable video for the user increases.

(59) In the playback method described above, it is preferable to control the recording device to record the video data to be played back.

This increases the likelihood that the video data to be played back can be recorded on the recording device.

(60) For example, a computer-readable program for playing back recorded video data may be a computer-readable program that processes the recorded video data based on acquired information and controls the display of video based on the processed video data.

In this way, users can view not the recorded video itself, but a video processed based on the acquired information. By applying processing tailored to the user viewing the video, a user-friendly video can be obtained. Because the recorded video data is processed based on the acquired information, rather than the recorded video itself, the likelihood of displaying a video more suitable for the user increases.

The inventions described in (1) through (60) above can be combined in any way. For example, one may combine all or part of the configuration of the invention described in (1) with at least part of the configuration of at least one of the inventions described in (2) and subsequent inventions. In particular, it is preferable to combine the invention described in (1) with at least part of the configuration of at least one of the inventions described in (2) and subsequent inventions. Alternatively, one may extract any configuration from the inventions described in (1) through (60) and combine the extracted configurations. The applicant of this application intends to obtain rights to inventions that include these configurations.

In this way, users can view not the recorded video itself, but a video that has been processed based on the acquired information. By applying processing that is suitable for the user viewing the video, a video more suitable for the user can be obtained. Since the recorded video data is processed based on the acquired information, rather than the recorded video itself, there is a higher possibility of displaying a video that is more suitable for the user viewing the video.
The effects of the present invention are not limited thereto, and the effects produced by the components of the structure disclosed in this specification and the drawings are also disclosed. The applicant intends to obtain rights to the components that produce such effects through divisional applications, amendments, etc. For example, the phrases "can do..." in this specification are descriptions that clearly indicate the effects produced, and there are components that produce effects even without such descriptions. Furthermore, there are effects that can be grasped by the component even without such descriptions.

This is a side view of a dashcam. This is a block diagram showing the electrical configuration of a dashcam. This shows the file structure of the video file. This is a flowchart showing the record processing procedure. This flowchart shows the procedure for generating header data. This is a flowchart showing the video data recording process. This flowchart shows the data output processing procedure. This flowchart shows the data output processing procedure. This is an example of the image displayed on the screen. This is a block diagram showing the electrical configuration of a personal computer. This is a flowchart showing the playback process. This is a flowchart showing the playback process. This is a flowchart showing the playback process. This is an example of a playback window. This is an example of a playback window. This is an example of a playback window. This is an example of a playback window. This is an example of a playback window. This is an example of a playback window. This is an example of a playback window. This is an example of a playback window. This is an example of a playback window. This is an example of a playback window. This is an example of a playlist display area. This is an example of a playlist display area. This shows I-frames and P-frames. This is a flowchart showing the record processing procedure. This is a flowchart showing the processing procedure for a dashcam installed in a truck. This flowchart shows the processing procedure for a drive recorder attached to a forklift. This is a flowchart showing the record processing procedure. This is a flowchart showing the processing procedure for an automated guided vehicle (AGV). This is a flowchart showing the processing procedure for operational evaluation. This is a flowchart showing the playback process. This is a flowchart showing the playback process. This is a flowchart showing the playback process. This is an example of a map image. This is a flowchart showing the playback process. This is an example of a playback window. This is an example of a playback window. This is an example of a personal computer and a communication robot. This is a block diagram showing the electrical configuration of a communication robot. This is a flowchart showing the processing steps of a communication robot. This shows a communication robot placed inside a vehicle.

(1) Recording process Figures 1 to 9 show the recording process using a drive recorder.

Figure 1 shows an embodiment of this invention, and is a side view of a drive recorder (for example, an example of a recording device) 1.

The drive recorder 1 comprises a mounting section 2 and a camera section 4.

The mounting section 2 is fixed to the windshield (not shown) of the vehicle to which the drive recorder 1 is installed, and allows for the insertion and removal of the memory card. Furthermore, a display screen 3 for displaying the recorded video is formed on a portion of the outer surface of the mounting section 2. The mounting section 2 also includes a DC (direct current) jack (not shown) for connection to the vehicle via a converter unit (not shown). Additionally, the mounting section 2 is equipped with predetermined buttons (not shown).

The camera unit 4 incorporates a front camera (not shown in Figure 1) that primarily captures the area in front (the direction of travel of the vehicle to which the drive recorder 1 is attached) and a rear camera (not shown in Figure 1) that primarily captures the area behind the vehicle. The camera unit 4 is equipped with a front lens 5 that focuses an image of the subject on the image-forming plane of the front camera, and a rear lens 6 that focuses an image of the subject on the image-forming plane of the rear camera. The front camera can capture the area in front of the vehicle, to the left and right of the front of the vehicle, and up and down in the front of the vehicle. The rear camera can capture the area behind the vehicle, to the left and right of the rear of the vehicle, and up and down in the rear of the vehicle. Therefore, the front and rear cameras together can capture an omnidirectional view centered on the drive recorder 1.

The camera unit 4 can move forward and backward within a predetermined angle θ relative to the mounting unit 2, allowing for shooting at a relatively suitable angle regardless of the angle of the windshield.

Figure 2 is a block diagram showing the electrical configuration of drive recorder 1.

The overall operation of drive recorder 1 is controlled by controller 10.

The drive recorder 1 is equipped with a memory card reader/writer 95. When a memory card 25 is inserted into the drive recorder 1, the memory card reader/writer 95 reads data recorded on the memory card 25 and writes data to the memory card 25. The memory card 25 contains an operating program, which will be described later. The controller 10 reads this operating program and installs it into the drive recorder 1. The controller 10 reads the operating program installed in the drive recorder 1 and controls various parts of the drive recorder 1 according to the operating program. If the operating program of the drive recorder 1 is stored on a recording medium other than the memory card 25, it may be read from such a recording medium. In this case, an interface such as a driver corresponding to the recording medium will be provided in the drive recorder. Alternatively, it may be received via the communication circuit 17 and installed into the drive recorder 1.

As described above, the drive recorder 1 is equipped with a front camera 11 and a rear camera 12. The video data captured by the front camera 11 and the rear camera 12 are input to the controller 10. The drive recorder 1 is also equipped with a display device having a display screen 3, and the display control device 13 controls the display on the display device 14. Furthermore, the drive recorder 1 is equipped with a clock 15, a vehicle-to-infrastructure/vehicle-to-infrastructure information receiving circuit 16, a communication circuit 17, and an acceleration sensor 18. The time data representing the time output from the clock 15 is output to the controller 10. The vehicle-to-infrastructure/vehicle-to-infrastructure information receiving circuit 16 receives, for example, data representing the surrounding conditions transmitted from roadside units placed on roads based on an advanced road information system (an example of vehicle-to-infrastructure information), and data representing the vehicle's own information transmitted from the vehicle (an example of vehicle-to-vehicle information), and outputs the received data to the controller 10. The vehicle-to-infrastructure/vehicle-to-vehicle information receiving circuit 16 utilizes a communication circuit based on, for example, DSRC (Dedicated Short Range Communications). The communication circuit 17 communicates with, for example, a communication robot if one is placed inside the vehicle. The output data from the communication circuit 17 is input to the controller 10. The acceleration sensor 18 detects the acceleration of the vehicle to which the drive recorder 1 is installed and outputs the detected acceleration data to the controller 10.

Furthermore, the drive recorder 1 is equipped with input devices 20 such as buttons, memory 21 such as RAM (Random Access Memory), and a GPS (Global Positioning System) receiver 22. The output signals from the input devices 20, the output data from the memory 21, and the output data from the GPS receiver 22 are also input to the controller 10.

Figure 3 shows the file structure of the video files that store the video data output from the front camera 11 and the video data output from the rear camera 11.

The video file contains a header recording area 31 and a video recording area 32. The header recording area 31 is for recording header data, and the video recording area 32 is for recording video data.

Figure 4 is a flowchart showing the recording process of drive recorder 1.

The recording procedure shown in Figure 4 is repeated for each frame (or any other frame) captured until the recording process is complete.

When the engine of the vehicle to which the drive recorder 1 is installed starts, the vehicle's battery supplies power to the drive recorder. Then, the recording process shown in Figure 4 begins. A video file is also generated on the memory card 25. Once the recording process begins, video data output from the front camera 11 and rear camera 12 is input to the display control device 13 via the controller 10. The captured video is displayed on the display screen 3 of the display device 14 (see Figure 9). The video displayed on the display screen 3 is either the video obtained from the front camera 11 or the video obtained from the rear camera 12, according to the program recorded on the memory card 25. However, it is also possible to display both the video obtained from the front camera 11 and the video obtained from the rear camera 12. Alternatively, the default display could be the video obtained from either the front camera 11 or the rear camera 12, and a toggle button could be provided on the input device 20 to switch between the two depending on the operation of the button.

First, header data is generated and temporarily recorded to memory 21 (step 41). Then, video data is temporarily recorded to memory 21 (step 42). Next, various data output processes are performed (step 43), and the data temporarily recorded in memory 21 is written to memory card 25 (step 44).

In Figure 4, after the generation of header data and temporary recording of the generated header data to memory 21 (step 41), temporary recording of video data to memory 21 is performed (step 42), followed by output processing of various data (step 43). However, the order of these processes is not important. Furthermore, while the header data and video data are temporarily recorded to memory 21, the header data and video data recorded in memory 21 are read, and the read header data and video data are written to the memory card 25, if possible, the data may be written directly to the memory card 25 without temporary storage in memory 21. Also, two or more of the following processes—header data generation and recording (step 41), video data recording (step 42), and data output processing (step 43)—may be performed in parallel.

When the engine of the vehicle to which drive recorder 1 is installed is stopped, and the power supply to drive recorder 1 is cut off, the recording process shown in Figure 4 ends.

Figure 5 is a flowchart showing the header data generation process (the process described in step 41 of Figure 4), illustrating the process of controller 10.

If the vehicle to which the drive recorder 1 is installed is capable of autonomous driving, data indicating that the vehicle is in autonomous driving mode (driving that is not based on the driver's input) is transmitted from the vehicle to the drive recorder 1. For example, by connecting the drive recorder 1 to the vehicle's in-vehicle network and detecting the data output to the in-vehicle network indicating whether the vehicle is in autonomous driving mode or manual driving mode (driving that is based on the driver's input), the controller 10 of the drive recorder 1 can recognize whether the vehicle is in autonomous driving mode or manual driving mode. Alternatively, if the vehicle and the drive recorder 1 are capable of wireless communication and the vehicle transmits data indicating whether it is in autonomous driving mode or manual driving mode, the controller 10 of the drive recorder 1 may recognize whether the vehicle is in autonomous driving mode or manual driving mode by receiving that data in the communication circuit 17 of the drive recorder 1.

The controller 10 of the drive recorder 1 determines whether the vehicle is in autonomous driving mode or manual driving mode (step 51). If the vehicle is in autonomous driving mode, data indicating that the data recorded on the memory card 25 is video data captured during autonomous driving is temporarily recorded in the memory 21 as header data (step 52). Similarly, if the vehicle is in manual driving mode, data indicating that the data recorded on the memory card 25 is video data captured during manual driving is temporarily recorded in the memory 21 as header data (step 53). For example, data indicating whether video data captured within a certain time period was taken during autonomous driving or manual driving is temporarily recorded on the memory card 25. Of course, it is also possible to create an autonomous driving video folder to store video data taken during autonomous driving and a manual driving video folder to store video data taken during manual driving, and store video data captured during autonomous driving in the autonomous driving video folder and video data captured during manual driving in the manual driving video folder. The storage of video data in a time-based or automated driving video folder, or in a manual driving video folder, establishes an association between the video data (indicating whether it was from automated or manual driving) and the video data itself.

Next, the controller 10 checks whether the vehicle-to-infrastructure/vehicle-to-infrastructure information receiving circuit 16 has received vehicle-to-infrastructure communication information or vehicle-to-vehicle communication information (step 54). If vehicle-to-infrastructure communication information or vehicle-to-vehicle communication information is received (YES in step 54), data representing the received communication information, along with data representing the time the information was received, is temporarily recorded in memory 21 (step 55). Since the time the communication information was received is known, it is also possible to determine what kind of video was being filmed at that time, and when playing back the filmed video, the vehicle-to-infrastructure communication information or vehicle-to-vehicle communication information can be output simultaneously. In this way, for example, vehicle-to-infrastructure communication information or field-of-view vehicle-to-vehicle communication information can be associated with video data based on time. If neither vehicle-to-infrastructure communication information nor vehicle-to-vehicle communication information is received (NO in step 54), the process in step 55 is skipped, and the received information is not temporarily recorded in memory 21.

Furthermore, the system checks whether the communication circuit 17 has received a communication signal (step 56). If the communication circuit 17 receives a communication signal (YES in step 56), data representing the communication status of that signal is temporarily recorded in memory 21 (step 57). For example, if the communication circuit 17 receives radar waves emitted from speed enforcement devices such as speed cameras installed on the road, or radio signals used by emergency vehicles, data indicating the content of these signals is temporarily recorded in memory 21. If the communication circuit 17 does not receive a communication signal (NO in step 56), the process in step 57 is skipped, and data representing the communication status of the communication signal is not recorded in memory 21.

Furthermore, the driver's gaze is detected by the controller 10 from the video data obtained by the rear camera 12 (step 58). The gaze data representing the detected gaze, along with time data, is temporarily written to memory 21 (step 59). This allows us to determine when and in which direction the driver was looking. The eyes are detected from the video, and the gaze is determined by the positional relationship between the pupil, iris, and cornea (the so-called black part of the eye) and the conjunctiva (the so-called white part of the eye). Similarly, the video data obtained by the rear camera 12 is analyzed to detect the direction the driver is facing, and the data representing the detected direction, along with time data, is temporarily written to memory 21.

Figure 6 is a flowchart showing the video data recording process (the process described in step 42 of Figure 4).

During the video data recording process, the controller 10 confirms whether the vehicle is in autonomous or manual operation mode (step 61). If the vehicle is in autonomous operation mode, relatively low-resolution video data is recorded in memory 21 (step 62). For example, the controller 10 controls the front camera 11 and rear camera 12 so that the number of pixels per frame represented by the video data output from the front camera 11 and rear camera 12 is reduced to less than a first threshold (resolution reduction), or so that the number of frames read per unit time is reduced to less than a first number of frames (frame rate reduction). This results in relatively low-resolution video data. If the vehicle is in manual operation mode, relatively medium-resolution video data is recorded in memory 21 (step 63). For example, the front camera 11 and rear camera 12 are controlled by the controller 10 so that pixel decimation is performed so that the number of pixels per frame represented by the video data output from the front camera 11 and rear camera 12 exceeds a first threshold, or so that the number of frames read per unit time exceeds a first number of frames. This results in relatively medium-quality video data. In this way, the image quality of the video data can be adjusted by adjusting the resolution and frame rate. While relatively low-quality video data is recorded in memory 21 during autonomous driving and relatively medium-quality video data is recorded during manual driving, it may also be possible to record relatively medium-quality video data in memory 21 during autonomous driving and relatively low-quality video data during manual driving. The only requirement is that the image quality of the video data recorded during autonomous driving differs from that recorded during manual driving.

Next, the controller 10 checks whether the specified conditions have been met (step 64). For example, the vehicle speed exceeding a certain level, driving in a specified location, or, if the input device 20 of the drive recorder 1 is equipped with a high-definition setting switch (an example of a setting means), the operation of that high-definition setting switch are examples of conditions being met. Whether the vehicle speed has exceeded a certain level can be detected by analyzing the video obtained from the rear camera 12, if the drive recorder 1 is mounted on the vehicle so that the rear camera 12 can capture the vehicle's speedometer. It can also be detected from the distance traveled calculated based on the GPS receiver 22 and the time taken for that distance, calculated based on the clock 15. Whether the vehicle is driving in a specified location can be determined based on the output data from the GPS receiver 22.

If the specified conditions are met (YES in step 64), relatively high-resolution video data is temporarily recorded in memory 21 (step 65). For example, the controller 10 controls the front camera 11 and rear camera 12 so that pixel decimation of the video data output from them is stopped, or the number of frames read per unit time is increased compared to the number of frames at medium resolution. This results in relatively high-resolution video data. If the specified conditions are not met (NO in step 64), the process in step 65 is skipped, and the recording of high-resolution video data to memory 21 is not performed.

Furthermore, it is checked whether a command to change the recording mode has been received by the controller 10 (step 66). If a command to change the recording mode has been received (YES in step 66), the recording mode is changed (step 67); if no command to change the recording mode has been received (NO in step 66), the recording mode is not changed. For example, as described above, if the communication circuit 17 receives radar waves emitted from speed enforcement devices installed on the road, such as speed cameras, or communication radio waves such as those used by emergency vehicles, it may be determined that a command to change the recording mode has been received. A command to change the recording mode allows, for example, the recording mode to be changed to record high-resolution video data to the memory 21.

In the video data recording process shown in Figure 6, after confirming whether the vehicle is in autonomous or manual driving mode (step 61), a confirmation of whether the specified conditions have been met (step 64) is performed. However, the confirmation of whether the specified conditions have been met (step 64) may be performed first, followed by the confirmation of whether the vehicle is in autonomous or manual driving mode (step 61), or both confirmations (steps 61 and 64) may be performed in parallel.

Figure 7 is a flowchart showing the data output processing procedure (the processing procedure for step 43 in Figure 4).

In the data output processing procedure, the controller 10 confirms whether the vehicle is in automatic or manual operation mode (step 71). If the vehicle is in automatic operation mode, a start command is sent from the communication circuit 17 to the communication robot (not shown) located inside the vehicle (step 72). If the vehicle is in manual operation mode, no start command is sent from the communication circuit 17 to the communication robot. A stop command may also be sent to the communication robot during manual operation. During automatic operation, the driver is not manually operating the vehicle, so the operation of the communication robot does not interfere with driving. Furthermore, the driver can communicate with the communication robot, allowing for more efficient use of time. During manual operation, the communication robot inside the vehicle does not operate, preventing the driver from being distracted by the robot's conversations. Additionally, by configuring the system to send a stop command to the communication robot during manual operation, the robot can be stopped if it is currently operating.

Next, the controller 10 checks whether there is a high probability of an event occurring (step 73). For example, if the road narrows or there is a lot of pedestrian traffic, the driver may suddenly apply the brakes, making it more likely that an event will occur. If there is a high probability of an event occurring (YES in step 73), the display control device 13 is controlled by the controller 10 to change the display mode of the video shown on the display screen 3 of the drive recorder 1 (step 74). For example, the display mode of the video may be changed by displaying the words "Caution!" on the display screen, or by overlaying the entire video or a part of the video, such as the surrounding area, with a semi-transparent red or yellow color. If there is no high probability of an event occurring (NO in step 73), the display mode of the video shown on the display screen 3 of the drive recorder 1 is not changed.

Furthermore, when the vehicle is in motion (or even when stopped), the controller 10 also checks whether the current location is the same as a previously visited location (step 75). The memory card 25 installed in the drive recorder 1 contains GPS data indicating previously visited locations in the video file header recording area. The controller 10 compares the GPS data recorded in the header recording area with the GPS data received by the GPS receiver 22 installed in the drive recorder 1 while the vehicle is in motion, thereby confirming whether the current location is the same as a previously visited location. If the vehicle is driving in the same location as before (YES in step 75), the drive recorder 1 notifies the driver that it is the same location (step 76). If the vehicle is not driving in the same location as before (NO in step 75), the drive recorder 1 does not notify the driver that it is the same location.

Figure 9 shows an example of the display screen 3 of drive recorder 1.

The display screen 3 of the drive recorder 1 shows the video footage captured by the front camera 11. At the bottom of the display screen 3, the text 7 "You are driving in the same location as you have driven in before" is displayed. By seeing this text 7, the driver can see that they are driving in the same location as they have driven in before. The driver can then tell whether they are driving in a place that they have the impression of having driven in before due to déjà vu, or a place they have actually driven in before. Whether it is the same location as a place they have driven in before does not have to be an exact match, but it is sufficient if it falls within a certain range.

The system indicates that the location is the same as a previously driven location using the string 7. However, instead of displaying string 7, a predetermined mark may be displayed, or the display mode of the display screen 3 may be changed. The driver will understand that they are driving in the same location as a previously driven location by seeing such a mark or by confirming a change in the display mode. Furthermore, if the drive recorder 1 is equipped with a speaker, the system may not display string 7, etc., but instead output voice or sound from the speaker to inform the driver that they are driving in the same location as a previously driven location. In any case, the goal is simply for the driver to understand that they are driving in the same location as a previously driven location.

Returning to Figure 8, once one frame of video data is temporarily recorded in memory 21, the controller 10 calculates the hash value of the recorded video data (an example of a first calculated value uniquely determined from the video data) (step 77). The data representing the calculated hash value and the communication circuit 17 are transmitted to the blockchain system (step 78). The blockchain system records the data representing the calculated hash value in the blockchain (block transactions). Since the blockchain is difficult to tamper with, it becomes possible to verify whether the video data has been tampered with using the hash value. The memory card 25 also records the data representing the calculated hash value. To verify whether the video data has been tampered with, the data representing the hash value transmitted to the blockchain system and the data representing the hash value recorded on the memory card 25 are compared. If the hash values match, it is determined that the video data has not been tampered with; if the hash values do not match, it is determined that the video data has been tampered with. It goes without saying that the video data recorded on the memory card 25 and the data representing its hash value are recorded in association with each other on the memory card 25. For example, the data representing the hash value, along with data indicating which time period of video data the hash value was calculated from, is recorded in the video file's header recording area 31.

In this way, when the header data and video data are temporarily recorded in memory 21, the recorded header data and video data are read from memory 21, the header data is written to the header recording area of the video file, and the video data is written to the video recording area of the video file.

(2) Playback process Figures 10 to 23 show the playback process using playback viewer software.

Figure 10 is a block diagram showing the electrical configuration of a tablet-type personal computer (an example of a playback device).

The overall operation of the tablet-type personal computer (hereinafter referred to as "personal computer") 80 is controlled by the CPU (Central Processing Unit) 80.

The personal computer 80 is equipped with a display device 82. This display device 82 is
The display control device 81 is controlled by the CPU 90. The personal computer 80 is also equipped with an accelerometer 93, and the output signal from the accelerometer 93 is input to the CPU 90. Furthermore, the personal computer 80 includes an SSD (solid state drive) 94 accessed by the CPU 90, and a memory card reader/writer 95 that reads data recorded on the memory card 25 and writes data to the memory card 25.

Furthermore, the personal computer 80 includes input devices 96 such as a keyboard and mouse, memory 97 such as RAM, and a communication circuit 98 for connecting to a network such as the Internet.

The operating program, described later, is received via the internet by the communication circuit 98 of the personal computer 80 and installed on the personal computer 80. The CPU 90 reads and executes the installed program, thereby controlling the various components. Alternatively, the operating program may be stored on a recording medium such as a memory card 25, and read from such a recording medium and installed on the personal computer 80.

Figures 11 to 13 are flowcharts showing the playback process procedure in the personal computer 80. Figures 14 to 24 are examples of playback windows displayed on the display screen 92 of the personal computer 80.

The memory card 25 that was installed in drive recorder 1 is removed from drive recorder 1 and installed in personal computer 80. When the operating program installed on personal computer 80 is started by the user, the display device 92
The playback window 130 shown in Figure 14 is displayed on the screen.

Figure 14 shows an example of the playback window 130.

The playback window 130 has a menu bar 131 at its top, and below this menu bar 131 is a video display area 141. Furthermore, below the video display area 141 are a map image display area 151, a vehicle information display area 161, and a playlist display area 171.

Menu bar 131 displays various items, and by selecting each item, a pull-down menu appears, allowing for detailed commands. Menu bar 131 displays the strings "File,""View,""Playback,""Tools," and "Information," as well as folder icons 132, camera icons 133, printer icons 134, reel icons 135, FD (flexible disk) icons 136, log icons 137, and gear icons 138. By pulling down the strings "File,""View,""Playback,""Tools," and "Information," a pull-down menu corresponding to the respective item appears, allowing for detailed commands. Clicking the folder icon 132 selects a folder; clicking the camera icon 133 converts the image to a still image; clicking the printer icon 134 prints the image; clicking the reel icon 135 converts the image to a video; clicking the FD (flexible disk) icon 136 backs up the data; clicking the log icon 137 converts the image to log data; and clicking the gear icon 138 changes the recording settings. The video display area 141 is the area that displays the video recorded by the drive recorder 1, etc. The map image display area 151 is the area that displays a map image of the vicinity of the shooting location of the video displayed in the video display area 141. The vehicle information display area 161 is the area that displays information such as the vehicle's speed at the time of video recording. The vehicle information display area 161 includes a vehicle status display area 162 that displays the status of the vehicle to which the drive recorder 1 that recorded the video displayed in the video display area 141 is installed, operation buttons 163 that give operation commands such as playing and stopping the video, a time display area 166 that displays the recording time of the video displayed in the video display area 141, and an acceleration graph display area 168 that displays the vehicle's acceleration in a graph. The playlist display area 171 is an area that displays a list of filenames of video files recorded on the memory card 25.

When the folder icon 132 is clicked and the folder containing the video files is selected, the filenames of the video files stored in that folder are displayed as clickable files in the playlist display area 171, as shown in Figure 14. While the video recording start time is used as the filename, it goes without saying that other times can also be used as the filename.

When any of the filenames displayed in the playlist display area 171 is clicked using the mouse included in the input device 96, the corresponding video file is read from the memory card 25. Playback of the video data stored in the read video file begins (Figure 11, step 111). Various header data corresponding to the recording time of the video data are recorded in the header recording area of the video file. When the video from the recording time is displayed, the recorded header data is also read, and various overlay processing is performed on the video according to the content of the read header data.

When playback of video data (the video data to be played may be footage or recordings taken when the user viewing the video was driving, or footage or recordings taken when a user other than the user viewing the video was driving) begins, the video data is displayed as a video in the video display area 141, as shown in Figure 15. At the same time, data representing GPS information recorded in the video file containing the video data is transmitted via the internet to a website that transmits data representing a map image. Then, from that website, image data representing a map of the vicinity of the location where the video displayed in the video display area 141 was recorded is transmitted to the personal computer 80. When the image data representing the map is received by the personal computer 80, the map image is displayed in the map image display area 151. The map image also displays an arrow icon 152 representing the location displayed in the video display area 141. Furthermore, the vehicle status display area 162 of the vehicle information display area 161 displays the vehicle's speed, longitudinal acceleration (X), lateral acceleration (Y), vertical acceleration (Z), latitude (N), and longitude (E) at the time the video displayed in the video display area 141 was being recorded. The time display area 166 displays the recording time of the video displayed in the video display area 141, and the acceleration graph display area 168 displays an acceleration graph.

Since the video data is recorded in all directions, it is possible to display not only images of subjects visible from the direction of travel of the vehicle, but also images of subjects behind (opposite direction to the direction of travel), to the left of the vehicle (left relative to the direction of travel), or to the right of the vehicle (right relative to the direction of travel). Figure 16 shows a playback window 130 with images of subjects behind in the video display area 141, Figure 17 shows a playback window 130 with images of subjects to the right in the video display area 141, and Figure 18 shows a playback window 130 with images of subjects to the left in the video display area 141. For example, if you drag to the right using the mouse in the state shown in Figure 15, the image will gradually switch from images of subjects in the direction of travel to images of subjects to the right, and as shown in Figure 17, the image of subjects to the right will be displayed in the video display area 141. If you drag further to the right using the mouse in the state shown in Figure 17, the image of the subject to the right will gradually switch, and the image of the subject behind will be displayed in the image display area 141, as shown in Figure 16. If you drag further to the right using the mouse in the state shown in Figure 16, the image of the subject to the left will gradually switch, and the image of the subject to the left will be displayed in the image display area 141, as shown in Figure 18. If you drag further to the right using the mouse in the state shown in Figure 18, the image of the subject to the left will gradually switch, and the image of the subject in the direction of travel will be displayed in the image display area 141, as shown in Figure 15. In this way, images of subjects in all directions can be displayed in the image display area 141. In the above-described example, the mouse is used to drag to the right, but even if you drag to the left, similar to the above, the image will gradually switch from the image of the subject in the direction of travel as shown in Figure 15 to the image of the subject to the left as shown in Figure 18, then gradually switch to the image of the subject behind as shown in Figure 16, then gradually switch to the image of the subject to the right as shown in Figure 17, and then gradually switch back to the image of the subject in the direction of travel as shown in Figure 15.

In this embodiment, when an image is displayed in the image display area 141, the field of view is superimposed on the image (Figure 11, step 111).

Figure 19 shows an example of a playback window 130 displaying the user's field of view.

In the video display area 141, a frame 143 indicating the field of view is superimposed by the CPU 90 onto the video data (this is an example of video processing). The video portion outside the field of view area 142 is blurred by the CPU 90 (hatching is shown to indicate blurring). The field of view is predetermined for each user's age, and the data representing the field of view is pre-recorded in the SSD 94. By specifying the user's age, the data representing the field of view corresponding to the specified age is read from the SSD 94, and the frame 143 indicating the field of view and the area outside the field of view area 142 are blurred, as shown in Figure 19 (this is an example of control to process the video data based on the acquired information and display the video based on the processed video data). Of course, the user's field of view may be detected in advance for each user, and data representing the user's unique field of view may be pre-recorded in the SSD 94. The frame 143 indicating the field of view for each user will be superimposed on the video. Because the user understands their field of view, they can recognize what areas are visible and what areas are not.

Furthermore, the CPU 90 also determines whether there are objects requiring attention (for example, infants) in the area 142 outside the field of view (Figure 11, step 113). If there are objects requiring attention in the area 142 outside the field of view (YES in Figure 11, step 113), a warning (an example of notification to the user) is given to the user (step 114). For example, the area where the object requiring attention is displayed may be surrounded by a frame or a mark may be displayed. Whether or not there are objects requiring attention in the area 142 outside the field of view can be determined by pre-defining the objects requiring attention, having the CPU 90 perform video analysis, and determining whether the CPU 90 detects that the image of the object is displayed in the area 142 outside the field of view. The user will understand that since there are objects requiring attention outside the field of view, turning their head as needed to effectively widen their field of view while driving will help prevent accidents. If there are no objects requiring attention in area 142 outside the field of view (NO in step 113 of Figure 11), the process in step 114 is skipped, and no warning is issued. It is sufficient for the user to be notified if there are objects requiring attention in area 142 outside the field of view; therefore, the objects do not necessarily need to be displayed in area 142. For example, the user could be notified by sound output.

Furthermore, the CPU 90 superimposes a mark 144 indicating the user's gaze onto the video display area 141 (this is an example of a processing step that displays the user's gaze on the video) (Figure 11, step 115). Since the data indicating the user's gaze is recorded in the video file's header recording area 31 in association with the video's recording time, by reading the data indicating the user's gaze from the header recording area 31, a mark representing the user's gaze can be superimposed onto the video displayed in the video display area 141 at the recording time of the displayed video. Because the user's gaze is superimposed on the video, it is possible to confirm whether the user is looking at subjects they should be looking at (for example, traffic lights, oncoming cars, pedestrians, etc.).

In the example shown in Figure 19, the field of view frame 143 and the mark 144 indicating the user's line of sight are displayed regardless of the user's line of sight. That is, the field of view frame 143 indicates the area that would be the field of view when the user views the image displayed in the image display area 141. Therefore, regardless of the user's line of sight while driving, the field of view frame 143 is displayed in the approximate center of the image display area 141. For example, even if the user's line of sight is not in the center of the image but at the edge of the image displayed in the image display area 141, the field of view frame 143 will still be displayed in the approximate center of the image display area 141. Alternatively, the field of view frame 143 may be displayed in the image display area 141 centered on the user's actual line of sight. This would allow the user's actual field of view while driving to be determined. In this case as well, blurring of the area 142 outside the field of view may be performed. The area within the field of view frame 143 is an example of what the user can see while driving the vehicle, and the area outside the field of view 142 is an example of what the user cannot see while driving the vehicle. This is an example of how these areas can be distinguished and displayed.

If the user's direction changes during the recording time of the video displayed in the video display area 142 (YES in step 116 of Figure 11), the field of view frame 143 is moved in the changed direction and superimposed on the video (step 117 of Figure 11). Whether the user's direction has changed can be determined based on the data indicating the user's direction, recorded in the video file's header recording area. If the user's direction does not change (NO in step 116 of Figure 11), the field of view frame 143 is not moved.

Figure 20 shows an example where the field of view frame 143 is moved.

Assume that when recording the video displayed in the video display area 141, the user's facing direction changes from approximately directly in front of their direction of travel to the lower right. Then, the user's position shifts from the field of view frame 143 to the field of view frame 143A. Similarly, their position shifts from the line of sight mark 144 to the line of sight mark 144A. The user can recognize that their field of view has changed due to the change in their facing direction. When the user's facing direction changes, the relative positions of the video displayed in the video display area 141 and the field of view frame 143 (143A) also change. For this reason, it may be possible to ensure that the relative positions of the video and the field of view frame 143 do not change (for example, so that the field of view frame 143 is displayed approximately in the center of the video). For example, similar to how the video displayed in the video display area 141 changes when dragged with the mouse, the video in the direction the user is facing may be displayed.

Next, it is determined whether the user's gaze matches the direction the user should be looking (Figure 12, Step 118). If they do not match (NO in Figure 12, Step 118), a mark indicating the area to be looked at is displayed on the video display area 141 (Figure 12, Step 119). If they match (YES in Figure 12, Step 118), the mark indicating the area to be looked at is not displayed on the video display area 141. However, if they match, the user may be notified of this (displayed on the video display area 141, sound output, etc.). Needless to say, the direction to be looked at will differ depending on the driving during recording (right turn, left turn, lane direction, etc.) or the user's driving operations (brake operation, turn signal lever operation, etc.).

Figure 21 shows an example of a display showing markers indicating areas to be examined.

If the user was supposed to look at the lower right corner of the image displayed in the image display area 141, but failed to do so (for example, by looking straight ahead), then mark 144 will be displayed in the image display area 141 indicating that the user should have looked at that lower right corner (this is an example of processing that displays the area the user should look at in the image). The CPU 90 determines which area to look at by analyzing the displayed image, or by reading event data (such as turn signal lever operation or sudden braking) that indicates events during the recording of the image, if such event data is recorded in the image file's header recording area. For example, if the user changed lanes from the left lane to the right lane by operating the turn signal lever, but did not look at the right side mirror, mark 145 will be displayed indicating that the right side mirror should have been looked at. Furthermore, if the user needs to check a predetermined location, such as when turning right or left, the system may display a mark indicating the location to look at if the user is not looking at that location.

Furthermore, if the user is not looking at the intended section, the video display in the video display area 141 may be temporarily paused, and an image (still image) of that scene may be displayed in the video display area 141. While the video ends instantly when the user is not looking at the intended section, displaying an image of the paused state in the video display area 141 can better draw the user's attention.

Furthermore, for portions of the video displayed in the video display area 141 where there is little to no change in the movement of the subject (YES in step 120 of Figure 12), the CPU 90 increases the display speed relatively (step 121 of Figure 12). For portions of the video where there is significant change in the movement of the subject (NO in step 120 of Figure 12), the display speed is not increased relatively. Whether there is little or no change in the subject's movement can be determined by comparing the differences between video frames. Portions of video where there is little to no change in the subject's movement are considered less important, such as pedestrians suddenly appearing or oncoming vehicles swerving into the road, so increasing the display speed is not problematic. Also, portions of video where the vehicle's position is not changing are considered less important because the vehicle is stationary and therefore less likely to cause accidents. For this reason, the display speed may also be increased for portions of video where the vehicle's position is not changing.

Furthermore, the CPU 90 also determines whether the orientation of the display device 92 has been changed (Figure 12, step 122). If the orientation has been changed (YES in Figure 12, step 122), the display range of the image displayed in the image display area 141 is changed according to that orientation (Figure 12, step 123). If the orientation has not been changed (NO in Figure 12, step 122), the display range of the image is not changed according to the changed orientation. Whether the orientation of the display device 92 has been changed, and in which direction it has been changed, can be detected by the acceleration sensor 93. Suppose the display screen of the display device 92, which was facing a certain direction, is displaying an image of the direction of travel of the vehicle (see Figure 17). If the display device 92, which was facing a certain direction, is turned to the right, the image of the right direction (see Figure 17) will be displayed in the image display area 141, similar to when the image displayed in the image display area 141 is dragged to the right. Similarly, if the display device 92, which was facing a certain direction, is turned to the left, the image displayed in the image display area 141 will be shifted to the left (see Figure 18), similar to when the image displayed in the image display area 141 is dragged to the left. Furthermore, if the display device 92, which was facing a certain direction, is turned to the rear, the image displayed in the rear (see Figure 18) will be shifted to the rear (see Figure 18). Changing the display range of the image based on the vehicle's direction of travel is one example of image data processing. This enables a pseudo-VR (virtual reality) display.

Furthermore, the CPU 90 determines whether information regarding the vehicle's status is recorded in the video file's header recording area 31 at the time the video displayed in the video display area 141 is recorded (Figure 12, step 124). If information regarding the vehicle's status is recorded (YES in Figure 12, step 124), that information is displayed in the video display area 141. If information regarding the vehicle's status is not recorded (NO in Figure 12, step 124), that information is not displayed in the video display area 141.

Figure 22 shows an example of information regarding the vehicle's status displayed in the video display area 141.

A speech bubble 146 in the first display mode is displayed in the video display area 141. This speech bubble 146 displays information about the vehicle's status at the time the video displayed in the video display area 141 was recorded. In the example shown in Figure 22, the text "Brakes applied!!" is displayed in the speech bubble 146, indicating that the vehicle's brakes (emergency braking) were applied at the time the video displayed in the video display area 141 was recorded. Data representing information about the vehicle's status is also recorded in the video file's header recording area 31, associated with the time data, and read from that header recording area 31. However, for example, the CPU 90 could infer the situation that occurred to the vehicle from a sudden change in acceleration and display this information in the speech bubble 146 as information about the vehicle's status. For example, if the vehicle's longitudinal acceleration suddenly changes in the negative direction at a certain recording time, it can be assumed that emergency braking occurred. In such a case, the same speech bubble 146 as shown in Figure 22 could be displayed in the video when such acceleration occurs.

Furthermore, if the user specifies any location in the map image displayed in the map image display area 151 using the mouse (YES in step 126 of Figure 13), data representing the specified location is sent to the server, and data indicating the latitude and longitude of that location is sent from the server to the personal computer 80. When the personal computer 80 receives the data indicating the latitude and longitude, it finds a scene from the video displayed in the video display area 141 that was taken at the location closest to that latitude and longitude, and the video from that scene is displayed in the video display area 141 (step 127 of Figure 13). If no location in the map image is specified (NO in step 126 of Figure 13), the process in step 127 is skipped, and no video is displayed by specifying a location in the map image. Furthermore, while the video of the specified location on the map image is displayed in the video display area 141, the video displayed in the video display area 141 may be either video recorded by a drive recorder installed in the user's vehicle, or video recorded by a drive recorder installed in a third party's vehicle, or both. In this case, the server stores data representing the video recorded by the drive recorder installed in the third party's vehicle, and the video data recording the location specified by the user is read from the server and transmitted to the personal computer 80. If the server has a large amount of data representing video recorded by drive recorders installed in third parties' vehicles, it is preferable to allow the user to select which video to display on the personal computer 80, and then display the selected video on the display screen of the display device 92.

Furthermore, the CPU 90 determines whether vehicle-to-infrastructure communication or vehicle-to-vehicle communication information is recorded in the video file's header recording area 31 at the time the video displayed in the video display area 141 is recorded (Figure 13, step 128). If vehicle-to-infrastructure communication or vehicle-to-vehicle communication information is recorded (YES in Figure 13, step 128), that information is displayed in the video display area 141. If vehicle-to-infrastructure communication and vehicle-to-vehicle communication information are not recorded (NO in Figure 13, step 128), that information is not displayed in the video display area 141.

Figure 23 shows an example of information from vehicle-to-infrastructure communication (the same applies to vehicle-to-vehicle communication) being displayed in the video display area 141.

In the video display area 141, a second display mode, which differs from the first display mode of the aforementioned (see Figure 22) of the speech bubble 146, is displayed. This speech bubble 147 displays information about vehicle-to-infrastructure communication at the time the video displayed in the video display area 141 was recorded. In the example shown in Figure 23, the text "Vehicle on the left ahead" is displayed in speech bubble 147, indicating that a vehicle was present to the left in front of the vehicle at the time the video displayed in the video display area 141 was recorded. Data representing vehicle-to-infrastructure and vehicle-to-vehicle communication information is also recorded in the video file's header recording area 31, associated with the time data, and read from that header recording area 31. A user who obtains the vehicle-to-infrastructure and vehicle-to-vehicle communication information can then verify what vehicle operations were performed based on that information at the time the video was recorded (for example, whether the vehicle slowed down).

In the playback processing procedure shown in Figure 11, the following processes are performed in order: checking whether an object of attention exists outside the field of view (Figure 11, step 113), checking whether the user's facing direction has changed (Figure 11, step 113), checking whether the user's line of sight and the direction the user should be looking coincide (Figure 12, step 118), checking whether there has been a change in the movement of the subject (Figure 12, step 120), checking whether the orientation of the display device 92 has been changed (Figure 12, step 122), checking whether there is information regarding the vehicle's status (Figure 12, step 124), checking whether a location in the map image has been specified (Figure 13, step 126), and checking whether information from vehicle-to-vehicle communication or vehicle-to-infrastructure communication has been obtained (Figure 13, step 128). However, the order of these processes is arbitrary, and each process may be performed in parallel.

(3) Modified Example Figure 24 shows a portion of the playlist display area 171 included in the playback window 130.

In the above-described embodiment, the filename of the video file displayed in the playlist display area 171 uses the recording date and time of the video data. However, in this embodiment, instead of the recording date and time of the video data, the recording location of the video data (an example of a filming location) is used as the filename of the video file.

The location where the video data is recorded is used as the filename for the video file as follows. For example, when a video file is generated in drive recorder 1, the latitude and longitude of the location where it is generated are read by GPS receiver 22, and this latitude and longitude data is transmitted to the server via the internet from communication circuit 17. The server finds the location corresponding to these latitude and longitude values, receives data representing the found location via the internet, and uses the location obtained from the received data as the filename for the video file. Alternatively, the latitude and longitude data recorded in the header recording area 31 of the video file corresponding to the recorded date and time may be transmitted to the server via the internet during playback, the data representing the found location is received via the internet, and the location obtained from the received data is used as the filename for the video file. In Figure 24, the filenames are displayed in the order they were recorded. Since the location at the time of recording is included in the filename, it becomes immediately clear where the video was filmed.

Figure 25 also shows a portion of the playlist display area 171 included in the playback window 130.

In Figure 25, the filenames of video files containing less frequently occurring recording locations are displayed higher up in the playlist display area 171 (this is an example of prioritizing display). The filenames of video files containing more frequently occurring recording locations are displayed lower down. It is thought that more frequently occurring recording locations are those recorded in daily life, such as during commutes, while less frequently occurring recording locations are those recorded in non-daily life, such as during travel. To make it easier to play video files recorded in daily life, the filenames of video files containing less frequently occurring recording locations are displayed higher up.

Furthermore, videos from the same location may be displayed together in the playlist display area 171. This makes it easier to distinguish between video files from the same location and those from different locations. Additionally, the video file name only needs to include the recording location; the recording date and time may also be included in the video file name. Furthermore, a map image corresponding to the video file name displayed in the playlist display area 171 may be displayed, for example, in the video display area 141. Displaying a map makes it easier to identify the location of the video file compared to displaying only the name. For example, the video file name including the location may be transmitted from the communication circuit 98 to a map server on the Internet (not shown), the map data transmitted from the map server is received by the personal computer 80, and the map image is displayed in the video display area 141. Selecting the map image displayed in the video display area 141 may start playback of the corresponding video file.

Figure 26 shows an example of a frame structure based on MPEG (Moving Picture Experts Group).

In Figure 26, elements labeled "I" represent I-frames, and elements labeled "P" represent P-frames. One I-frame is followed by three P-frames, and these one I-frame and three P-frames constitute a GOP (Group of Picture).

As shown in Figure 8, step 77, when calculating the hash value, the section immediately before the P-frame, rather than immediately before the I-frame, is used as the unit for dividing the video data, and the controller 10 calculates a hash value for each unit. For example, the controller 10 calculates a hash value for the P-frame, the I-frame, and the three P-frames following the I-frame as a single unit. The hash value calculated in this way is then transmitted to the blockchain system.

Alternatively, instead of calculating a hash value from video data, when various events (e.g., sudden braking, skidding) occur in the drive recorder, the controller 10 may calculate a hash value (an example of a second calculated value uniquely determined from event data) from the data representing those events, and then transmit the calculated hash value to the blockchain system.

Figure 27 is a flowchart showing an example of the recording process procedure for drive recorder 1.

In this embodiment, the driver's (for example, a professional driver's) behavior during roll call is filmed and recorded. The recorded video data is stored in a video file, transmitted to a server for each driver, and stored on the server. Drive recorder 1 accesses the server and reads the video file containing the driver's roll call video data (step 181). The controller 10 of drive recorder 1 records the roll call video file onto the memory card 25 installed in drive recorder 1 (step 182).

Next, the driver's name and the date and time are used as the filename for the video file that stores the video data during driving, and the controller 10 generates the video file (step 183). The driver's name may be pre-stored in the memory 21 of the drive recorder 1, or it may be entered from the input device 20. From the time the engine of the vehicle to which the drive recorder 1 is installed starts, recording of video data during driving begins, and until recording ends (YES in step 185), the video data continues to be stored in the video file as described above.

In this way, video files containing video data from the roll call and video files containing video data from the driving session are recorded on the memory card 25. When such a memory card 25 is inserted into the personal computer 80, the video from the roll call and the video from the driving session can be displayed in the video display area 141 of the playback window 130. The video files containing the roll call data and the video files containing the driving session data are associated by the driver's name (this is an example of association).

In this embodiment, the drive recorder 1 stores the video file from the roll call and the video file from the driving session on the memory card 25. However, the video file from the roll call and the video file from the driving session may be stored on the memory card 25 during playback. For example, the memory card 25 containing the video file from the driving session is inserted into the personal computer 80, and the file names of the video files are displayed in the playlist display area 172. When any video file name is selected from the video file names displayed in the playlist display area 172, data representing the driver's name, recorded in the header recording area 31 of the video file corresponding to the selected video file name, is sent to the server recording the roll call video file, and the roll call video file is sent from that server to the personal computer 80. When the roll call video file is received by the personal computer 80, the video from the roll call, represented by the driver's roll call video file, and the video from the driving session, represented by the driver's driving video file, are displayed in the video display area 141. The video footage from the roll call and the video footage from the driving scene are preferably displayed simultaneously and side-by-side in the video display area 141, but this is not necessarily required.

Figures 28 and 29 are flowcharts illustrating an example of the recording process for drive recorder 1. Figure 28 shows an example of the recording process when drive recorder 1 is installed on a truck, and Figure 29 shows a flowchart of the recording process for drive recorder 1 installed on a forklift when the driver of the truck moves the cargo loaded on the truck using a forklift.

Referring to Figure 28, while the truck is in operation, video data is recorded by the drive recorder 1 attached to the truck onto the memory card 25 installed in the drive recorder 1 (step 191). When recording of video data to the memory card 25 is complete, data representing the truck driver's name and data representing the date and time are stored in the header recording area of the video file, and the video file with video data recorded in the video recording area of the video file is sent from the drive recorder 1 to the server (step 192). Data representing the driver's name is pre-stored in the memory 21 of the drive recorder 1. Furthermore, the drive recorder sends a command to the server to associate it with a video file from the forklift (step 193). For example, a command is sent from the drive recorder attached to the truck to the server to associate it with a video file that is sent from the same location as the last location (truck's parking location) represented by the video data stored in the video file sent from the truck's drive recorder 1, and which contains video data that was started by the drive recorder attached to the forklift within a few minutes of the truck's parking time.

Referring to Figure 29, when the drive recorder 1 attached to the forklift detects that a driver has approached the forklift, recording begins using the front camera 11 and rear camera 12 included in the drive recorder 1 (step 201).

The system analyzes the video footage captured by the front camera 11 or rear camera 12 and detects that the driver has boarded the forklift and that the forklift has started operating, or that the driver has boarded the forklift (YES in step 202). If this is detected, recording of the video data to the memory card 25 installed in the drive recorder 1 attached to the forklift begins (step 203). Once recording is complete, the video file is sent to the server (step 205).

On the server, if there are two video files where the recording end time and recording start time are approximately the same, and the location at the end of recording and the location at the start of recording are approximately the same, these video files will be treated as, for example, video files showing the same driver switching from a truck to a forklift, as described above. By sending these video files representing a series of videos to the personal computer 80, the video of the truck driving and the video of the forklift driving can be displayed in the video display area 141 of the playback window 130.

In the above embodiment, the driver operates the truck first, followed by the forklift. However, the reverse order—driving the forklift first, then the truck—may also be used to obtain separate video files for each operation. Alternatively, a series of video files could be obtained showing the loading of goods onto the truck using the forklift, then driving the truck to the destination, and finally unloading the goods at the destination using the forklift.

Figure 30 is a flowchart showing an example of the recording process procedure for a drive recorder 1 installed in a vehicle.

The drive recorder 1 is mounted on the upper inside of the vehicle's windshield, in a position to capture the vehicle's speedometer. When the vehicle's engine starts, recording begins with the front camera 11 and rear camera 12, and the rear camera 12 captures the vehicle's speedometer (step 211). The video of the vehicle's speedometer is analyzed by the controller 10 to determine whether the vehicle's speed has reached, for example, 30 km/h or higher (this is an example of a certain speed, but other speeds are also acceptable) (step 212). If the speed is 30 km/h or higher (YES in step 212), recording of video data to the memory card 25 begins (step 213). If the vehicle's speed falls below 30 km/h (YES in step 214), recording of video data to the memory card 25 stops (step 215).

Thus, when the vehicle speed exceeds 30 km/h, the captured video data is recorded to the memory card 25, and when the speed falls below 30 km/h, the recording of the captured video data to the memory card 25 is stopped. When the vehicle is being driven at a certain speed or higher, the video data obtained through recording is recorded to the memory card 25. In this embodiment, when the vehicle speed exceeds 30 km/h, the captured video data is recorded to the memory card 25, and when the speed falls below 30 km/h, the recording of the captured video data to the memory card 25 is stopped. However, it is also possible to record the captured video data to the memory card 25 when the vehicle speed exceeds 30 km/h, but not stop when the speed falls below 30 km/h, or to record the captured video data to the memory card 25 regardless of the vehicle speed, and stop when the speed falls below 30 km/h.

Figure 31 is a flowchart illustrating an example of the recording process procedure for a drive recorder installed in an automated guided vehicle (AGV).

The aforementioned drive recorder 1 is mounted on the automated guided vehicle (AGV). The AGV is equipped with a laser sensor that detects the approach of surrounding objects.

When the automated guided vehicle (AGV) engine starts (the AGV's power is turned on), power is supplied to the drive recorder 1. Then, the front camera 11 and rear camera 12 on the drive recorder 1 begin recording (step 221). When the laser sensor detects the approach of an object (an example of a surrounding object; it could also be another AGV) to the AGV (YES in step 222), recording of video data to the memory card 25 installed in the drive recorder 1 begins (step 223). If the laser sensor does not detect the approach of a surrounding object to the AGV (step NO), recording of video data to the memory card 25 does not begin. Even if a surrounding object accidentally comes into contact with the AGV, the incident can be recorded. Thus, the vehicles include not only those with a driver, but also those without a driver.

Figure 32 is a flowchart showing an example of the operational evaluation process.

The user drives the vehicle to which the drive recorder 1 is installed, removes the memory card 25 containing the recorded video data from the drive recorder 1, and inserts it into the personal computer 80. The playback window 130 shown in Figure 14 is displayed on the display screen of the display device 92, and the video data recorded on the memory card 25 is played back (step 231). If there is movement in the subject of the video being played back (YES in step 232), the video is analyzed, and a driving evaluation is performed based on that video (step 233). For example, if the brakes are applied suddenly when there is movement in the subject, or if other vehicles such as pedestrians, bicycles, motorcycles, or cars get too close, the driving evaluation will be low. Whether the brakes were applied suddenly can be determined from the acceleration data, and whether pedestrians or other vehicles are too close can be determined from the relative size of the pedestrians or other vehicles in the video. If there is no movement in the subject of the video (NO in step 232), no driving evaluation is performed for that part of the video. Steps 232 and 233 are repeated until the video ends. In video footage where the subject is stationary, the vehicle is likely stationary, making the driving evaluation largely meaningless. Since driving evaluations are not performed in such locations, the likelihood of unnecessary evaluations is reduced. Alternatively, driving evaluations could be performed only when the subject is moving, and not when the subject is stationary. Or, driving evaluations could be performed only when the vehicle is moving, and not when the vehicle is stationary.

Figure 33 is a flowchart showing an example of the playback processing procedure for video data recorded by drive recorder 1.

In this embodiment, the drive recorder 1 is installed in a vehicle that transports passengers, such as a bus, taxi, hired car, or train, but it does not necessarily have to be installed in such a vehicle.

The memory card 25, which was installed in drive recorder 1, is removed from drive recorder 1 and installed in personal computer 80. The playback window 130 shown in Figure 14 is displayed on the display screen 14 of personal computer 80, and the video data recorded on memory card 25 is played back (step 241).

The CPU 90 analyzes the video and determines whether or not a passenger is included in the video (step 242). For example, if there are people sitting in the seats of the vehicle excluding the driver, it can be determined that there is a passenger. If a passenger is included in the video (YES in step 242), the video is displayed in the video display area 141 (step 243). If a passenger is not included in the video (NO in step 242), the video display in the video display area 141 is stopped. Steps 242 and 243 are repeated until the video ends (NO in step 244). In the above embodiment, the video display in the video display area 141 is stopped when a passenger is not included in the video, but the video may be displayed, but the display speed may be increased, or the video may be skipped to the part that contains a passenger.

Figure 34 is a flowchart illustrating an example of the playback procedure for recorded video data. While recording by drive recorder 1 is preferred, recording by other recording devices is also acceptable. This playback process involves finding the desired scene within the video.

The memory card 25 containing the video data is inserted into the personal computer 80, and the playback window 130 is displayed on the display screen of the display device 92. When the user clicks the word "Tools" in the menu bar 131 of the playback window 130, a pull-down menu including the "Search" item appears. When the "Search" item is selected from the pull-down menu, a pop-up window for entering keywords appears on the playback window 130. The user enters keywords describing the situation of the video portion they want to find into this pop-up window using the input device 96 (step 251). The CPU 90 analyzes the video data and finds video footage related to the entered keywords (step 258). The video footage of the found scene is displayed in the video display area 141 (step 254). The keywords describing the situation of the video portion entered by the user do not have to be clear, such as "video containing a Shinkansen," but can also be vague, such as "bright video." The video portion corresponding to the vague keyword is found. For example, if the keyword is "bright video," the video portion with relatively high brightness is found. Furthermore, keywords describing the vehicle's condition at the time of video recording can be entered to find the corresponding video footage. For example, keywords such as the vehicle's state when the drive recorder was installed, speeding, sudden acceleration, or turn signal activation can be used to find the corresponding video portion. Since the data describing the vehicle's condition is recorded along with the recording time in the header recording area 31 of the video file containing the video data, this data can be used to find the video portion. Alternatively, keywords that search for footage after a dynamic movement, such as a red car overtaking a blue car, can be entered to find the corresponding video portion.

Figure 35 shows a flowchart illustrating an example of the playback procedure for video data recorded by Tribe Recorder 1.

The memory card 25, which was installed in drive recorder 1, is removed from drive recorder 1 and installed in personal computer 80. The playback window 130 shown in Figure 14 is displayed on the display screen 92 of personal computer 80.

The CPU 90 of the personal computer 80 detects the vehicle's movement range by reading data indicating the latitude and longitude of the vehicle recorded in the header recording area 31 of the video file stored on the memory card 25 (step 261). A request command for map image data including the detected movement range is sent from the personal computer 80 to the server, and the map image data is sent from the server to the personal computer 80. When the map image data is received by the personal computer 80, the map image represented by the received image data is displayed in the video display area 141 (step 262). Furthermore, data representing the vehicle's operation information at the time of video data recording is read from the header recording area 31 of the video file, and the vehicle's operation status at each location is displayed on the map image displayed in the video display area 141 (step 264).

Figure 35 shows an example of a map image displayed in the video display area 141.

In the map image, the vehicle's movement is indicated by a dashed line 148. This dashed line 148, representing the movement, is obtained from data indicating the vehicle's position recorded in the header recording area 31. Furthermore, within the movement indicated by the dashed line, the vehicle's operation status is displayed by a callout 146. By viewing the map image, the user can relatively easily understand the vehicle's movement range and, at what locations, what kind of vehicle operations were performed during that movement.

Figure 37 is a flowchart showing an example of the playback processing procedure for video data recorded by the drive recorder 1. In this embodiment, video footage obtained from the user's driving and video footage obtained from a third party's driving are displayed in the video display area 141.

The memory card 25 installed in the vehicle being driven by the user is removed, and the removed memory card 25 is installed in the personal computer 80. Video data obtained from the user's driving of the vehicle is read from the memory card 25 and played back (step 271). Furthermore, data indicating the recording location (vehicle movement range) of the video data obtained from the user's driving (data indicating latitude and longitude) is read from the video file's header recording area 31 and sent to a server where numerous video files obtained from third-party driving are stored. The server finds a video file containing video data obtained from driving in the same location as the user and sends it to the personal computer 80. It is sufficient to find a third-party video file that closely matches, even if it does not perfectly match, the user's direction and movement range. The found third-party video file is received by the personal computer 80 (step 273), and the video obtained from the user's driving and the video obtained from the third-party driving are displayed in the video display area 141 (step 274).

Figure 38 shows an example of the playback window 130.

The video display area 141 is divided into two video display areas 141A and 141B, left and right (or up and down). The left video display area 141A displays video obtained by the user's driving (an example of video based on processed video data), and the right video display area 141B displays video obtained by a third party's driving, represented by a video file found on the server. In video display area 141A, a speech bubble 146A is displayed indicating the user's vehicle operation information at the time the video displayed in video display area 141A was recorded. It goes without saying that the operation information displayed in speech bubble 146A is obtained from acceleration data, etc., recorded in the header recording area 31 of the video file where the video data is stored. Similarly, a speech bubble 146B indicating the third party's vehicle operation information at the time the video displayed in video display area 141B was recorded is also displayed in video display area 141B. It is preferable that the CPU 90 performs playback control so that the position indicated by the image displayed in image display area 141A and the position indicated by the image displayed in image display area 141B are approximately coincident.

The user can compare the video of their driving displayed in the video display area 141A with the video of a third party's driving displayed in the video display area 141B, allowing them to improve their driving skills by comparing their own driving with that of others. In particular, the speech bubbles 146A and 146B allow users to see the similarities and differences in their operations, making it possible to determine whether their driving is consistent with or different from that of others.

Furthermore, the system may compare the user's driving with that of a third party and display information representing the evaluation of the user's driving (an example of information based on a comparison between the user's driving and that of a third party) in the video display area 141. For example, if the user's driving skills are evaluated as worse than those of a third party, a warning message such as "Improve your driving skills by referring to the third party's driving" may be displayed. Conversely, if the user's driving skills are evaluated as better than those of a third party, information indicating a positive evaluation, such as "Your driving skills appear to be high," may be displayed.

Figure 39 shows an example of the playback window 130.

When recording the video displayed in the video display area 141, alphabetical characters indicating the areas and order in which the vehicle driver should look are superimposed on the video. The example shown in Figure 39 is, for example, video of a lane change, and it is preferable to view the areas indicated by the letters in alphabetical order. When recording the video data, the driver's gaze is detected as described above, and the CPU 90 detects whether the driver's gaze is moving through the areas indicated by the letters in alphabetical order. If the driver's gaze is moving through the areas indicated by the letters in alphabetical order, for example, the string "Ideal driving." is displayed in the video display area 141. The user can confirm that the driving skills are not poor. If the driver's gaze is not moving through the areas indicated by the letters, a string such as "There are areas that have not been checked." is displayed in the video display area 141 to inform the user that there were areas that were not checked. Also, if the driver's gaze is not moving in alphabetical order, a string such as "The checking order is not ideal." is displayed in the video display area 141 to inform the user that the checking order is not good. Users can learn whether their eye movements while driving are appropriate.

Furthermore, in the example shown in Figure 19, the area 142 outside the field of view was blurred. However, it is also possible to detect areas that the user is not looking at within a certain period of time and blur those areas. For example, areas that the user is not looking at from the start to the end of a right or left turn could be detected and blurred. It goes without saying that the driver's line of sight should be detected when recording the video data, and this line of sight data should be used to detect areas that the user is not looking at. For example, if an area that is not being looked at is detected, it can be blurred or made semi-transparent, similar to the area 142 outside the field of view, as shown in Figure 19.

Furthermore, when playing back video data recorded by drive recorder 1, the CPU 90 may perform correction (an example of processing) to match the driver's memory of the colors. To correct to the memory colors, the contrast, tone, and saturation of the resulting video are adjusted. The degree of adjustment for each of these can be predetermined, or several preset adjustment values can be defined and selected during playback.

Furthermore, a communication robot may be used in conjunction with the personal computer 80 during video playback.

Figure 40 shows an example of a communication robot 280 placed near a personal computer 80.

The communication robot 280 includes a body 281 and a head 282. An eye camera is provided inside the eye 283 of the head 282, and a speaker is provided in the mouth 284. The head 282 is rotatable clockwise and counterclockwise relative to the body 281. The communication robot 280 is positioned to the right of the personal computer 80. The user sits in front of the personal computer 80. Thus, the positions of the personal computer 80, the communication robot 280, and the user are predetermined. The communication robot 280 is controlled to face the display device 92 when displaying the playback window 130 on the display screen 92 of the personal computer 80 and outputting audio comments about the video displayed in the video display area 141, and to face the user when outputting audio comments to the user viewing the video.

Figure 41 is a block diagram showing the electrical configuration of the communication robot 280.

The communication robot 280 includes a CPU 291 that oversees the overall operation, a memory 292 for storing data, a wireless communication circuit 294 for communicating with the personal computer 80, drive recorder 1, etc., and a wireless LAN (local area network) board 293 connected to the wireless communication circuit 294. The communication robot 280 also includes a head motor 296 for rotating the head 282, an eye camera 298, a speaker 300, and a microphone 301. The head motor 296 is controlled by a motor control board 295, and video data captured by the eye camera 298 is received by the communication robot 280 via a sensor board 297. Furthermore, audio output from the speaker 300 and audio input from the microphone 301 are controlled by an audio input/output board 299.

Figure 42 is a flowchart showing the processing procedure of the communication robot 280. It goes without saying that the operation program of the communication robot 280 is stored in memory 292.

The display screen of the personal computer 80's display device 92 shows the playback window 130 described above, and the video recorded by the drive recorder 1 is displayed in the video display area of the playback window 130. The personal computer 80 and the communication robot 280 communicate, and acceleration data is transmitted from the personal computer 80 to the communication robot 280 (other data, such as data recorded in the video file header recording area 31 and event data, may also be transmitted from the personal computer 80 to the communication robot 280. It is sufficient that data is transmitted from the personal computer 80 to the communication robot 280 that allows the communication robot 280 to provide advice to the user, etc., after viewing the video).

Based on the acceleration data transmitted to the communication robot 280, if the CPU 291 of the communication robot 280 determines that there was a sudden change in acceleration in the longitudinal direction of the vehicle to which the drive recorder 1, which records the displayed video, is attached (step 311), it is considered that the vehicle's brakes were applied suddenly. Then, as shown in Figure 40, the communication robot 280, which was facing the user, rotates its head 282 90 degrees clockwise (step 312) to face the display device 92 of the personal computer 80. The display screen of the display device 92 shows the video of the sudden braking incident, and when the communication robot 280 faces the display device 92, an audio message indicating that it was a dangerous driving situation is output from the speaker 300 (step 313). For example, the message "This kind of driving is dangerous" is output. The audio data is stored in memory 292.

Subsequently, the head 282 of the communication robot 280 is rotated 90 degrees counterclockwise (step 314), and the communication robot 280 faces the user. Once the communication robot 280 faces the user, a voice message is output from the speaker 300 warning the user not to engage in dangerous driving like the scene shown on the display screen of the display device 92 (step 315). For example, the message "Let's drive safely" might be output. Because the warning comes from the communication robot 280 and not from a human instructor, the user is more likely to heed the advice.

Figure 43 shows the communication robot 280 described above placed on the dashboard 311 inside the vehicle.

As described above, the drive recorder 1 is mounted on the upper part of the windshield 310. The drive recorder 1 and the communication robot 280 communicate, and as described above, when the drive recorder 1 recognizes that the vehicle is in autonomous driving mode, it sends an operation start command to the communication robot 280. The communication robot 280 then operates and begins communicating with the vehicle's driver or other relevant party.

Furthermore, when playing back video data recorded by the drive recorder 1, the CPU 90 may analyze the video data for road information (for example, information such as the road narrowing or being a dead end) of the vehicle to which the drive recorder 1 is installed. The display of the video in the video display area 141 may be changed according to the analysis results. For example, a string indicating the detected road information may be superimposed on the video, or some of the hue, brightness, and saturation of the entire or a portion of the video may be adjusted. Since the video displayed in the video display area 141 changes, it is possible to know that there has been a change in the road conditions.

Furthermore, event recording trigger information may be recorded in the video file header recording area 31. For example, information about turn signal operation, sudden braking operation, and wiper operation may be recorded as event recording trigger information. Additionally, while the drive recorder 1 is recording, the user may sit in the passenger seat and wear a glasses-type display device on their head. The display screen of this device shows driving caution information, and the user can see the scenery in front of the vehicle while driving (an example of the actual scenery visible from a moving car) through the display screen. The driving caution information is displayed superimposed on the scenery in front of the vehicle. It goes without saying that if the road to be driven is predetermined, caution information for that road will be displayed on the display device. Furthermore, a third party other than the user (for example, a driving instructor) may ride in the vehicle and select the caution information to be displayed on the display device, and the selected caution information will be displayed on the user's display device.

In the above embodiment, the drive recorder 1 was capable of recording in all directions, including the front, sides, and rear of the vehicle. However, it does not necessarily have to be capable of recording in all directions.

Furthermore, in addition to the normal continuous recording and event recording, the drive recorder 1 may also perform other types of recording. For example, for event recording, the accelerometer and trigger switch may record before and after the event. This is kept separately from the continuous recording as event recording. The 30 seconds before and after the accelerometer reaction (1 minute) may also be recorded. For example, if the first event occurs, then the second event occurs, for example, if the car hits the guardrail the first time, is thrown off balance, and then hits the guardrail a second time, the recording may be made. If the recording times overlap, they may be treated as a single event. If there is a gap between events, they may be recorded as separate events, or if they do not overlap, they may be processed as two separate events.

Furthermore, the aforementioned drive recorder 1 may be used on a factory production line to ensure smooth operation, monitor the line, and adjust it using cameras. When assembling with machine tools, cameras may be temporarily installed at points where problems are likely to occur to capture images of the assembly process. When setting up a new production line in a factory, cameras may be temporarily installed for about three months to verify malfunctions. Recording may be done before and after an alert is triggered by a machine tool, or the system may be configured to capture images before and after triggers such as green, yellow, or red lights on the machine tool. The video data obtained from such cameras may be displayed in the playback window 130 mentioned above. Recording may be limited to events, or continuous recording may be performed.

Even with 360-degree drive recorders (drive recorders that record in all directions) and wide-angle camera drive recorders, the range of human vision is not the same as the camera's imaging range, and the visible range varies depending on a person's age and individual differences. During senior driver training and license renewal, a narrow field of view is dangerous and often leads to hitting bumps. Therefore, it would be beneficial to record driving footage using a drive recorder (Dash Recorder 1) and show instances of hitting bumps during training. Displaying the visible range of the driver's seat doesn't criticize the driver; rather, it allows for training that encourages drivers to look properly and turn their heads to see. For example, displaying a frame during playback would help show the visible range.

When recording driving conditions (scenery, landscape) with drive recorder 1, the driver's eye movements can be recorded simultaneously, and if the field of view is shifted, a vector or circle indicating where the driver is looking within the field of view can be displayed. Recording where the driver's gaze was directed while driving and displaying it in the playback window 130 should be adjusted so that the displayed range and frame are shifted in the direction the driver is looking. If drive recorder 1 can capture a full 360-degree (omnidirectional) image, the entire area in front of, to the side of, and behind the vehicle is recorded, so the direction the driver is facing can be determined by image recognition. Accordingly, the displayed frame should be moved. In addition to the present, the playback window 130 should display not only the present but also the extent to which the driver was checking things in the present, past, and future, showing things like whether they were looking straight ahead now but not in the past. For example, the areas that were not being looked at could be filled in. This would show areas that were not being looked at at all. It would also show how far the head was turned, the line of sight, and how far the driver was looking. Alternatively, drive recorder 1 can be connected to a gaze detection sensor, and the line of sight captured by the sensor can be displayed. It can identify areas the user is not looking at. The imaging time changes depending on the driving situation, such as normal driving, high-speed driving, and driving at intersections. For example, the time spent driving in a driving school, or the time before and after stopping at an intersection. The viewing range is predetermined at each point in the driving school, and the system judges how well the user's actions match that range. It would be good to have a standard, such as a minimum percentage of viewing required to pass. Even in normal driving, it would be good to judge how much the user looks when stopped at an intersection, or how much they look at different speeds, such as between 40 km/h and 80 km/h, or above 80 km/h. At high speeds, you don't look to the sides, but you do look in the rearview mirror, while in city driving, you must also pay attention to the sides. When driving slowly at 20 km/h or less, you must look 360 degrees, checking mirrors and blind spots while driving. The viewing range changes based on the turn signal, brakes, and steering angle. If you are turning to the right, you must look to the right relatively accurately. By using the playback window 130, it is possible to see, for example, whether the user checked before activating the turn signal or before turning the steering wheel. Furthermore, during senior driver training, it would be beneficial to have the elderly driver actually drive within the driving school grounds, attach a dashcam, and then show the footage later in the playback window 130 showing how they were driving. A simulator would also work, but elderly drivers won't be convinced unless they actually drive, so showing them actual footage allows them to understand. This has two effects: it helps the driver understand and it's easier for the instructor to understand. It would also be good to provide information for the instructor. The areas that the driver isn't looking at are blacked out, which can be interpreted as information for the instructor. For example, if the driver isn't looking at the mirrors, you could have them pre-determine their focus points, such as mirrors, and check if they are looking at the corresponding instruments (speedometer, etc.). You could also record the driving behavior when the driver is heavily intoxicated and wearing glasses using dashcam 1 and show them that footage.

Since buses have over a dozen mirrors, it would be good to evaluate whether the driver is looking at all of them, in addition to the tachograph, by also evaluating their visual perception. Displaying their line of sight (gaze) would allow for an assessment of what percentage of their driving time they are looking at a particular spot. It would also be possible to evaluate where they are looking and where they are not, before and after using the turn signal. Buses and other commercial vehicles always have routines; there are manuals for checking before bus stops and before departure, and forklifts also require checking in all directions (front, back, left, and right). Therefore, it would be possible to evaluate whether drivers are operating buses, trains, trucks, and forklifts according to their usual routines. Passenger movements could also be shown.

When playback is in progress, the system records the viewing state and eye movement history. By using triggers such as activating a turn signal, and analyzing the moments before and after, it can determine whether the driver is properly checking blind spots. In driving, it's necessary to look at the right place at the right time, which differs from normal eye-tracking analysis. In the video, the car is moving, and the points of focus are predetermined before triggers, turn signals, and turning the steering wheel. The conditions change over time. Sometimes the timing of the look is delayed. It's important that the analysis is appropriate in relation to time. There are places that should be looked at. The system shows what should be looked at and provides a guiding example to show whether the user was looking at the bottom. If approaching an intersection, look here; if turning, look here. The system considers the type of trigger, what to use as a trigger, speed, turn signal, location (intersection), red light, road sign, checking left and right at a crosswalk, looking behind when backing up, not backing up while looking ahead, checking mirrors, only looking at the backup monitor, looking left and right. It checks whether the driver is checking these things, and it's strange to back up when the eyes are directed forward, or driving while looking behind. The driver's gaze direction can be determined from the video footage. It may also detect whether the driver's eyes are open and display this information during playback. Scenes of poor driving could be shown as thumbnails, similar to film reels. For example, since there are specific actions that the driver should be performing, such as not checking mirrors during lane changes, or not using turn signals or the accelerator, the system could extract and display thumbnails of instances where these actions are not being followed.

When scoring driving, stationary sections may be excluded from scoring. Scoring should focus on smooth movement, stopping, and avoiding sudden steering or braking. Evaluation should include situations where buildings are moving, the car in front is stationary, and the extent of movement in specific areas. Stationary sections should not be evaluated. Stationary sections should not be used in the evaluation. Stationary sections can be skipped, and the video can be fast-forwarded to the next point. The video can also be advanced to the point where the traffic light changes from red to green. This saves time for the scorer when reviewing the video. In addition to the exterior view, the interior should only show footage when a passenger gets in. Playing only when there is a change in the passenger would create a special feature focusing on passenger moments. For taxis, it's sufficient to show the passenger's face.

Also, when searching, if you specify that it was bright, you can find bright footage; if you specify that there was a bullet train, you can find footage of the bullet train and pick out the footage of the bullet train. In this way, it is good to pick out the footage that matches the conditions from a general overview. When you have a week's worth of footage and someone gives you vague information, you will be able to find it. When you want to show something in a presentation, you can find it if it was around that area. You can also search for something like, "I want to show footage of the lighting conditions in a supermarket parking lot." It is good to use keywords such as whether it is a specific object, the situation, the time, the state of the car that was recording, speeding, sudden acceleration, or using the turn signal to find it. After a red car overtakes a blue car,
To find the footage following a dynamic movement, it's best to locate the desired scene within the video by looking for it after this sequence of scenes.

The system could display the actual image at the top and the recognized image at the bottom, or it could identify the vehicle type from the image or show its outline. Because of the ambiguity of "such a scene," it could present a list of desired video scenes for the user to select. Each image could be tagged with information such as "what was here," and "how long was this there?" It could also check if something was stopped for more than 5 minutes, or the video playback software could include voice recognition functionality.

During playback, it would be good to display where the driver was looking. If it is determined that the driver is looking away, a mark can be placed in the playback window 130, or the image can be turned red the moment the driver looks away. This is because a 360-degree camera captures the image and shows where the driver is looking. If the driver is looking away, an alarm may sound. An alarm may also sound if the driver closes their eyes. The system can skip playback to that section or play only that section. If the camera is not a wearable camera like a 360-degree camera, the direction the user is looking can be determined and displayed. A wearable camera is also acceptable. Instead of a wearable camera, a wearable sensor can be used, such as one placed on the head or attached to the user using something like a headband. A sensor that can determine where the driver is facing would be useful. An accelerometer (camera) could be attached to the ear, or a sensor (camera) could be attached to a woman's hair clip. A sensor (camera) could be clipped to the brim of a hat. The sensors and 360-degree images can be combined to show which direction the user is facing. Analysis can be performed using the 360-degree images. The system can also indicate that there is a framed area on the opposite side, or on the back side, that shows the field of view. It can also indicate that the frame is displayed, but it's not currently visible. The outer frame could be green, or the visible area could be clearly displayed, while the rest is grayed out. The clear area could expand as the user checks it. The unseen area would be immediately apparent. The area could be cleared during periods of no head movement. The visible and unseen areas could be displayed only before and after turning the steering wheel, or before and after using the turn signal. The camera's angle could be displayed on the map below. Both the line of sight angle and the car's angle could be displayed. The angle of the video itself could be displayed. The extent of visibility could be shown based on the car's direction and the user's line of sight. The current field of view could be displayed, or the field of view of the displayed area could be displayed.

A 360-degree camera drive recorder may be equipped with VR (virtual reality) display capabilities. Since the 360-degree camera captures the entire 360 degrees, it can produce an image similar to VR on a flat surface. In this case, the reference direction may be the front camera side, the direction of shooting, or north. It is necessary to determine how much of the VR field of view should be mapped to the direction with north as the reference direction. In the case of drive recorder 1, the forward direction is forward. It may also be a playback device that shows the direction in which the main unit is moved. This range is the range that the user can see, but it can be shown that a person 10 years younger can see more. A comparison function may also be included. It can be seen that what you can see now will be reduced to this extent in 10 years or 30 years. A function may be provided that allows instructors to switch the angle range for each age group. It may also be possible to switch by age group. VR display may be based on the direction the driver can see. Playback software for the video recorded by drive recorder 1 that can be displayed on a VR device may be provided. VR devices can display acceleration data as an overlay or as an OSD (On-Screen Device). Acceleration data can also be displayed in 3D. While the image itself cannot be 3D, it can be displayed semi-transparently in front of the visible surface within the sphere. Panels can be placed in areas that are not needed. This can be understood simply by moving the head. Where the gaze is directed should be in front, and overlay-like elements should be in front. The intensity can change depending on the viewing time, or the intensity can change depending on the frequency of viewing. When the user turns sideways, they can see the side image, and when they look up, they can see the image above. The direction the user is facing can be detected using an accelerometer, geomagnetic sensor, and gyroscope. The direction straight ahead can be determined from the fixed camera. The center of the sphere should be considered the front. The camera is always facing forward. Image data can be provided to the framework.

From the perspective of an administrator, there's no point in watching footage that doesn't change while waiting at a traffic light or stopped. It would be good to have a function that automatically recognizes and skips such footage in the playback software (program).

On a map, the vehicle's movement path could be displayed, and a function could be added to jump to a specified point on that path. The time of day corresponding to a point on the map would be displayed, and the time would appear when the cursor is moved over, allowing the user to find the corresponding video. Street View could also be displayed. A function to switch between Street View footage and dashcam footage could be added. Since Street View includes both past and present footage, it's possible to see what the scenery looked like a year ago. Footage from three years ago of a driven route could be viewed. Footage before and after buildings were constructed could also be seen. Displaying this in a separate window would be beneficial. It would also be good to be able to switch to older maps. If the shooting date is included, it could be possible to display footage of a desired season, such as autumn or spring. If the user drives the same route during their commute, summer and winter footage could be displayed. The current footage of the driving route could also be displayed. On rainy days, for example, the scenery could be shown.

The playlist display area 172 of the playback window 130 may display information indicating the location of the video. For example, it could display information indicating that the video traveled from Okazaki City to Toyota City. This information could be displayed as text or as a zoomed-out map. This would make it much easier to understand and use. If municipal and area data is available, it would be possible to display information such as "from City XX to City YY". By using the Location API (Application Programming Interface) and sending latitude and longitude to a search server, information such as "the location is City XX" would be returned, and this information could then be output. The destination of the video file could be automatically assigned. When videos are displayed side-by-side, if you commute by car every day and don't care about the commute videos, and only want to see travel videos, categorizing the videos from here to here as "commuter" would make selection easier. For example, in the current City XX, it would be good to categorize videos where the starting and ending points are the same, bringing the rarer ones to the top. This is because videos of places visited only once or twice are more likely to be travel videos.

The video file header recording area 31 stores GPS data periodically, such as frame by frame, every second, or every 0.5 seconds. To name the video file after a specific place name, such as "Higashi-Okazaki," the coordinates are sent to the location search server's API, and the information returned from the server is used to create the video file name. GPS data can be sent to a server that stores map images, and the map can be retrieved and displayed from that server. If the personal computer 80 has map data and place name information, there is no need to send it to an external server. Event information can also be displayed. Data consisting only of events can be displayed in the playback window 130, showing information such as "when you traveled this road, events occurred at this scene, at this scene." Where sudden braking occurred, etc., can also be shown on the map. For a commute route, events that occurred and changed at each point can be grouped together and displayed. The road can be displayed, and information such as "when you traveled this road, these events occurred" can be overlaid on the map. For example, if you tap on a road on a map, such as National Route 248, events that occurred while traveling on National Route 248 over a certain period, say five years, will be displayed on the map, showing that that area is particularly dangerous. All logs will appear, and clicking will display the logs, heart rate data, and it could be categorized, for example, by cycling. It could also identify and display locations where minor incidents occurred, where the accelerator was pressed hard, or other accident-prone areas. It could also display categorized video files of commutes, and clicking on them would map them all onto the map.

Furthermore, a driving evaluation could be performed at regular intervals, for example, from this intersection to that intersection, and automatically saved to the server. When playback is initiated, other users' evaluation data would also be accumulated for each section. Therefore, it would be beneficial if the evaluation—whether it's reckless or safe—could be displayed in the playback window 130 during playback, allowing users to compare their own driving to that of others. Below the video, a checkmark or a crying face could appear depending on the evaluation, or a score could be displayed, or the average score of other users could be shown.

Furthermore, the video footage could be saved to the server at regular intervals, allowing for comparison of the same location. For example, a comparison could be made between a specific point on a road, such as National Route 1. Driving scores are assigned using acceleration sensors, speed sensors, gyroscopes, and braking technique, but the question remains: is that score truly acceptable or unacceptable for that particular road? On winding roads, the steering wheel is naturally turned multiple times, so a score indicating excessive weaving might not be accurate, as it's due to the curves of the road. Instead of absolute evaluation, relative evaluations could be used, or footage of other drivers could be displayed. The system could show the video of the top-ranked driver, indicate that the user is ranked 30th, and present examples of excellent driving. While there's the question of whether it's acceptable to display the top-ranked video, a permission function could be provided. The system could display a message asking if it's okay to share this, or if it should be made available to others, or include an upload button. The camera (drive recorder 1) itself could have communication capabilities to upload or transmit score data. You can confirm the upload details when installing the playback software on the Personal Computer 80, or you can do it all at once during installation, or you can allow users to upload each video file individually, or you can have an permission button appear the moment they attempt to upload. You can confirm the video files before uploading, but upload the scores without confirmation. You can also have a rule that you will not link users to information about their location. You can delete date data when uploading. Day of the week and time are acceptable, but personally identifiable information should be removed. If only location and time are included, without vehicle type or personal name, it should be acceptable as long as the data does not belong to the user.

It would be good to be able to specify the resolution for each location: roads with no obstacles, such as farm roads, are less likely to have accidents, so they will be filmed (recorded) but at a lower resolution, while roads in urban areas, where there is a risk of sudden appearances, will be filmed (recorded) at a higher resolution. It would be good to be able to specify that the range from here to here should be recorded at a low resolution, and this area at a high resolution. This will allow for longer recording times. The location is specified using the playback window 130. The user can specify in advance the range that is acceptable to record. It would be good to circle this range on the map and set the playback window 130 to film (record) at a low resolution when entering this range. Insert the memory card 25 with the recording settings into the drive recorder 1. Then it will operate accordingly. If the drive recorder 1 has an LCD display, the settings can also be made on the drive recorder 1. You can also adjust the resolution here when parking surveillance. You can also change the resolution while driving, because there is a place where absolutely nothing will happen. You can just film (record) in detail using event recording. If all the data is stored on the cloud (server) side, it's possible to know whether an event will occur or not in a given location, allowing for resolution adjustments on the cloud side. Low-probability areas can be recorded at low resolution (frame rate), while high-probability areas can be recorded at high resolution. Since it's an event-based switching, it's best to record at high resolution when an event occurs. You could also record at high resolution at your destination and at low resolution during the journey there. You don't need to record while driving on a highway. You can also record only when an event occurs (event recording). For example, if you're going to Kyoto, you could record in high resolution within Kyoto city and stop recording once you leave Kyoto. You could also record at one frame per second or at a low frame rate. You can pre-set locations where you want to upload videos to video sharing sites. For example, if you go to a racetrack, you could record the racetrack itself at high resolution and not record the journey there. You can also specify areas to record at a high frame rate (high quality) instead of areas to record at a low frame rate (low quality). When using parking surveillance, you can record at a low frame rate while parked.

For example, if you decide to record in high resolution when going to Kyoto, it's a good idea to record the journey there and back as well. Not recording these areas defeats the purpose of a dashcam, and even if you try to set it after arriving in Kyoto, you can't because you're driving. For instance, you could record in high resolution only near Kiyomizu-dera Temple, but this prevents you from forgetting to set it. Also, it's a good idea to record in high resolution while driving on mountain passes. You could record in high resolution on highways and low resolution on regular roads. You could also record in high resolution on the Metropolitan Expressway and not record on regular highways, or you could set it to record when the speed exceeds 100 km/h, or when acceleration occurs in corners. Area settings can be controlled by a slide switch on the dashcam to select between high and low resolution.

A 760MHz ITS service system, providing advanced road information, has been operating as an experimental service since around 2016. It would be beneficial if this information could be recorded together with the information from drive recorder 1. The information includes, for example, other vehicles in the vicinity of the vehicle (position, direction of movement, acceleration, etc.), traffic lights (how many meters away, time until they turn red, etc.), road signs (how many meters away, speed limit, etc.), pedestrians (position, direction of movement, etc.), emergency vehicle approach notifications, sudden braking warnings, traffic congestion warnings, malfunctioning signal vehicle warnings, sign notifications, motorcycle approach notifications, signal timing notifications, construction information notifications, and weather warnings. ITS has both vehicle-to-infrastructure communication and vehicle-to-vehicle communication. Vehicle-to-infrastructure communication is communication between the road and vehicles, and vehicle-to-vehicle communication is communication between vehicles. In vehicle-to-infrastructure communication, information such as vehicle number information and information obtained from sensors of surrounding pedestrians is transmitted. In vehicle-to-vehicle communication, information such as the direction the vehicle is going and information about the current accelerator status is transmitted. For example, if vehicle-to-infrastructure data is received and saved along with the video, then, for instance, if an LED traffic light is flashing, and the frame rate of the camera coincides with the moment the flashing stops, the traffic light may not appear in the video at all, and in some cases, it may remain completely dark. Even if the traffic light is not visible, the ITS information can still record the traffic light's status as data. If the ITS information indicates that the signal is green, then this information can be displayed at the edge of the playback window 130. Since this information is now being provided using a dedicated frequency, it can be received by a receiver, recorded along with the drive recorder's video, and displayed in the playback window 130.

It would be good to record ITS service information with drive recorder 1 and display that information. It would be good if drive recorder 1 could record information such as a car coming from the side, a bicycle coming, crossing on a red light, crossing on a yellow light, or not braking before the yellow light onto memory card 25. It would be good if drive recorder 1 could receive and record information about other vehicles in the vicinity of your vehicle (position, direction of movement, acceleration), how many meters ahead there is a traffic light, how many meters ahead the light will turn red, the type of road sign, and also detect and transmit pedestrians. It would be good if drive recorder 1 could record information about approaching emergency vehicles, and information that is transmitted between vehicles via a bucket brigade when the car in front brakes suddenly, on memory card 25. It would also be good if it could record information about how many cars ahead braked suddenly, as this information can be known in advance. In the case of vehicle-to-infrastructure traffic, this information is transmitted from antennas near intersections. When you enter an area of about 5 meters from that antenna, Drive Recorder 1 will automatically receive information. Between vehicles, vehicles that can receive information from the road can also transmit their own information. This includes things like current speed in kilometers per hour, turning left based on the car navigation system, and whether the brakes were applied. When receiving information between vehicles and the road, the drive recorder receives it. The drive recorder also receives information between vehicles, and can receive information transmitted by the car next to it because it uses the same frequency. It is best to have the drive recorder only receive and not transmit. The ITS device is a transceiver, and Drive Recorder 1 records the communication between the car with the drive recorder and other cars. It simply intercepts the information. You could also install software on the drive recorder to transmit the received information to other cars.

This system would be useful in situations where there's a building at an intersection and a car approaches from a blind spot. It would communicate with another vehicle about the car's behavior, including whether it's braking, to prevent accidents. It could also use cameras at the intersection to alert drivers of pedestrians. Since the system is designed to encourage caution when turning, knowing if a driver is braking when a hidden vehicle is present would make management easier. It would also be beneficial to record instances where a driver failed to brake despite a pedestrian being detected, even if the vehicle had alerted the system. Furthermore, if a driver doesn't slow down despite a red light (which won't turn green for 10 seconds) being displayed on the car's screen, it would be helpful to receive feedback indicating poor fuel efficiency.

In the playback window 130, information about the vehicle to which drive recorder 1 is installed is displayed at the bottom. For example, speed and turn signal operation status. It would be beneficial to display information obtained from the ITS system separately from this information.

For example, since we know there's a car a certain number of meters ahead to the right, we could use an overlay to display the time remaining until the traffic light appears the moment the car is detected, if there's a traffic light a certain number of meters ahead. This could also be displayed as text. It could be displayed as text or an icon. The callout is just one example of an icon.

Since dashcam footage is used as objective evidence of the accident, it is desirable that it cannot be manipulated. It is also desirable that any tampering be detectable. Furthermore, the footage recorded by dashcam 1 is not a recording of the scenery as remembered by the driver, but rather objective footage recorded in accordance with the use of the cameras included in dashcam 1 (front camera 11, rear camera 12) at that location. This is because the installation location of dashcam 1 is different from the driver's line of sight, and the recording system of dashcam 1 differs from the human visual system. The differences between a dashcam's recording system and the human eye's imaging system include, of course, differences in field of view, sensitivity, head movement, eye movement, and optical system, as well as differences in the imaging sensor and the structure of the eye. However, the most significant difference lies in the cognitive structure of the brain. Therefore, it would be beneficial to obtain the image remembered by the driver and equip the dashcam (Dashcam 1) with a function that produces an image as close to that as possible. Based on the differences between the objective image recorded by Dashcam 1 and the recorded image, the driver can be educated or their cognitive distortions corrected.

It would be beneficial to allow users to manipulate the colors and other aspects of the video recorded by dashcam 1 to match their intended image. The video should be processed to reflect the objective perception, displaying an image that closely resembles the driver's actual experience.

To detect tampering, it is recommended to calculate the hash value of the data recorded when recording video from drive recorder 1, store the hash value in relation to the video, and store it in a blockchain on a P2P (peer-to-peer) network connected to the network. Previously, one entity stored all the video data on a server, but with a distributed data database, everyone connected to the internet shares that data, and each server holds the same data. This involves attaching blocks, which are thrown onto the internet, to the existing blockchain, creating a chain that continues to grow. The data in that chain is the same data recorded in the database on all servers. The advantage is that since everyone has the same data, tampering can be detected unless a majority of people tamper with it.

The way the data is created involves calculating the hash value of the previous block and always including it in the next block when linking blocks together. It's impossible to tamper with a single block; to do so, all subsequent blocks would have to be tampered with, making tampering extremely difficult. The computational load required to calculate the value of a blockchain is set so that it takes about 10 minutes for Bitcoin, and recalculating all the blocks would be impossible time-wise. The entire flow of a contract can be recorded on the blockchain, or a bundle of contracts such as export and import procedures, including the people who transport goods within the destination country, the people who make payments, the people who exchange currency, and how orders are handled. For example, information such as "it passed through here on this date" and "it was loaded onto the ship" can all be recorded on the chain so that it can be verified later. It can also be used in areas where interests conflict and where a verification mechanism is needed. Furthermore, in a contract where an advertiser purchases advertising space and, for example, agrees to air several 15-second commercials between 7 and 8 PM (the prime time) over a month, it's impossible to verify whether all the ads were actually aired. Similarly, with transit advertising, it's impossible to verify where all the ads were displayed. With online advertising, it's also impossible to verify whether the ads actually received the required impressions or were displayed on the target audience's browsers. However, in situations where audits are needed between the advertiser and the person actually airing the ad, blockchain technology can be used. For example, if an ad is aired via blockchain, it can be tracked within the blockchain, creating a chain of notifications. This allows for verification on the blockchain that someone requested 1000 airings, and that 800 airings have actually occurred.

To prevent tampering, video data should be recorded on a blockchain. For example, in the case of video based on Motion JPEG (Joint Photographic Experts Group), where data is stored frame by frame, the hash value of each frame can be calculated, transmitted as a block, and linked to the blockchain system, thus preventing tampering. Alternatively, the hash value of the entire video file (one minute) within a certain range could be calculated.

In MPEG-based video data, splitting the data immediately before an I-frame makes it easy to tamper with; therefore, it's better to split it immediately before a P-frame.

In a public blockchain, like Bitcoin, anyone can participate and view all the data in the blocks flowing through it. In contrast, a consortium-type or private blockchain is closed off, for example, within a banking association, and can only be used among banks that are members of that association. For example, you can use a public blockchain system. Not only hash values, but also location information, such as the current location of a drive recorder, can be sent to the blockchain system, and that block will transmit information such as who is where on what date. For example, if information that drive recorder 901-1 is located here, in Okazaki City, Aichi Prefecture, is sent to the blockchain system, the server managing all the blockchains will broadcast it via P2P. If video data, etc., is sent to the blockchain system, it can be sold to users who want it.

Similarly, advertising can also be delivered. For example, you could send an advertisement for a store to users in a specific location, such as Okazaki City, Aichi Prefecture, and then pinpoint the ad to only those who are just before reaching that store. While maintaining a blockchain incurs costs, a small amount is deducted as a transaction fee and distributed to the user managing the server and the user who initially created the block. The currency information used for payment can also be recorded on the blockchain. For instance, if someone requests a photo for 100 yen, the 100 yen is deducted from their account, approximately 80 yen goes to the user of the dashcam, and 20 yen is distributed to the server owner.

It would be beneficial to be able to zoom in and out of the video using a remote control or similar device during playback. Furthermore, it would be good to allow users to move a frame using a cursor, for example, and to store the position of that frame in relation to the video's time. Ideally, the system should be able to record the frame's position during playback, and later create a video stream by extracting only that portion of the video based on the recorded frame position information.

It would be beneficial if the field of view could be manually adjusted by the user rather than automatically detected. For example, if the frame is moved using a joystick, mouse, or keyboard, the system should remember the moved position and display that frame during playback. This would allow others to recognize the range of vision a person had, as it varies depending on factors such as concentration levels while driving, age, and eye conditions. For example, a person could claim that they saw this range or looked at this range. It would also be beneficial to allow for subjective adjustments to brightness and color, as well as adjustments to remembered colors. In particular, it would be good to display both objective and subjective footage, allowing a comparison between the footage captured by the camera and the footage the user believes they actually saw. Displaying two types of footage of the same scene would allow a person to explain in court that while the camera actually recorded it this way, they saw it this way. In terms of criminal liability, two important factors are the objective occurrence of the accident and its consequences, and intent. It is crucial to prove whether the person hit the vehicle intentionally or negligently. Objective images cannot be manipulated, but subjective images can be presented in a way that differs from what one actually perceives, or both can be presented simultaneously.

Furthermore, the hash values of the vehicle's location information and event information may be calculated and transmitted to the blockchain system. This makes it difficult to tamper with event information such as sudden deceleration or acceleration.

Furthermore, to adjust the brightness, color, and memory color to subjectively perceived levels, a brightness adjustment slider may be provided on the drive recorder 1, and the brightness, color, and memory color may be adjusted by operating this slider. Alternatively, brightness adjustment buttons, color adjustment buttons, and memory color adjustment buttons may be provided on the playback window 130, and the brightness may be adjusted by operating these buttons.

Furthermore, in the case of autonomous driving of vehicles such as automobiles, it is preferable to record data in the header recording area 31 of the video file, as described above, that identifies whether the drive recorder 1 is recording during manual driving, where a driver is operating the vehicle, or during autonomous driving, where the vehicle is operating automatically. In this case, it is preferable to also record the level of autonomous driving in the header recording area 31 of the video file. When identification data indicating whether autonomous driving or manual driving is in progress, and data indicating the level of autonomous driving if autonomous driving is in progress, are transmitted within the vehicle's CAN (Controller Area Network), the drive recorder 1 can receive the data communicated within the CAN (for example, by connecting the drive recorder 1 to the CAN or by receiving the data communicated within the CAN in the communication circuit 17), thereby recording identification data indicating whether autonomous driving or manual driving is in progress, and data indicating the level of autonomous driving if autonomous driving is in progress, in the header recording area 31 of the video file. When playing video data, the playback window 130 can display whether the video displayed in the video display area 141 is from an automated driving scenario or a manual driving scenario. If it is from an automated driving scenario, the level of automated driving can also be displayed.

The drive recorder 1 may be configured to record video data with different image quality settings depending on whether the vehicle is in autonomous or manual driving mode. For example, it may record at a relatively high resolution during autonomous driving and a relatively low resolution during manual driving, or vice versa. Furthermore, the image quality may be adjusted according to the level of autonomous driving. The higher the level of autonomous driving, the higher the image quality may be recorded, or conversely, the higher the level of autonomous driving, the lower the image quality may be recorded.

Furthermore, if the drive recorder includes multiple cameras (front camera 11, rear camera 12), the camera recording may be switched depending on whether the vehicle is in autonomous or manual driving mode. For example, during autonomous driving, video data obtained only from the front camera 11 may be recorded, and during manual driving, video data obtained only from the rear camera 12 may be recorded, or vice versa. Furthermore, during playback, only autonomous driving footage may be played back, skipping manual driving footage, or vice versa. Additionally, video data recorded at one level of autonomous driving may be played back, while video data recorded at other levels may be skipped.

Furthermore, when playing back video data, the communication robot may provide guidance on the driving techniques shown in the played-back video. While direct explanations from an instructor to a user can sometimes anger the user, a communication robot saying "This was dangerous" reduces the psychological burden on both the instructor and the user. The communication robot is positioned near the instructor and user who are viewing the display device 92 of the personal computer 80 playing the video. For example, it is preferable to always place the communication robot to the right of the display device 92 of the personal computer 80, and to pre-determine the seating positions of the instructor and user. The video data playback software (program) may also allow settings such as whether the communication robot is placed to the right or left of the display device 92 of the personal computer 80. Of course, the communication robot may also be equipped with a camera that captures a 360-degree view of its surroundings, recognizing the positions of the display device 92, instructor, and user. The robot can recognize individuals in uniform as instructors and other individuals as users.

Furthermore, the system may detect the road width included in the video footage obtained by the drive recorder 1 or the video data obtained by the drive recorder 1's recording, and modify the video processing based on that road width. For example, in areas with narrow roads, the system may display left and right-facing footage, or change the type of video during playback according to the road conditions (narrow, gravel road, one-way street, etc.). Also, at intersections, if the drive recorder 1 is capable of displaying video during recording, it may display the front-facing footage, and similarly display the front-facing footage during playback. Additionally, for example, the display screen 3 of the drive recorder 1 may show right or left-facing footage that shows the road on the right or left side.

Furthermore, when recording video data with drive recorder 1, the system may analyze the video slightly ahead of the vehicle to which the drive recorder is installed to determine if a dangerous situation is occurring, and switch the video to a camera that makes the danger more visible a few seconds before the danger occurs. The display mode may also be switched so that the currently displayed video shows a danger in 2 seconds. For example, if a car is crossing in front of the vehicle in 2 seconds, the system may switch from the celestial mode (a mode that displays the situation in front, behind, and to the sides of the vehicle) to a mode that displays only the front. This allows the dangerous scene to be seen in 2 seconds. After that, it may be switched back to the original celestial mode. In this way, the display mode may be automatically changed when something unusual is likely to occur. Alternatively, the system may automatically analyze all video files in advance to detect unusual parts and mark the video to be played back. Whether or not there is danger can be determined by whether there is a large change in the video or a large change in acceleration. The display can be changed without the user having to operate a switch button. For example, it would be preferable to switch a few seconds before an event such as a danger is likely to occur, for example, 5 seconds before.

Furthermore, the drive recorder 1 may receive communication data from the engine starter and car security system, and use the received data as a trigger for event recording. The drive recorder 1 may be equipped with communication circuits based on LTE (Long Term Evolution), Wi-Fi, and Bluetooth®, record data representing the communication status of each, and perform processing to change the state of the displayed video according to the communication status. The display change processing may be a change in the display format of the video displayed on the display screen 3 of the drive recorder 1, or a change in the display format of the video in the playback window 130 during playback. For example, when a command signal to turn the car security system on or off is received, the system may automatically switch to video recording of the left and right sides of the vehicle, around the doors. Also, when a Wi-Fi signal is received, it is assumed that the vehicle has entered the home area, so the video may be stopped. When there are multiple communication means, it is advisable to decide which video to display according to each communication means. The display mode may be switched according to the communication means or the content of the communication. When using Bluetooth® to unlock car doors or start the engine, the system may switch from displaying the surrounding vehicle video to displaying the interior video as well, depending on the Bluetooth signal received. It may also capture the interior of the vehicle in addition to the surrounding area. This allows the system to capture the interior state at the moment the engine starts. For example, when using the remote starter, the system may revert to the original state after about 30 seconds. When unlocking the doors, the system may maintain recording both the surrounding area and the interior until the doors are locked, and then switch back to recording only the surrounding area or only the front of the vehicle.

Furthermore, in the operations of buses, taxis, and logistics companies, roll calls are always conducted. For example, alcohol checks are mandatory to confirm the driver's physical condition for the day. The video of these roll calls is recorded. It may be possible to view the roll call video and the drive recorder video together in the playback window 130. For example, by specifying the roll call video file, it would be possible to play the video recorded by the drive recorder 1 of the driver who was rolled over. For instance, if roll calls are conducted sequentially and drivers board the vehicles in that order, the roll call video and the drive recorder 1 video can be correlated based on their chronological relationship.

If a drive recorder 1 is installed on both a forklift and a vehicle such as a truck, it would be beneficial to transmit the video footage recorded by either drive recorder 1 to a server, allowing for a unified, continuous view of the daily movements of the same driver operating both vehicles (truck and forklift). For example, if a driver arrives at a certain location while driving a truck, then operates a forklift at another location, and finally drives a vehicle to a delivery destination, it would be possible to view this entire sequence of video. By combining this with roll call footage, the driver's entire daily routine can be viewed.

Furthermore, a composite reality system can be used for driving instruction. For example, a user drives once, and video is recorded on a dashcam. Then, in the playback window 130, the video is played back, and an automatic evaluation of the driving skills is performed. During this automatic evaluation, information on points to pay attention to is added to the video data. Subsequently, the user wears MR glasses and drives or rides in the passenger seat over the same location, displaying the information on the MR glasses. The user can see the information on points to pay attention to superimposed on the actual scenery. For example, information such as where to look for traffic lights and where to look for stop lines is displayed on the lenses (glass) of the MR glasses, allowing the user to confirm this information while viewing the actual scenery.

Furthermore, the driving history data can be recorded in memory 21 of drive recorder 1, and when approaching a previously visited location, it could display information such as when, where, and what route was taken. The driving history data could also be used to display and announce whether the location has been visited before.

Furthermore, the rear camera 12 of the drive recorder 1 may capture and recognize the vehicle's meter, triggering the start of video data recording. Alternatively, the vehicle's interior light may be captured, and if it is detected that the interior light is on, event recording may begin, and data indicating that the interior light is on may be sent to the server. From that server, for example, data indicating that the interior light is on may be sent to the vehicle's user. The user would then know that the interior light is on.

One option is to equip automated guided vehicles (AGVs) in factories with drive recorders (cameras). AGVs are equipped with LIDAR (Light Detection and Ranging) sensors, which can detect surrounding objects by rapidly scanning with lasers. Event recording (recording for a set period, e.g., tens of seconds to several minutes, depending on the trigger) can be initiated when an approaching object is detected.

Image recognition can be used to trigger event recording the moment the forklift's forks move. Alternatively, the front camera 11 and rear camera 12 of drive recorder 1 can be used to record, and recording can be triggered when the forklift moves. Furthermore, event recording can be started when a person wearing a helmet gets on the forklift, or conversely, when someone without a helmet gets on. In this case, recording can start when a person gets on and stop when they get off. The start and end of recording playback can also be determined by when a person gets on or off. A characteristic of forklifts is that while they are owned by a company, they are driven not only by company employees but also by transportation companies. Because the driver changes, recording can be started when such a person gets on, or when a person approaches, or until three minutes after they get off, then stopped. Forklift accidents are a problem, and this could help determine who caused the damage.

Furthermore, when a photograph is taken, Exif information is attached, and text data is embedded within the JPEG data. The exposure and shutter speed used at the time of shooting are embedded. This information may be displayed in the playback window 130 by playback software. Still image files may be generated from video footage, and the Exif information may be recorded in the header recording area of the still image file. For example, it is preferable to record the information displayed in the playback window 130 as header recording area Exif information. For example, a series of acceleration changes in the video may be historically recorded in the header recording area of the still image file. GPS information and speed, the maximum acceleration when moving within a certain range, the speed of the car, whether the brakes were applied, whether the turn signals were used, etc., may be recorded as Exif information in the header recording area. The entire range of speed may be recorded, or all values for the types of events that occurred within plus or minus 3 seconds before and after a certain event, information indicating which camera's footage it is, how many cameras were recording, and information indicating where the camera is mounted on the vehicle may be recorded as Exif information in the header recording area. Furthermore, if a drive recorder is installed in an autonomous vehicle within a factory, the trigger cause, whether the indicator light error was red or yellow, the status of the forklift, the amount of rotation of the forklift's turn, the status of the gyroscope, battery, etc., may be recorded as Exif information in the header recording area. Information obtained from AWS (Automatic Guided Vehicle), CWS (Collision Warning System), stoppers, vehicle-to-vehicle communication, and vehicle-to-infrastructure communication may also be recorded as Exif information in the header recording area. Information such as whether there are people or no people, whether there is a driver or no driver, whether there is a person in the passenger seat or not, whether someone is sitting or not in the back seat, whether it is a commercial vehicle or a different vehicle, a taxi, a bus, and whether there are people inside the bus or not may also be used as triggers, and this information may also be recorded as Exif information in the header recording area. Furthermore, the information recorded as Exif information may be transmitted to a blockchain system and recorded in the blockchain as a transaction.

Furthermore, points may be awarded to the user based on their use of playback software (program) that displays the playback window 130. For example, points may be added when the user connects to the internet and presses the "Import" button, uploading the imported data. If using a communication-type drive recorder 1, once permission is granted, video data can be transmitted automatically.

The scope of this invention is not limited to the configurations explicitly described in the specification, but also includes combinations of various aspects of the invention disclosed herein. While the configurations for which patent protection is sought are specified in the appended claims, we intend to include configurations not currently specified in the claims, as disclosed herein, within the scope of future claims.

The present invention is not limited to the configuration described in the embodiments above. The components of each embodiment and modification described above can be arbitrarily selected and combined. Furthermore, any component of each embodiment and modification can be arbitrarily combined with any component described in the means for solving the invention, or with components that embody any component described in the means for solving the invention. We intend to obtain rights for these combinations as well, through amendments or divisional applications of this application.

The scope of the present invention is not limited to the configurations explicitly described in the specification, but also includes combinations of various aspects of the present invention disclosed herein. While the configurations for which patent protection is sought are specified in the appended claims, the present inventors intend to include configurations not currently specified in the claims within the scope of the claims disclosed herein in the future.
The present invention is not limited to the configuration described in the embodiments above. The components of each embodiment and modification described above may be arbitrarily selected and combined. Furthermore, any component of each embodiment and modification may be arbitrarily combined with any component described in the means for solving the invention or any component that embodies any component described in the means for solving the invention. We intend to obtain rights for these as well in amendments or divisional applications of this application.
Furthermore, the applicant intends to obtain rights to the overall design or a partial design by filing an application for amendment to the design application. The drawing depicts the entire device with solid lines, but it is a drawing that includes not only the overall design but also partial designs claimed for parts of the device. For example, it is a drawing that includes not only a partial design for a part of the device, but also a partial design for a part of the device regardless of whether it is a part or not. A part of the device may be a part of the device's components, or a part of a component. The applicant intends to obtain rights not only to the overall design, but also to a partial design where any part of the solid lines in the drawing is represented by dashed lines.

1: Drive recorder, 2: Mounting section, 3: Display screen, 4: Camera section, 5: Front lens, 6: Rear lens, 7: Text display, 10: Controller, 11: Front camera, 12: Rear camera, 13: Display control device, 14: Display device, 15: Clock, 16: Infrastructure/vehicle-to-infrastructure information receiving circuit, 17: Communication circuit, 18: Acceleration sensor, 20: Input device, 21: Memory, 22: GPS receiver, 25: Memory card, 31: Header recording area, 32: Video recording area 80: Computer, 81: Display control device, 82: Display device, 90: CPU, 92: Display device, 93: Accelerometer, 95: Memory card reader/writer, 96: Input device, 97: Memory, 98: Communication circuit, 130: Playback window, 131: Menu bar, 132: Folder icon, 133: Camera icon, 134: Printer icon, 135: Reel icon, 136: FD icon, 137: Log icon, 138: Gear icon 141: Video display area, 141A: Video display area, 141B: Video display area, 142: Area outside the field of view, 143: Frame, 143A: Frame, 144: Mark, 144A: Mark, 145: Mark, 146: Speech bubble, 146A: Speech bubble, 146B: Speech bubble, 147: Speech bubble, 148: Dashed line, 151: Map image display area, 152: Icon, 161: Vehicle information etc. display area, 162: Vehicle status display area, 163: Operation button, 166: Time display area, 168: Acceleration graph display area, 17 1: Playlist display area, 172: Playlist display area, 280: Communication robot, 281: Torso, 282: Head, 283: Eyes, 284: Mouth, 291: CPU, 292: Memory, 293: Communication LAN board, 294: Wireless communication circuit, 295: Motor control board, 296: Head motor, 297: Sensor board, 298: Eye camera, 299: Audio input/output board, 300: Speaker, 301: Microphone, 310: Windshield, 311: Dashboard

Claims (11)

A system equipped with a function to record a series of videos in association with first video data obtained by a first recording device installed on a truck driven by a driver, and second video data obtained by a second recording device installed on a forklift operated by the driver after the driver switches from the truck to the forklift , so that they can be played back and viewed. The system according to claim 1, comprising the function of transmitting first video data obtained by a first recording device installed on the truck to a server, transmitting second video data obtained by a second recording device installed on the forklift to the server, and recording these video data in association with each other on the server. The system according to claim 1 or 2, comprising a function for storing first video data obtained by a first recording device installed on the truck and second video data obtained by a second recording device installed on the forklift in the same video file. A system according to any one of claims 1 to 3, comprising: recording first video data obtained by the first recording device installed on the truck onto a recording medium loaded in the first recording device, wherein the recording medium is removable from the first recording device installed on the truck, and when the recording medium is loaded into the second recording device installed on the forklift, the system having a function to store second video data obtained by the second recording device installed on the forklift in the same video file as the first video data obtained by the first recording device installed on the truck. The system according to any one of claims 1 to 4, further comprising a function for recording the second video data based on at least one of the operation of the forklift and the fact that the driver is on the forklift. The system according to any one of claims 1 to 5, further comprising a function that starts recording by a second recording device installed on the forklift when the second recording device detects that a driver has approached the forklift. The system according to any one of claims 1 to 6, comprising a function for processing a second video data, transmitted from the same location as the truck's parking location as the first video data, and recorded by a second recording device installed on the forklift within a predetermined time from the truck's parking time, in association with the first video data. The system according to any one of claims 1 to 7, which, when there are two video data sets—first video data obtained by a first recording device installed on a truck and second video data obtained by a second recording device installed on a forklift—where the recording end time and recording start time are approximately the same, and the location at the end of recording and the location at the start of recording are approximately the same, has a function to treat these video data sets as video data of a transfer from the truck to the forklift. The system according to any one of claims 1 to 8, wherein the truck is replaced with a forklift, or the forklift is replaced with a truck. A system according to any one of claims 1 to 9, comprising a function to receive a video file of a driver's roll call from a server and to display the video of the driver's roll call and the video of the driver driving in the video display area of a display means. A program for a computer to implement the functions of the system described in any one of claims 1 to 10.