JP7776100B2 - Regeneration control system and computer program - Google Patents

Description

This disclosure relates to a playback control system and a computer program.

There is a known imaging device equipped with a center position correction function to reduce the discomfort felt by users viewing captured images due to the vanishing point shifting from the center of the captured image when the vehicle turns a corner (see, for example, Patent Document 1). This imaging device calculates the position of the vanishing point based on the optical flow of the captured image, and corrects the captured image so that the position of the vanishing point coincides with the center of the captured image.

JP 2012-10145 A

A sense of discomfort felt by users viewing captured images can occur even when the vehicle is traveling straight if the camera is pointing in a direction that is not aligned with the vehicle's direction of travel. In this case, correcting the captured image may be an option.

However, there is still room for improvement in conventional technology when it comes to correcting captured images. For example, there are known drive recorders that record video data to portable recording media or server devices.

Users can play back video data recorded on recording media or a server device on a separate device outside the vehicle, such as a personal computer. However, with conventional technology, users were unable to use correction functions to reduce the sense of discomfort on the separate device, and had to view the captured image without correction.

Accordingly, according to one aspect of the present disclosure, it is desirable to provide a novel technology suitable for reducing the sense of discomfort felt by users in playback images caused by the camera orientation being off-axis with respect to the moving object's direction of travel. For example, it is desirable to provide a novel technology that can reduce the sense of discomfort felt by users in playback images caused by the camera orientation being off-axis with respect to the moving object's direction of travel, even in an environment where video data can be played back on a separate device away from the drive recorder.

According to one aspect of the present disclosure, a playback control system is provided that includes an acquisition unit and a setting unit. The acquisition unit is configured to acquire video data representing images captured by a camera installed on a moving object.

The setting unit is configured to generate control data for processing the video data and displaying a playback image on the display device, so that a captured image with its front direction adjusted to the traveling direction of the moving object is displayed on the display device as a playback image based on the video data. The setting unit is further configured to write the control data to the storage device in association with the video data.

As described above, with this playback control system, control data associated with video data is written to a storage device. Therefore, when a user attempts to view a playback image based on the video data, the video data playback device can process the video data based on the control data associated with the video data so that the front direction of the playback image matches the direction of travel of the moving object, and display the playback image on a display device, without analyzing the video data or requiring the user to specify or search for appropriate image processing.

Therefore, one aspect of the present disclosure provides a novel technology that can advantageously reduce the sense of discomfort felt by the user due to the camera orientation not matching the direction of travel of the moving object.

According to one aspect of the present disclosure, the setting unit may be configured to determine the relative relationship between the camera orientation and the direction of travel of the moving object based on the video image data, and to generate control data based on that relative relationship. With this configuration, even if the user does not accurately understand the relative relationship, appropriate control data can be generated that allows the playback image to be displayed in a way that reduces the sense of discomfort described above.

According to one aspect of the present disclosure, the setting unit may be configured to display a setting screen on the display device that is separate from the playback screen on which the playback image is displayed, and to accept setting operations through the setting screen.

The setting screen may be a screen for accepting setting operations by the user regarding the above adjustment. The setting unit may be configured to generate control data based on the above setting operations. With this configuration, the control data can be generated according to the user's wishes or with assistance from the user.

According to one aspect of the present disclosure, the setting unit may be configured to determine the relative relationship between the camera orientation and the direction of travel of the moving object based on the setting operation, and generate control data.

According to one aspect of the present disclosure, the moving body may be a vehicle. The traveling direction of the moving body may be the vehicle's straight-ahead direction. The setting unit may be configured to determine the relative relationship based on a specific reference object inside or outside the vehicle that appears in the captured image.

There are suitable reference objects inside and outside the vehicle for determining relative relationships. Therefore, based on the specific reference objects shown in the replayed image, the relative relationships can be properly determined, and image processing for the above adjustments can be performed appropriately.

According to one aspect of the present disclosure, the setting unit may be configured to determine the relative relationship based on the optical flow of the captured image. Based on the optical flow, the relative relationship can be determined even when or at a stage where the specific reference object is not clearly visible in the captured image.

According to one aspect of the present disclosure, the setting operation may include an operation of pointing to a specific reference object inside or outside the vehicle that appears in the captured image within the captured image displayed on the setting screen. The setting unit may determine the relative relationship based on the specific reference object pointed to by the setting operation.

This configuration allows the relative relationship to be determined with assistance from the user, making it easier and more accurate to determine the relative relationship than identifying a specific reference object through image analysis without user guidance.

According to one aspect of the present disclosure, the specific reference object may include, for example, one of the windshield inside the vehicle, a road dividing line outside the vehicle, and the horizon outside the vehicle that appear in the captured image.

According to one aspect of the present disclosure, the setting operation may include an operation for specifying an operation mode of the setting unit. The setting unit may be configured to determine the relative relationship between the operation mode specified by the setting operation among multiple operation modes.

The multiple operating modes may include two or more of an operating mode that determines the relative relationship based on a first reference object inside the vehicle that appears in the captured image as a specific reference object, an operating mode that determines the relative relationship based on a second reference object outside the vehicle that appears in the captured image as a specific reference object, and an operating mode that determines the relative relationship based on optical flow in the captured image.

By determining the relative relationship and generating control data based on one of multiple operating modes, it is possible to determine the operating mode desired by the user, or to generate control data for the operating mode that the user considers optimal in terms of reducing discomfort and/or discrimination accuracy.

According to one aspect of the present disclosure, a playback control system may be provided that includes an acquisition unit, a playback control unit, and a setting unit. The acquisition unit may be configured to acquire video data representing images captured by a camera installed on a mobile object.

The playback control unit can be configured to control the display device so that a playback image corresponding to the captured image is displayed on the playback screen of the display device based on the video image data acquired by the acquisition unit.

The setting unit may be configured to display a setting screen separate from the playback screen on the display device and to accept setting operations through the setting screen. The setting screen may be a screen for accepting setting operations by the user to adjust the front direction of the playback image to the traveling direction of the moving object.

The setting unit may further be configured to set an image processing method for the moving image data for the above adjustment based on the setting operation. The playback control unit may be configured to process the moving image data in accordance with the image processing method set by the setting unit, thereby causing the display device to display, as a playback image, a captured image whose front direction has been adjusted to match the traveling direction of the moving object.

With this configuration, an appropriate image processing method can be set based on user operation, and the front direction of the reproduced image can be aligned with the direction of travel of the moving object with high precision, effectively reducing any sense of discomfort the user may have with the reproduced image.

According to one aspect of the present disclosure, when the moving object is a vehicle, the setting operation may include an operation of pointing to a specific reference object inside or outside the vehicle that appears in the captured image within the captured image displayed on the setting screen. The setting unit may be configured to determine the relative relationship between the camera orientation and the vehicle's straight-ahead direction based on the specific reference object pointed to by the setting operation, and to set an image processing method for the video image data to adjust the forward direction of the reproduced image to the vehicle's straight-ahead direction based on the relative relationship.

According to one aspect of the present disclosure, some or all of the functions of the playback control system described above may be implemented by a computer program.

According to one aspect of the present disclosure, a computer program may be provided to cause a computer to execute processing including acquiring video data representing images captured by a camera mounted on a moving object, processing the video data to generate control data for displaying a replayed image on a display device, such that a captured image with its front direction adjusted to the direction of travel of the moving object is displayed on the display device as a replayed image based on the video data, and writing the control data to a storage device in association with the video data.

According to one aspect of the present disclosure, a computer program may be provided that causes a computer to execute processing including: acquiring video data representing images captured by a camera mounted on a moving object; controlling a display device based on the acquired video data so that a playback image corresponding to the captured image is displayed on a playback screen of the display device; causing the display device to display, as a setting screen separate from the playback screen, a screen for accepting a setting operation by a user to adjust the front direction of the playback image to the traveling direction of the moving object, and accepting the setting operation through the setting screen; and setting an image processing method for the video data for the adjustment based on the setting operation. Controlling the display device may include processing the video data in accordance with the set image processing method, thereby displaying, as a playback image, the captured image whose front direction has been adjusted to the traveling direction of the moving object on the display device.

According to one aspect of the present disclosure, a computer program may be provided that causes a computer to execute a process of acquiring the video data and control data associated with the video data from a storage device, and displaying, on a display device, a playback image corresponding to a captured image recorded in the video data based on the acquired video data and control data.

According to one aspect of the present disclosure, there may be provided a computer program that causes a computer to acquire video data and ancillary data associated with the video data from a storage device, and to display on a display device a replay image corresponding to a captured image recorded in the video data by image processing the acquired video data, wherein the captured image is an image captured by a camera mounted on a moving object, and the ancillary data includes information that can identify a difference between the orientation of the camera and the traveling direction of the moving object, and displaying the replay image on the display device includes image processing the video data based on the acquired ancillary data so that the captured image displayed on the display device has its front direction adjusted to match the traveling direction of the moving object as the replay image.

These computer programs may be recorded on a non-transitory computer-readable recording medium or provided via a telecommunications line.

FIG. 2 is a block diagram illustrating a configuration of a drive recorder. FIG. 2 is a functional block diagram of a control unit. 10 is a flowchart showing a reproduction-related process executed by a control unit. 10 is a flowchart showing a setting process executed by a control unit. FIG. 10 is a diagram illustrating a setting mode designation operation. FIG. 10 is a diagram illustrating an example of a screen displaying a setting completion message. FIG. 10 is a diagram illustrating a setting operation using the "windshield." 10A and 10B are diagrams illustrating a setting operation using the "horizon line." FIG. 10 is a diagram illustrating a setting operation using a "white line." FIG. 10A is an explanatory diagram regarding the display of a reproduced image, and FIG. 10B is an explanatory diagram regarding the display of a reproduced image without adjustment as a comparative example. 10A and 10B are explanatory diagrams relating to display of a playback image in a full view panoramic display mode. FIG. 12A is a diagram illustrating an adjusted round image, and FIG. 12B is a diagram illustrating an unadjusted round image as a comparative example. FIG. 10 is an explanatory diagram of a modified drive recorder in which moving image data is taken out of the vehicle via a server device.

Exemplary embodiments of the present disclosure will now be described with reference to the drawings.
The drive recorder 1 of this embodiment shown in FIG. 1 is an electronic device that is mounted on and used in a vehicle such as a four-wheeled automobile as a mobile body.

For example, the drive recorder 1 is used by being fixed to the room mirror (i.e., rearview mirror) inside the vehicle cabin, which the driver uses to view the area behind the vehicle. The room mirror is usually located in the center in the left-right direction above the windshield (i.e., the windshield). Alternatively, the drive recorder 1 is configured as a mirror-type drive recorder that also functions as a room mirror. Alternatively, the drive recorder 1 is fixed to the windshield or another location inside the vehicle cabin via a stay, and is used independently of the room mirror.

As shown in FIG. 1, the drive recorder 1 includes a control unit 10, a camera 20, sensors 30, a display unit 40, an operation unit 50, and a media reader/writer 60. The control unit 10 includes a processor 11 and a memory 13.

Memory 13 includes volatile memory (RAM) used as a working area when processor 11 executes processing, and non-volatile memory (NVRAM) such as electrically rewritable flash memory. Non-volatile memory stores computer programs and data that must be retained even during shutdown.

The processor 11 controls each part of the drive recorder 1 by executing processes in accordance with the computer program stored in the memory 13. The processes executed by the control unit 10 described below are realized by the processes executed by the processor 11 in accordance with the computer program.

Camera 20 is mounted, for example, on the surface of drive recorder 1 facing the windshield, and is used to capture images of the vehicle's surroundings, including the area ahead of the vehicle. Camera 20 may be a wide-angle camera or an omnidirectional camera capable of capturing images in a 360-degree azimuth.

Examples of wide-angle cameras include ultra-wide-angle cameras that can capture images of an azimuth angle of 180 degrees or more. Ultra-wide-angle cameras and omnidirectional cameras can be cameras configured by combining multiple wide-angle cameras. Video data representing captured images of the vehicle's surroundings generated by camera 20 is input to control unit 10.

The sensors 30 consist of one or more devices for measuring the vehicle's driving conditions. The sensors 30 include, for example, a positioning device such as a GPS receiver that measures the vehicle's position, and an acceleration sensor that measures the vehicle's acceleration. The acceleration sensor can be used, for example, to record video data representing captured images of the vehicle's surroundings when a vehicle impact is detected as important data. Measurement signals from the sensors 30 are input to the control unit 10.

The display unit 40 has a display surface facing the interior of the vehicle and is configured as a display device that displays information for vehicle occupants such as the driver through the display surface. The display unit 40 is configured as, for example, a liquid crystal display. The display unit 40 is controlled by the control unit 10 to display various information.

The operation unit 50 is configured as a user interface capable of accepting operations from a user corresponding to the driver or other vehicle occupant. For example, the operation unit 50 is provided with a touch panel on the display surface of the display unit 40, and is configured to accept user operations on the display surface through user operations on the touch panel. Operation signals from the user are input to the control unit 10 via the operation unit 50.

The media reader/writer 60 is configured to allow portable recording media 70 to be connected and disconnected, and is configured to read and write data from and to the connected recording media 70 in response to requests from the control unit 10. The recording media 70 may be a memory card with built-in flash memory. Examples of recording media 70 include SD cards.

Next, the functions realized by the control unit 10 will be explained using Figure 2. The control unit 10 functions as a video recording unit 110, a playback unit 120, an image processing unit 130, and a setting unit 150 by executing processes in accordance with a computer program.

The video recording unit 110 is configured to write video data input from the camera 20 to the recording media 70 via the media reader/writer 60. For example, the video recording unit 110 is configured to divide the video data input from the camera 20 into predetermined time intervals and write the data to the recording media 70 as a recording file that records images captured at the corresponding predetermined time intervals. The main function of the drive recorder 1, i.e., the function of recording captured images of the vehicle's surroundings while the vehicle is traveling, is realized by the video recording unit 110.

The video recording unit 110 may be configured to record video data representing images captured during a limited period of time, based on input from the sensors 30, onto the recording media 70, rather than continuously recording video data.

The video data recorded on the recording media 70 may include data that can identify the shooting date and time and shooting location. The shooting location may be the GPS location at the time of shooting, which can be obtained from the sensors 30.

When a user performs a playback operation in accordance with an input signal from the operation unit 50, the playback unit 120 reads video image data (e.g., one or more video files in which images captured during a period specified by the user are recorded) from the recording medium 70 via the media reader/writer 60. The playback unit 120 performs playback processing on the data obtained by this reading, thereby outputting playback images, specifically video images, corresponding to the images captured during the specified period.

The playback image output from the playback unit 120 is a captured image of the entire shooting range corresponding to the angle of view 2α of the camera 20. For example, if the camera 20 is an omnidirectional camera with an angle of view 2α of 360 degrees in azimuth, the playback image output from the playback unit 120 may be a panoramic image with an azimuth angle of 360 degrees.

The image processing unit 130 functions as a playback control unit in cooperation with the playback unit 120, and is configured to process the playback image input from the playback unit 120, convert it into a playback image for display, and display the converted playback image for display on the display unit 40. The image processing is performed in accordance with playback control data attached to the video data that is recorded together with the video data on the recording medium 70. Details of the playback control data and image processing will be described later.

The playback image input from the playback unit 120 to the image processing unit 130 is converted by the image processing described above into a playback image for display whose front direction corresponds to the direction of travel of the vehicle at the time of shooting. As a result, the playback image displayed on the display unit 40 does not appear strange to the user.

In addition, the playback image displayed on the display unit 40 has an aspect ratio of 16:9 and an angle of view 2β, which is narrower than the angle of view 2α of the camera 20. In other words, in image processing, a playback image with an angle of view 2β is generated by extracting a portion of the playback image with angle of view 2α.

The setting unit 150 is configured to generate playback control data in accordance with the user's setting operation via the operation unit 50, and to record this playback control data in association with the video data on the same recording medium 70 as the video data, in order to cause the image processing unit 130 to perform the above-mentioned image processing of aligning the front direction of the playback image with the direction of travel of the vehicle.

Next, the playback-related processing executed by the control unit 10 to function as the playback unit 120, image processing unit 130, and setting unit 150 will be described in detail using Figures 3 and 4. The control unit 10 repeatedly executes the playback-related processing shown in Figure 3 while the drive recorder 1 is running.

When playback-related processing starts, the control unit 10 waits until an operation signal is input from the user via the operation unit 50 (S110). When an operation signal is input (Yes in S110), the control unit 10 executes processing from S120 onwards.

At S120, the control unit 10 determines whether the user's operation in accordance with the input operation signal is a setting operation for adjusting the forward direction of the playback image to the direction of travel of the vehicle (S120). If it is determined to be a setting operation (Yes at S120), the control unit 10 proceeds to S125 and executes setting processing for the above adjustment. Thereafter, the control unit 10 terminates the playback-related processing.

On the other hand, if it is determined that the user's operation is not the above-mentioned setting operation but an operation to play video data recorded on the recording medium 70 (No in S120 and Yes in S130), the control unit 10 switches the screen displayed by the display unit 40 to a playback screen and accepts an operation to specify the playback period (S140). The playback screen can be equipped with a graphical user interface for accepting an operation to specify the playback period.

When the user specifies the playback period through a specification operation, the control unit 10 executes a process of displaying, on the playback screen of the display unit 40, playback images corresponding to the captured images captured during the period corresponding to the playback period, from among the captured images recorded in the video image data in the recording media 70 (S145).

Here, the playback screen displays the playback image, and instead of the graphical user interface for accepting the operation to specify the playback period, it can display a graphical user interface for accepting user operations related to video playback, such as changing the playback speed, fast-forwarding, rewinding, and stopping.

At S145, the control unit 10 functions as the playback unit 120 and image processing unit 130, and performs image processing that takes into account the discrepancy between the orientation of the camera 20 and the direction of travel of the vehicle, based on the playback control data recorded on the recording media 70 together with the video data.

In other words, the control unit 10 performs playback processing on the video image data and performs image processing in a direction that suppresses misalignment, causing the display unit 40 to display a playback image that corresponds to the captured image as seen from a line of sight along the direction of travel of the vehicle at the time of capture. After playback is complete, the control unit 10 returns the screen displayed on the display unit 40 from the playback screen to the screen before switching (S150), and ends the playback-related processing shown in Figure 3.

In addition, if a user operation other than a setting operation or a playback operation is performed (No in S120 and No in S130), the control unit 10 executes processing corresponding to the performed operation (S160) and terminates the playback-related processing.

Next, the details of the setting process executed by the control unit 10 in S125 will be explained using Figure 4. By executing the setting process, the control unit 10 functions as the setting unit 150. When the setting process starts, the control unit 10 switches the screen displayed by the display unit 40 to a setting screen (S210). The screen before switching to the setting screen may be a main screen (not shown) that displays a graphical user interface for accepting main operations superimposed on the image currently being captured by the camera 20.

The setting screen after switching may be a screen in which a graphical user interface for specifying the setting mode for the above adjustments is superimposed on the image currently being captured by the camera 20, as shown in Figure 5.

According to this embodiment, the drive recorder 1 has setting modes for the above adjustments: a "vector" setting mode, a "windshield" setting mode, a "horizon" setting mode, and a "white line" setting mode.

As shown in FIG. 5, the setting screen can have a graphical user interface for accepting designation of one of these setting modes from the user. The setting screen illustrated in FIG. 5 has selection buttons for each setting mode as a graphical user interface. When one of these selection buttons is pressed via the operation unit 50, the control unit 10 can determine that the setting mode corresponding to the pressed selection button has been designated. The setting screen can further have a graphical user interface for canceling the setting operation.

After displaying the setting screen, the control unit 10 waits until an operation to specify one of the setting modes or a cancellation operation is performed (S220, S230, S240, S250, S280). When an operation to specify the "Vector" setting mode is performed as an operation to specify one of the setting modes (Yes in S220), the control unit 10 detects a vanishing point in the captured image based on the optical flow of the captured image obtained by analyzing captured images recorded in the video data from the present to a predetermined period ago (S225).

The control unit 10 further estimates the difference Dv in the horizontal direction of the captured image between the detected vanishing point and the center line extending vertically in the captured image as the difference δθ in the azimuth angle between the front direction of the camera 20 and the vehicle's straight-ahead direction (S225). The horizontal direction of the captured image corresponds to the azimuth angle direction. The difference Dv can be expressed as the signed distance in the horizontal direction of the captured image from the center line of the captured image to the vanishing point.

The control unit 10 then generates playback control data describing the estimated difference Dv and writes the generated playback control data to the recording medium 70 (S260). This playback control data is interpreted as data instructing that the video data be processed to generate a playback image by shifting the center of the captured image recorded in the video data horizontally by the amount of the difference Dv.

Image processing based on this playback control data generates a playback image for display in which the forward direction of the playback image corresponds to the vehicle's straight-ahead direction.

If playback control data has already been recorded on the recording medium 70 before writing the playback control data in S260, the control unit 10 can write the newly generated playback control data to the recording medium 70 in a format that overwrites the recorded playback control data.

As another example, the control unit 10 can write newly generated playback control data to the recording medium 70 in an appended format, rather than overwriting already recorded playback control data. In this case, the playback control data can include registration date and time information, and the image processing unit 130 (in other words, the control unit 10) can process the playback image from the playback unit 120 based on the most recent playback control data registered before the corresponding shooting date and time. Alternatively, the image processing unit 130 can process the image based on playback control data specified by the user.

After processing in S260, the control unit 10 temporarily displays a setting completion message on the display unit 40 explaining that the setting has been completed, as shown in FIG. 6, and then returns the screen displayed on the display unit 40 to the screen that was displayed before switching to the setting screen in S210 (S270), thereby terminating the setting process.

Examples of the screen before switching include the main screen described above. After the setting process, the main screen may display a captured image that has been subjected to image processing based on the playback control data most recently generated as a playback image corresponding to the image currently being captured by the camera 20. In other words, a captured image whose front direction has been adjusted to match the vehicle's straight-ahead direction may be displayed on the display unit 40 as a playback image. Based on the playback control data describing the difference Dv, an image captured by the camera 20 that has been image processed so that the vanishing point is positioned on the center line of the playback image may be displayed on the display unit 40.

As shown in Figure 6, the screen displaying the setting completion message output in processing S270 may also display the captured image that has been subjected to image processing based on the playback control data generated immediately before, along with the setting completion message.

In addition, when the user operates the setting screen to specify the "Windshield" setting mode (Yes in S230), the control unit 10 receives the user's instruction to trace the boundary of the windshield by causing the display unit 40 to display a panoramic image of the image currently captured by the camera 20 on the setting screen, as shown in FIG. 7, and also displays a message on the setting screen instructing the user to trace the boundary of the windshield as seen in this panoramic image (S231). The panoramic image is a captured image of the entire shooting range corresponding to the angle of view 2α of the camera 20.

The user can use the operation unit 50 to trace the boundary of the windshield shown in this panoramic image. The dotted line in Figure 7 indicates the boundary of the windshield to be traced. The panoramic display on the settings screen is designed to show the entire windshield, thereby enabling the user to trace the boundary of the entire windshield.

When the user finishes tracing the boundary of the windshield, the control unit 10 estimates the difference Df in the azimuth angle between the front direction of the camera 20 and the front direction of the windshield as the difference δθ based on the distortion of the shape of the windshield in the panoramic image identified by the operation (S235). The front direction of the windshield corresponds to the straight-ahead direction of the vehicle. The difference Df can be expressed as a signed angle in the azimuth angle from the front direction of the windshield to the front direction of the camera 20.

Actual windshields have a symmetrical shape. Therefore, if the difference Df is zero, the windshield shown in the panoramic image is symmetrical with respect to the center line extending vertically from the center of its shape. In contrast, if the difference Df is non-zero, the opposite sides extending horizontally are not parallel, and the vertical lengths of the windshield at positions equidistant from the center line are different. From this difference in length, the difference Df can be geometrically estimated. When estimating the difference Df, the lens characteristics of the camera 20 (e.g., distortion) can also be taken into account.

After estimating the difference Df in S235, the control unit 10 executes the processes of S260 and S270 described above, and ends the setting process. As a result, the difference Df estimated in S235 is written to the playback control data and stored in the recording medium 70. This playback control data is interpreted as data instructing that the video data be processed to generate a playback image so as to shift the center of the captured image recorded in the video data in the azimuth direction of the captured image by the difference Df.

In addition, when the user operates the setting screen to specify the "horizon" setting mode (Yes in S240), the control unit 10 receives the user's operation to trace the horizon, and causes the display unit 40 to display a panoramic view of the image currently being captured by the camera 20 on the setting screen, as shown in FIG. 8, and also displays a message on the setting screen instructing the user to trace the horizon that appears in the image captured by the camera 20 (S241).

When the user finishes tracing the horizon, the control unit 10 estimates the difference Dh in the azimuth angle between the front direction of the camera 20 and the front direction of the windshield as the difference δθ based on the inclination of the horizon in the panoramic image identified by the user's operation relative to the bottom edge of the windshield identified by image analysis (S245). The difference Dh can be expressed as a signed angle in the azimuth angle from the front direction of the windshield to the front direction of the camera 20.

The horizon appears the same in the panoramic image regardless of the orientation of the camera 20 in terms of azimuth. In contrast, the bottom edge of the windshield changes depending on the orientation of the camera 20, as explained in the "Windshield" setting mode above. If the difference Dh is zero, the horizon and the bottom edge of the windshield are symmetrical about the center line of the panoramic image, and if there is no distortion due to the lens characteristics (distortion aberration) of the camera 20, the horizon and the bottom edge of the windshield are approximately parallel. In contrast, if the difference Dh is non-zero, the bottom edge of the windshield is not symmetrical about the center line of the panoramic image, and the horizon appears tilted relative to the bottom edge of the windshield. From this tilt, the difference Dh can be estimated.

After estimating the difference Dh in S245, the control unit 10 executes the processes of S260 and S270 described above, and ends the setting process. As a result, the difference Dh estimated in S245 is written to the playback control data and stored in the recording medium 70. This playback control data is interpreted as data instructing that the video data be processed to generate a playback image so as to shift the center of the captured image recorded in the video data in the azimuth direction of the captured image by the amount of the difference Dh.

In addition, when the user operates the settings screen to specify the "white line" setting mode (Yes in S250), the control unit 10 proceeds to S251, and in order to receive the user's operation to trace the white lines that serve as road dividing lines, as shown in FIG. 9, the display unit 40 displays a panoramic view of the image currently captured by the camera 20 on the settings screen, and also displays a message on the settings screen instructing the user to trace the white lines that appear in the image captured by the camera 20 (S251).

The white lines referred to here are the two outer perimeter lines on both sides of the road. The outer perimeter lines are usually displayed on the road as solid white lines. The user is required to trace the two outer perimeter lines.

When the user has finished tracing the white lines, the control unit 10 estimates the difference Dw in the horizontal direction of the panoramic image between the vanishing point and the center line extending up and down the panoramic image as the difference δθ, based on the slope of the white lines in the panoramic image identified by that operation (S255). The intersection of the two white lines when extended corresponds to the vanishing point. Therefore, the difference Dw can be estimated based on the difference between the intersection point and the center line.

After estimating the difference Dw in S255, the control unit 10 executes the processes of S260 and S270 described above, and then ends the setting process. As a result, the difference Dw estimated in S255 is written to the playback control data and stored in the recording medium 70. This playback control data is interpreted as data instructing that the video data be processed to generate a playback image by shifting the center of the captured image recorded in the video data horizontally by the amount of the difference Dw.

In addition, when a cancel operation is performed (Yes in S280), the control unit 10 returns the screen displayed by the display unit 40 to the screen before switching to the setting screen in S210 (S290), and ends the setting process.

Next, we will explain the image processing method performed by the image processing unit 130 based on the playback control data. As described above, the playback control data stores differences Dv, Df, Dh, and Dw that are substantially equivalent to the difference δθ in the azimuth angle between the front direction of the camera 20 and the straight-ahead direction of the vehicle. Therefore, below we will use Figure 10A to explain the image processing method when there is a difference δθ in the azimuth angle between the front direction of the camera 20 and the straight-ahead direction of the vehicle.

In addition, the forward direction of the camera 20 here corresponds to the line of sight from the camera 20 toward the subject located at the center of the image captured by the camera 20, in terms of azimuth angle. For the sake of explanation, the azimuth angle θ is defined below as an azimuth angle where the forward direction of the vehicle, i.e., the straight-ahead direction of the vehicle, is set to zero.

Figure 10A shows that the front direction of camera 20 is offset by an azimuth angle δθ from the vehicle's straight-ahead direction. Here, we assume that the angle of view of camera 20 is 2α, and that when the front direction of camera 20 coincides with the vehicle's straight-ahead direction, the captured images recorded in the video image data are captured images that capture the field of view within the azimuth angle range of -α≦θ≦+α.

When the front direction of camera 20 is shifted by δθ in the negative azimuth direction relative to the vehicle's straight-ahead direction, as shown in Figure 10A, images captured within the azimuth angle range of -α-δθ≦θ≦+α-δθ are recorded in the video data. In other words, the capture range of the video data is the azimuth angle range of -α-δθ≦θ≦+α-δθ.

At this time, the image played back from the playback unit 120 is a moving image with a shooting range of azimuth angles -α-δθ≦θ≦+α-δθ, with the image shot at azimuth angle δθ positioned at the center of the played back image as if it were at an azimuth angle of zero. In other words, the image played back from the playback unit 120 is a played back image with a field angle of 2α, with its center, in other words, the front direction of the played back image, at an azimuth angle of δθ.

When the image processing unit 130 displays this playback image on the display unit 40, as shown in FIG. 10A, it extracts a playback image with an azimuth angle of -β≦θ≦+β as a playback image with a narrower angle of view 2β from the playback image with an angle of view 2α output from the playback unit 120 based on the playback control data. This extraction generates a playback image for display. As described above, the playback image for display is a playback image with an aspect ratio of 16:9, and is different from the above-mentioned panoramic image which represents the entire shooting range corresponding to the angle of view 2α of the camera 20.

The image processing unit 130 displays the playback image for display generated by the above extraction on the display unit 40, causing the display unit 40 to display a playback image with a field of view of 2β, in which the front direction of the playback image (in other words, the center of the playback image) coincides with the vehicle's straight-ahead direction. As a result, even if the camera 20 is facing in a direction that deviates from the vehicle's straight-ahead direction, the display unit 40 displays a moving image captured by the camera 20, with the image center adjusted to an azimuth angle of zero and a display range of azimuth angles -β≦θ≦+β.

Figure 10B shows, as a comparative example, the correspondence between the display range of the playback image on the display unit 40, the front of the camera 20, and the vehicle's straight-ahead direction (azimuth angle zero) when the image processing unit 130 does not take the difference δθ into account, considers the center of the playback image from the playback unit 120 to be at an azimuth angle of zero, and extracts a playback image with a narrower angle of view 2β from a playback image with an angle of view 2α to generate a playback image for display. In this case, the forward direction of the playback image displayed on the display unit 40 does not match the vehicle's straight-ahead direction. As a result, the playback image displayed on the display unit 40 is unnatural to the user. People are accustomed to the flow of their field of vision when facing forward, so if the forward direction of the playback image displayed on the display unit 40 does not match the vehicle's straight-ahead direction, the user will feel uneasy.

In the drive recorder 1 of this embodiment described above, the setting unit 150 generates playback control data for image processing of video image data in accordance with the user's setting operations via the operation unit 50 and the setting screen, so that a captured image is displayed on the display device with the front direction adjusted to the vehicle's straight-ahead direction, and writes this data to the same recording medium 70 as the video image data. The generation of the playback control data and writing it to the recording medium 70 correspond to the setting of the image processing method.

Therefore, even when the recording medium 70 is removed from the media reader/writer 60 and the user views a playback image based on the video data recorded on the recording medium 70 on another device, such as the user's own personal computer, the other device can process the video data so that the forward direction and the direction of travel coincide, and display the playback image on a display device, without analyzing the video data or requiring the user to specify or search for appropriate image processing.

For example, if software for causing a personal computer to execute a playback process similar to the process executed by the control unit 10 at S145 is provided to the user of the drive recorder 1, for example via a web page, the user can effectively utilize the playback control data generated by the drive recorder 1 to view a playback image on the personal computer, with the front direction matching the direction of travel of the vehicle, just like the drive recorder 1, on a display device built into or connected to the personal computer.

The display unit 40 of the drive recorder 1 has a smaller display surface size than display devices used in personal computers. Therefore, it would be convenient for the user if they could view playback images based on video data recorded on the recording media 70 on their personal computer.

According to this embodiment, not only can users view playback images based on video data on their personal computers, but they can also view playback images with a natural front direction adjusted to match the direction of travel of the vehicle, without the need for tedious settings on the personal computer. Therefore, this embodiment can provide users with a convenient drive recorder 1.

In particular, with a rearview mirror type or rearview mirror-mounted drive recorder 1, the orientation of the camera 20 changes relative to the vehicle's straight-ahead direction not only when the drive recorder 1 is mounted but also when the orientation of the rearview mirror is changed, making the functions of the drive recorder 1 of this embodiment extremely useful.

According to this embodiment, it is also advantageous that multiple setting modes for adjusting the front direction of the playback image to match the direction of travel of the vehicle, i.e., multiple operating modes of the setting unit 150, are provided. The setting unit 150 switches setting modes in response to a user's setting mode designation operation, and can accept the user's setting operation in the designated setting mode and generate playback control data.

The setting unit 150 determines the differences Df, Dh, and Dw that indicate the relative relationship between the orientation of the camera 20 and the straight-ahead direction of the vehicle, using the windshield inside the vehicle, the horizon outside the vehicle, and road dividing lines (white lines) outside the vehicle as specific reference objects inside or outside the vehicle that appear in the captured image.

For example, in the "Windshield" setting mode, the setting unit 150 determines the relative relationship based on the distortion of the shape of the windshield inside the vehicle, based on the user pointing at the windshield by tracing the boundary of the windshield through the setting screen. At this time, the display unit 40 displays a screen containing a panoramic image with a full angle of view, different from the 16:9 captured image displayed on the playback screen, as a setting screen separate from the playback screen, to allow the user to perform appropriate setting operations by pointing at the windshield.

In the "Horizon" setting mode, the setting unit 150 determines the relative positional relationship between the horizon outside the vehicle shown in the panoramic image and the bottom edge of the windshield inside the vehicle based on the user's indication of the horizon via a setting screen that also displays a panoramic image different from the playback screen.

In the "white line" setting mode, the setting unit 150 determines the relative relationship based on the arrangement of white lines outside the vehicle that appear in the panoramic image, based on the user's indication of the white lines via a setting screen that also displays a panoramic image different from the playback screen.

The reference object suitable for determining relative relationships varies depending on the scenery captured in the image captured by camera 20. By selecting a setting mode, the user can choose an appropriate reference object and perform the setting operations for the adjustment described above, allowing the image processing unit 130 to align the front direction of the reproduced image with the vehicle's straight-ahead direction with high precision. In particular, the setting screen displays a panoramic image rather than a 16:9 image so that the specified reference object is easily captured, allowing the user to appropriately point to the specified reference object and perform the setting operations for adjustment.

Furthermore, according to this embodiment, the setting unit 150 can determine the relative relationship based on the optical flow of the captured image using the "vector" setting mode, even without using a specific reference object. Therefore, according to this embodiment, even in an environment around the vehicle where a specific reference object is not clearly visible in the captured image, the direction between the forward direction of the reproduced image and the vehicle's traveling direction can be appropriately aligned by driving the vehicle.

Furthermore, the present disclosure is not limited to the above embodiment and can adopt various aspects. For example, the drive recorder 1 may have multiple display modes for displaying playback images. In S145, the control unit 10 may control the display unit 40 to display images captured during the playback target period as playback images on the display unit 40 in a display mode specified by the operation unit 50. In this case, the image processing unit 130 can perform image processing so that the playback images input from the playback unit 120 are displayed in the specified display mode.

For example, when the camera 20 is an omnidirectional camera, the drive recorder 1 can have a standard display mode, which is a display mode that displays a playback image with an aspect ratio of 16:9, as well as a panoramic display mode and a round display mode.

The full-view panoramic display mode is a mode in which the entire captured image recorded in the video image data is displayed in a panoramic view on the display unit 40, with a field of view 2α of 360 degrees (= 2π). In other words, α = β = 180 degrees (= π).

In full view panoramic display mode, as shown in FIG. 11, the image processing unit 130 can convert the playback image input from the playback unit 120 into a playback image for display by treating the image area with an azimuth angle of -π-δθ≦θ<-π of the moving image whose shooting range is in the range of azimuth angles -π-δθ≦θ≦+π-δθ as an image area with an azimuth angle of +π-δθ≦θ<+π.

In other words, the image processing unit 130 performs image processing by changing the position of the corresponding image area in the playback image input from the playback unit 120 so as to move the image area with an azimuth angle of -π - δθ ≦ θ < -π to the image area with an azimuth angle of +π - δθ ≦ θ < +π, thereby generating, as a playback image for display, a captured image with an azimuth angle in the range of -π ≦ θ ≦ +π, with an azimuth angle of zero at the center of the playback image. The image processing by the image processing unit 130 or the playback processing by the playback unit 120 may include processing to suppress image distortion due to lens characteristics, for a good panoramic display.

As a result, the display unit 40 displays a 360-degree panoramic image captured by the camera 20, centered on the vehicle's straight-ahead direction, i.e., a 360-degree panoramic image in which the front direction of the playback image coincides with the vehicle's straight-ahead direction.

In addition, the round display mode, as shown in Figure 12A, is a mode in which the entire 360-degree captured image recorded in the video image data is displayed in a circular shape on the display unit 40. The circumferential direction of the circle corresponds to the azimuth angle direction. Here, the 360-degree captured image displayed in a circular shape is referred to as a round image.

In the round view display mode, the image processing unit 130 processes the playback image input from the playback unit 120 based on the playback control data, and displays a round view image on the display unit 40 in which the subject portion with an azimuth angle of zero, which corresponds to the vehicle's direction of travel, is positioned at the top of the circle. The front direction of the round view image corresponds to the top of the circle.

If the playback control data is not used, the top of the round-view image will not match the direction of travel of the vehicle, and the round-view image will be displayed with the direction of travel of the vehicle tilted from the top of the round-view image by the difference δθ, as shown in Figure 12B.

The image processing unit 130 uses image processing based on the playback control data to rotate such tilted round image by the difference δθ, allowing the display unit 40 to display a round image in which the subject portion with an azimuth angle of zero, corresponding to the vehicle's direction of travel, faces up the circle.

In the above embodiment, image processing was described in a direction that cancels out the difference in azimuth angle δθ between the front direction of the camera 20 and the straight-ahead direction of the vehicle. However, along with the difference in azimuth angle δθ, the difference in elevation/depression angle δφ between the front direction of the camera 20 and the reference direction may also be estimated, and this estimated value may be recorded in the playback control data.

When video data is played back, image processing may be performed to cancel out the difference in azimuth angle δθ and the difference in elevation/depression angle δφ so that a playback image taking the elevation/depression angles into account is displayed. The reference direction here may be a direction parallel to the horizontal plane along the vehicle's straight-ahead direction, or it may be a direction tilted downward from the horizontal plane, taking into account the general line of sight of vehicle occupants.

In the above embodiment, an example was described in which video data is provided to a device outside the vehicle via recording media 70. However, the video data may also be provided to an external device 95, which is a device outside the vehicle, via server device 90, for example. In this case, the playback control data may be included in the video data as metadata, and thereby recorded in storage 91 of server device 90 in a state associated with the main body of the video data. Server device 90 may also be understood to correspond to cloud storage.

In the modified example shown in FIG. 13, assuming that the drive recorder 1 can communicate with the server device 90 via a wide area network, the control unit 10 of the drive recorder 1 writes the generated playback control data as metadata into the video data in S260, and uploads the written video data to the server device 90.

Video data containing playback control data uploaded to server device 90 is downloaded by external device 95. Examples of external devices 95 include personal computers, smartphones, and tablets. When playing back the video data, external device 95 references the playback control data included as metadata and performs image processing that takes into account the discrepancy between the orientation of camera 20 and the direction of travel of the vehicle.

By allowing the drive recorder 1 to provide video data and playback control data to the external device 95 via the server device 90, the user can conveniently view playback images based on the video data on the external device 95 without having to handle the recording media 70.

In particular, the user can view a natural playback image on the external device 95 with the front direction adjusted to match the direction of travel of the vehicle, without any complicated settings. Therefore, this modified example can provide the user with a convenient drive recorder 1.

In the above embodiment, the playback control data was generated by the drive recorder 1, but the video data may be provided to the external device 95 via the server device 90, and the external device 95 may execute processing corresponding to the setting processing shown in FIG. 4 to generate the playback control data. In this case, the playback control data may be recorded on the server device 90 as metadata for the video data.

Video data with playback control data can be downloaded from the server device 90 to the drive recorder 1 or external device 95 and played back. Information processing terminals equipped with touch panels, such as smartphones, offer excellent operability. Therefore, generating playback control data in accordance with user operations via the information processing terminal can improve operability for user settings.

In addition, with regard to the setting process in FIG. 4, the drive recorder 1 may have a manual setting mode in addition to the "vector" setting mode described above. In the manual setting mode, the control unit 10 displays a panoramic image from the camera 20 on the setting screen and can accept an operation from the user to adjust the center position of the panoramic image.

The control unit 10 regards the amount of adjustment of the center position of the adjusted panoramic image from its pre-adjustment position as the deviation between the orientation of the camera 20 and the direction of travel of the vehicle, and generates playback control data.When playing back video data, the control unit 10 can perform image processing based on the playback control data so that the adjusted center position is displayed in the center of the screen.

The technology disclosed herein may be applied not only to drive recorders installed in vehicles as mobile objects, but also to mobile devices equipped with cameras, such as smartphones. While the drive recorder 1 described above has a built-in camera 20, the drive recorder 1 may also be a drive recorder connected by cable or wirelessly to a separate camera installed in the vehicle. The graphical user interface does not need to be superimposed on the playback image or captured image, and may be displayed in an area separate from the playback image or captured image area.

The functions of one component in the above embodiments may be distributed among multiple components. The functions of multiple components may be integrated into one component. Part of the configuration of the above embodiments may be omitted. At least part of the configuration of the above embodiments may be added to or substituted for the configuration of another of the above embodiments. All aspects included in the technical concept identified by the wording of the claims are embodiments of the present disclosure.

1...Drive recorder, 10...Control unit, 11...Processor, 13...Memory, 20...Camera, 30...Sensors, 40...Display unit, 50...Operation unit, 60...Media reader/writer, 70...Recording media, 90...Server device, 91...Storage, 95...External device, 110...Video recording unit, 120...Playback unit, 130...Image processing unit, 150...Setting unit.

Claims (6)

an acquisition unit configured to acquire video image data representing images captured by a camera provided in the vehicle;
a setting unit configured to generate control data for processing the moving image data to display a captured image on the display device, the captured image having a front direction adjusted to the straight-ahead direction of the vehicle, as a playback image based on the moving image data, and to write the control data to a storage device in association with the moving image data;
Equipped with
the setting unit displays, on the display device, a screen for accepting setting operations by a user as a setting screen separate from the playback screen on which the playback image is displayed, accepts, as the setting operation, an operation of pointing, within the captured image displayed on the separate setting screen, at a specific reference object inside or outside the vehicle that appears in the captured image, determines a relative relationship between the orientation of the camera and the straight-ahead direction of the vehicle based on the specific reference object pointed by the setting operation, and generates the control data ;
A playback control system in which the specific reference object includes one of the windshield inside the vehicle and the horizon outside the vehicle that appears in the captured image . 2. The playback control system of claim 1 ,
the setting operation includes an operation of specifying an operation mode of the setting unit,
the setting unit determines the relative relationship based on an operation mode designated by the setting operation among a plurality of operation modes;
The plurality of operation modes include:
an operation mode in which the relative relationship is determined based on a windshield inside the vehicle that appears in the captured image as the specific reference object, and an operation mode in which the relative relationship is determined based on the horizon outside the vehicle that appears in the captured image as the specific reference object. an acquisition unit configured to acquire video image data representing images captured by a camera provided on the vehicle;
a playback control unit configured to control the display device so that a playback image corresponding to the captured image is displayed on a playback screen of the display device based on the moving image data acquired by the acquisition unit; and
a setting unit configured to display, on the display device, a screen for accepting a setting operation by a user as a setting screen separate from the playback screen, accept, as the setting operation, an operation of pointing to a specific reference object inside or outside the vehicle that appears in the captured image displayed on the separate setting screen, determine a relative relationship between the orientation of the camera and the straight-ahead direction of the vehicle based on the specific reference object pointed to by the setting operation, and set an image processing method for the video image data based on the relative relationship;
Equipped with
the playback control unit is configured to process the moving image data in accordance with the image processing method set by the setting unit, thereby causing the display device to display, as the playback image, a captured image whose front direction is adjusted to a straight-ahead direction of the vehicle;
A playback control system in which the specific reference object includes one of the windshield inside the vehicle and the horizon outside the vehicle that appears in the captured image . 4. The playback control system according to claim 3 ,
the setting operation includes an operation of specifying an operation mode of the setting unit,
the setting unit determines the relative relationship based on an operation mode designated by the setting operation among a plurality of operation modes;
The plurality of operation modes include:
an operation mode in which the relative relationship is determined based on a windshield inside the vehicle that appears in the captured image as the specific reference object; and
a playback control system including an operation mode in which the relative relationship is determined based on the horizon outside the vehicle that appears in the captured image as the specific reference object. A computer program for causing a computer to execute a process,
The process comprises:
acquiring video image data representing images captured by a camera installed in a vehicle;
generating control data for performing image processing on the moving image data to display a captured image, the front direction of which is adjusted to the straight-ahead direction of the vehicle, on the display device as a reproduced image based on the moving image data; and
writing the control data to a storage device in association with the video data;
Including ,
Generating the control data includes displaying, on the display device, a screen for accepting setting operations by a user as a setting screen separate from the playback screen on which the playback image is displayed, accepting, as the setting operation, an operation of pointing, within the captured image displayed on the separate setting screen, at a specific reference object inside or outside the vehicle that appears in the captured image, determining a relative relationship between the orientation of the camera and the straight-ahead direction of the vehicle based on the specific reference object pointed by the setting operation, and generating the control data .
The computer program , wherein the specific reference object includes one of the windshield inside the vehicle and the horizon outside the vehicle that appears in the captured image . A computer program for causing a computer to execute a process,
The process comprises:
acquiring video image data representing images captured by a camera installed in a vehicle;
controlling the display device so that a playback image corresponding to the captured image is displayed on a playback screen of the display device based on the acquired moving image data;
displaying, on the display device, a screen for accepting a setting operation by a user as a setting screen separate from the playback screen, and accepting, as the setting operation, an operation of pointing, within the captured image displayed on the separate setting screen, to a specific reference object inside or outside the vehicle that appears in the captured image;
determining a relative relationship between the orientation of the camera and the straight-ahead direction of the vehicle based on the specific reference object indicated by the setting operation;
setting an image processing method for the moving image data based on the relative relationship;
Including ,
controlling the display device includes processing the moving image data in accordance with the set image processing method to display, on the display device, as the playback image, a captured image whose front direction is adjusted to the straight-ahead direction of the vehicle ;
The computer program , wherein the specific reference object includes one of the windshield inside the vehicle and the horizon outside the vehicle that appears in the captured image .