JP7760655B2 - Image processing device and image processing method - Google Patents

Description

The present invention relates to an image processing device and a processing method for processing images.

In recent years, playback devices that allow landscape videos to be captured using a camera mounted on a moving object such as a vehicle and that allow scenes captured from the vehicle to be viewed later have become popular.

For example, Patent Document 1 discloses a vehicle display system that displays on a display an instruction display image including a time bar corresponding to the shooting time from the start to the end of a landscape video captured by an onboard camera mounted on a moving object, a slider that indicates a position on the time bar that can be designated by the user, and landmark icons that indicate, at corresponding positions on the time bar, landmarks that existed along the driving route when the landscape video was captured; and further, displays on the display a landscape video associated with the shooting time that corresponds to the position of the slider on the time bar.

For example, Patent Document 2 discloses a video search and viewing device that extracts still images from video data, generates thumbnail images, and displays them on a timeline at regular time intervals, making it easy to find the part of the video where a desired scene is recorded.

JP 2013-32949 A Japanese Patent Application Laid-Open No. 2002-281432

However, with the display system of Patent Document 1, when trying to find a portion of a video of an external landscape captured from a moving object that contains a desired scene, the user must search for the desired scene while changing the slider position or playing the video from near a landmark icon and visually checking the image displayed on the display. Therefore, if the video is long, it may take some time to reach the desired scene.

Furthermore, with the video image search and viewing device of Patent Document 2, for example, if the moving object's speed is slow and the scenery changes little over a given time period, multiple similar scenes or images will be displayed in thumbnail images taken at regular time intervals, and it may take some time to find a thumbnail image showing the desired scenery.

The present invention has been made in consideration of the above points, and aims to provide an image processing device and image processing method that can efficiently search for desired scenes from landscape videos captured from a moving object.

The invention described in claim 1 comprises an acquisition unit that acquires a landscape video captured from a moving object; a video generation unit that generates an information-containing video by adding importance information to the landscape video based on information about the shooting conditions at each point during the shooting period of the landscape video; and a thumbnail group image generation unit that extracts multiple thumbnail images from multiple frame images that make up the landscape video based on the importance information and generates a thumbnail group image in which the thumbnail images are arranged in chronological order, wherein the thumbnail group image generation unit changes the number of thumbnail images extracted per unit shooting time according to the importance information.

The invention described in claim 10 is an image processing method in which an image processing device extracts thumbnail images from an information-containing video to which importance information has been assigned, and is characterized by including an interval determination step of determining an extraction interval for extracting thumbnail images based on the importance information and the number of thumbnail images to be generated; a frame image determination step of determining, based on the extraction interval, a plurality of frame images to be extracted as thumbnail images from a plurality of frame images constituting the information-containing video; and an extraction step of extracting a plurality of frame images to be extracted as thumbnail images from the information-containing video to generate a thumbnail group image.

The invention described in claim 11 is an image processing program executed by an image processing device provided in a computer, characterized in that the computer is caused to execute an interval determination step of determining an extraction interval for extracting thumbnail images based on importance information and the number of thumbnail images to be generated; a frame image determination step of determining, based on the extraction interval, a plurality of frame images to be extracted as thumbnail images from a plurality of frame images constituting an information-containing video; and an extraction step of extracting a plurality of frame images to be extracted as thumbnail images from the information-containing video to generate a thumbnail group image.

1 is a perspective view of an automobile M equipped with an image processing device 10 according to an embodiment of the present invention. 1 is a view of a rear door of an automobile M equipped with an image processing device 10 according to an embodiment of the present invention, as viewed from a rear seat of the automobile M. FIG. FIG. 2 is a diagram showing an example of an operation acceptance display displayed on a touch panel 17 of the image processing apparatus 10 according to an embodiment of the present application. 1 is a functional block diagram of an image processing apparatus 10 according to an embodiment of the present application. 10 is a flowchart FC1 showing a process of generating an information-containing video by adding a virtual time code to a landscape video captured by an in-vehicle camera CM in the image processing device 10 according to the embodiment of the present application. 10 shows an example of the elapsed time from the start of shooting a landscape video by the image processing device 10 according to an embodiment of the present application, the frame numbers of multiple frames at predetermined intervals, the basic time codes of the multiple frames, and the virtual time codes. A figure showing the relationship between the frame numbers of multiple frames that make up the information-containing video in the example of the information-containing video shown in Figure 6, the virtual time codes assigned to each frame number, and the relationship between the thumbnail images that are extracted. In the ten thumbnail images SM1 to SM10 extracted from the scenic video in this embodiment, the time code, frame number, and importance associated with each thumbnail image SM1 to SM10 are shown. 10 is a flowchart FC2 showing the process of generating thumbnail images by the image processing apparatus 10 according to the embodiment of the present application. 9A to 9C are diagrams showing examples of displays on the touch panel 17 when the information-containing moving images shown in FIGS. 6, 7, and 8 are played back. 9A to 9C are diagrams showing examples of displays on the touch panel 17 when the information-containing moving images shown in FIGS. 6, 7, and 8 are played back.

Examples of the present invention are described in detail below.

Figure 1 is a perspective view of an automobile M equipped with an image processing device 10 according to an embodiment of the present application.

The image processing device 10 is an image processing device that captures a video of the scenery around the automobile M using an onboard camera installed in the automobile M and plays back the captured video of the scenery. In the following explanation, we will describe a configuration that can display the video of the scenery on the window of the automobile M.

The image processing device 10 is mounted, for example, on the dashboard of the automobile M. The image processing device 10 is configured to be able to play and output video that is acquired from an external source or that is stored within the image processing device 10.

The on-board camera CM serving as the acquisition unit is mounted, for example, on the roof or door panel of the automobile M, and is configured to be able to capture images of the scenery in front of, to the side of, or behind the automobile M. The on-board camera CM is communicatively connected to the image processing device 10, and is configured to be able to transmit captured still images or videos to the image processing device.

The display 15 serving as the video display unit is composed of a transparent panel, such as a transparent liquid crystal or organic electroluminescence display, and is attached to the window WD of the automobile M, capable of displaying images visible from inside the vehicle. The display 15 is connected to the image processing device 10 so that it can communicate with the image processing device 10, and is configured to be able to display still images or videos output from the image processing device 10.

For example, the display 15 is configured to display video of scenery in front of, to the side of, or behind the automobile M captured by the onboard camera CM on the front window, door windows, and rear window, respectively. The display 15 may also be, for example, a display for a car navigation system installed inside the automobile M.

Figure 2 is a view of the rear door of an automobile M equipped with an image processing device 10 according to an embodiment of the present application, viewed from the rear seat of the automobile M.

As described above, the display 15 is composed of a transparent panel, such as a transparent liquid crystal or organic electroluminescence display, and is attached to the window WD of the vehicle M. Therefore, for example, when no video is being displayed on the display 15, it is possible to see the outside scenery from inside the vehicle M through the display 15.

The touch panel 17 serving as the operation reception unit is a touch panel display attached to the inside of the door panel. For example, the touch panel 17 is a touch panel display that includes a display 17a such as a liquid crystal panel, and a touch pad 17b that is attached to the display surface of the display and receives operations using a person's finger, etc.

The touch panel 17 is communicatively connected to the image processing device 10 and is capable of displaying images output from the image processing device 10. The touch panel 17 is also capable of accepting operations to play, fast-forward, or rewind the video displayed on the display 15, and transmitting instructions based on the accepted operations to the image processing device 10.

Specifically, for example, the touch panel 17 displays thumbnail images of the video being played on the display 15 for each predetermined time period, arranged in order of playback time.

Figure 3 shows an example of an operation acceptance display displayed on the touch panel 17 of the image processing device 10 according to an embodiment of the present application.

The operation acceptance display includes, for example, a thumbnail image string SR in which thumbnail images SM1 to SM6 of the scenery video being played on the display 15 are arranged in chronological order for playback, a button PL for switching between playback and pause, a button FW for instructing a fast-forward operation, a button RW for instructing a fast-rewind operation, a bar BA indicating the current playback time position of the scenery video, and a time display unit TM. Furthermore, the thumbnail images included in the thumbnail image string SR are displayed while scrolling from right to left as the playback time of the scenery video elapses.

Figure 4 is a functional block diagram of an image processing device 10 according to an embodiment of the present application. The image processing device 10 is a device in which an input unit 20, an output unit 30, an operation input unit 40, a memory unit 50, and a video control unit 60 cooperate with each other via a system bus 13, for example.

The input unit 20 is an interface that acquires data from devices external to the image processing device 10. The input unit 20 is connected to an on-board camera CM installed in the automobile M. For example, the image processing device 10 can capture landscape images from the on-board camera CM via the input unit 20.

The input unit 20 is also connected to a speed sensor SS that acquires the vehicle's traveling speed, and a GPS receiver GP that can receive information from the Global Positioning System (GPS), a Global Navigation Satellite System (GNSS) that acquires the vehicle's position information. In other words, the image processing device 10 is configured to be able to acquire the speed of the vehicle M from the speed sensor SS and to acquire GPS information from the GPS receiver GP.

The output unit 30 is an interface that outputs data to devices external to the image processing device 10. The output unit 30 is connected to the above-mentioned display 15 and display 17a of the touch panel 17, and is capable of outputting images to these. Specifically, for example, the image processing device 10 outputs a video to the display 15 via the output unit, and outputs an operation image that serves as an operation acceptance display for playback operations to the display 17a.

The operation input unit 40 is an interface that acquires operation input signals from external devices. The operation input unit 40 is connected to the touchpad 17b and can acquire signals indicating operations accepted in response to the operation image on the touchpad 17b.

The storage unit 50 is composed of, for example, a hard disk drive, SSD (solid state drive), flash memory, etc., and stores various programs related to the generation of thumbnail images, video playback, etc. in the image processing device 10. Note that the various programs may be obtained, for example, from another server device, etc. via a network (not shown), or may be recorded on a recording medium and read via various drive devices.

The various programs stored in the memory unit 50 can be transmitted via a network, or recorded on a computer-readable recording medium and transferred. The memory unit 50 can also store data imported from outside the image processing device 10, such as landscape video data imported from the camera CM, and the speed and position information of the automobile M when capturing the landscape video imported from the speed sensor SS and GPS receiver GP. The memory unit 50 can also store data generated within the image processing device 10.

The video control unit 60 is composed of a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc., and functions as a computer. In the video control unit 60, the CPU realizes various functions by reading and executing various programs stored in the ROM and storage unit 50.

The video control unit 60 has a video generation unit 63 that adds information to the scenery video acquired via the input unit 20 to generate an information-containing video for playback. The video generation unit 63 reads out the scenery video recorded in the storage unit 50 and data on the speed and position information of the automobile M at each point in time when the scenery video was captured. For each of the multiple frame images that make up the read scenery video, information based on the speed or position information of the automobile M at each point in time when each of the multiple frames was captured is added. For example, the video generation unit 63 determines the importance of each frame image according to the traveling speed of the automobile M, and adds importance information based on that importance to the scenery video.

The importance is determined based on, for example, the traveling speed of the automobile M, the shooting location, the visual appeal of the scenery associated with the position on the map information corresponding to the shooting location, or the shooting direction taken by the in-vehicle camera CM relative to the traveling direction of the automobile M.

For example, when the automobile M is traveling at a low speed or is stopped, the surrounding scenery seen from the automobile changes very little. Therefore, when determining the importance based on the automobile's traveling speed, the video generation unit 63 determines that the importance of the scenery when the automobile M is traveling at a low speed or is stopped is low. Conversely, when the automobile is traveling at a high speed, the scenery video changes significantly in a short period of time, so the importance is determined to be high.

Furthermore, the video generation unit 63, for example, links the vehicle's GPS position information with a pre-recorded map database and determines that the vehicle is of high importance when it is located in a position with a high scenic spot rank on the map database. Conversely, in the case of a scenery video in which the vehicle M is located in a position with a low scenic spot rank on the map database, the video generation unit 63 determines that the vehicle is of low importance.

In addition, the importance of location information may be determined without linking it to a map database. For example, in mountainous areas at high altitudes, the scenery may be very impressive. Therefore, the importance may be determined to be high when positioning within a specified location range is detected from positioning information such as GPS.

Furthermore, in scenery videos taken while the automobile M is traveling, scenery videos taken to the side in the direction of travel change significantly over a short period of time compared to scenery videos taken in front or behind the automobile M's direction of travel. Therefore, the video generation unit 63 determines that scenery videos taken to the side in the direction of travel are more important than scenery videos taken in front or behind the automobile M's direction of travel.

Furthermore, if the GPS receiver GP is unable to acquire GPS information for a predetermined period of time or longer, it is highly likely that the automobile M is traveling through a structure such as a tunnel, and in such cases, the scenery video will continue to be monotonous. Therefore, the video generation unit 63 determines that the importance of scenery videos when GPS information is not received is lower than that of scenery videos when GPS information is received.

The above-mentioned criteria for determining importance can be set arbitrarily. Furthermore, the criteria for determining importance are not limited to one of the following: the traveling speed of the automobile M, the shooting location, the visual appeal of the scenery associated with the position on the map information corresponding to the shooting location, or the shooting direction taken by the shooting unit relative to the traveling direction of the automobile M. Importance may be determined based on multiple criteria.

In other words, the video generation unit 63 determines the importance based on the moving speed of the automobile M, the shooting location, the visual appeal of the scenery associated with the position on the map information corresponding to the shooting location, the shooting direction based on the moving direction of the automobile M, or the radio wave reception status of the satellite system.

Note that importance is not limited to the speed or position when the video was shot. For example, by comparing each of the multiple frame images that make up the video, frames with large image changes between them may be determined to have high importance.

Furthermore, for example, the video generation unit 63 generates an information-containing video that is assigned a virtual time code that indicates the passage of time different from the time code (hereinafter referred to as the basic time code) that indicates the actual passage of time during shooting of the landscape video, based on the level of importance.

The virtual time code is calculated from the average importance value for each specified time period, and is importance information in a format that reflects the importance over virtual time.

Specifically, the video generation unit 63 assigns a basic time code and a virtual time code including different time lapses to each of a plurality of frame images of the scenery video for each predetermined time period, according to the average importance value calculated for each predetermined time period. In other words, the video generation unit 63 functions as an importance information acquisition unit that generates importance and a virtual time code from the scenery video and data on the speed or position of the automobile M at the time the scenery video was captured. It also functions as a video generation unit that assigns the generated virtual time code as importance information to the scenery video to generate an information-containing video.

The virtual time code is set so that the higher the importance, the closer it is to the time progression of the basic time code, and the lower the importance, the earlier it is to the time progression of the basic time code.

In addition, the video generation unit 63 can arbitrarily set the specified time period for calculating the average importance level.

The information-containing video generated by the video generation unit 63 is stored in the storage unit 50. The importance information assigned to the video may include one or more of the following: the moving speed of the automobile M, the shooting location, the visual appeal of the scenery associated with the position on the map information corresponding to the shooting location, the shooting direction based on the moving direction of the automobile M, the importance based on the radio wave reception status of the satellite system, and a virtual time code.

The thumbnail generation unit 65, which serves as a thumbnail group image generation unit, extracts frame images at predetermined time intervals based on the virtual time code and the number of thumbnail images generated from the multiple frame images that make up the information-containing video recorded in the storage unit 50, and generates thumbnail images to be displayed in the thumbnail display area SR of the touch panel 17. The thumbnail generation unit 65 supplies the generated thumbnail images to the display 17a via the output unit 30, where they are displayed in the thumbnail image string SR.

The number of thumbnail images to be generated can be set arbitrarily. For example, a predetermined number can be set for each landscape image, or the number can be set automatically based on the recording time of the landscape video.

The playback control unit 67 reads out information-containing videos recorded in the storage unit 50 and controls playback of the information-containing videos in response to instructions to play, pause, fast-forward, or fast-rewind. The playback control unit 67 supplies the played information-containing videos to the display 15 via the output unit 30. The playback control unit 67 also supplies an operation image corresponding to the information-containing video being played to the display 17a via the output unit 30. The playback control unit 67 acquires an operation signal received on the touchpad 17b and controls playback of the videos based on the operation signal.

In other words, it comprises an in-vehicle camera CM that acquires scenery videos shot from a moving body, a video generation unit 63 that generates an information-containing video by assigning a virtual time code to the scenery video based on information about the shooting conditions at each point during the shooting period of the scenery video, and a thumbnail generation unit 65 that extracts multiple thumbnail images from multiple frame images that make up the scenery video based on the virtual time code and generates a thumbnail group image in which the thumbnail images are arranged in chronological order, and the thumbnail generation unit 65 changes the number of thumbnail images extracted per unit shooting time according to the virtual time code.

The image processing device 10 also includes a video generation unit 63 that acquires information about the shooting conditions at each point during the shooting period of the landscape video; a storage unit 50 that records the information-containing video generated by the video generation unit 63; a playback control unit 67 that reads the information-containing video from the storage unit 50 and controls playback, fast-forwarding, or fast-rewinding of the landscape video to output the image; and a touch panel 17 that displays a thumbnail image string SR in which thumbnail images generated by the thumbnail generation unit 65 are arranged in playback time order, and accepts instruction operations including an operation to select a thumbnail image included in the thumbnail image string SR or an operation to scroll the thumbnail image string SR, and the playback control unit 67 performs control based on the instruction operations.

The video generation unit 63 also generates an information-containing video by assigning, as importance information, a virtual time code, which is a series of virtual time codes based on information about the shooting conditions when each of the multiple frame images was shot, to each of the multiple frame images that make up the shot scenery video, and the thumbnail generation unit 65 extracts thumbnail images based on the virtual time codes.

Note that thumbnail images may be the extracted frame images themselves or reduced versions of the extracted frame images. Below, both the process of extracting frame images from a landscape video and using them as thumbnail images as they are, and the process of reducing the extracted frame images to generate thumbnail images will be described as simply extracting thumbnail images.

The video generation unit 63 also determines the importance of each of the multiple frame images based on information about the shooting conditions, and assigns virtual time codes corresponding to that importance to the multiple frame images.

In addition, the video generation unit 63 assigns virtual time codes so that the lower the importance, the faster the time indicated by the virtual time codes in a group of frame images passes.

Figure 5 shows a flowchart FC1 in which an image processing device 10 according to an embodiment of the present application generates an information-containing video by adding a virtual time code to a landscape video captured by an in-vehicle camera CM. Here, an example is described in which the importance of a landscape image is set according to the traveling speed of the automobile M.

First, in step S101, the video generation unit 63 determines whether filming of the landscape video has finished (step S101). This determination may be made, for example, by accessing the camera CM and checking whether video filming has finished. This determination may also be made, for example, by checking whether reception of video data from the camera CM has finished.

If it is determined that the shooting of the scenery video has ended (step S101: Yes), the video generation unit 63 reads the scenery video recorded in the memory unit 50 and the traveling speed of the automobile M acquired by the speed sensor SS during the shooting period (step S102).

Next, the video generation unit 63 assigns to each of the multiple frame images that make up the read landscape video the traveling speed of the automobile M at the time the frame was captured (step S103).

The video generation unit 63 determines the importance level corresponding to the traveling speed of the automobile M for each of the multiple frames constituting the scenery video to which the traveling speed of the automobile M has been assigned, and further assigns this importance level to the scenery video (step S104).

Next, the video generation unit 63 divides the landscape video, in which importance levels have been assigned to each frame, into predetermined time intervals and calculates the average importance level for each predetermined time interval (step S105).

Then, the video generation unit 63 assigns a virtual time code that includes a time lapse that differs from the basic time code, based on the average importance value for each predetermined time period (step S106).

After step S106 is completed, the video generation unit 63 records the landscape video to which the virtual time code has been assigned as an information-containing video in the storage unit 50 (step S107).

By performing the above process, an information-containing video is generated that is assigned importance and a virtual time code based on the landscape video captured by the in-vehicle camera CM and the traveling speed of the vehicle M acquired by the speed sensor SS.

In this embodiment, it has been explained that the assignment of the traveling speed of the automobile M, the assignment of importance, and the assignment of a virtual time code to the scenery video are performed after the scenery video is shot.

However, the travel speed of the automobile M, the importance level, and the virtual time code may be assigned to the scenery video while the scenery video is being filmed. When this is done while filming is being done, the travel speed of the automobile M at the time of that frame image is assigned, and the importance level is determined and assigned each time a frame image of the scenery video is supplied. In addition, each time the recording time of the scenery video exceeds the specified time for calculating the average importance level, the average importance level is calculated and a virtual time code is assigned.

Next, we will explain the virtual time code assigned to information-containing videos using Figure 6.

Figure 6 shows an example of the elapsed time from the start of landscape video shooting by the image processing device 10 according to an embodiment of the present application, the frame numbers of multiple frames per predetermined time, and the basic time codes of multiple frames. It also shows the average importance per predetermined time calculated by the video generation unit 63, the multiplication factor of the virtual time code with respect to the basic time code, and the assigned virtual time code.

In this example, we will explain the case where a 600-second landscape video shot by an in-car camera commercial is taken at a shooting frame rate of 30 fps, the predetermined time for calculating the average importance level is 60 seconds, and the importance level is divided into four levels (0 to 3) depending on the driving speed.

Furthermore, the virtual time code multiplier will be described as being set so that the virtual time code is assigned at a time code interval equal to the basic time code for importance level 3, 1/2 the basic time code for importance level 2, 1/3 the basic time code for importance level 1, and 1/5 the basic time code for importance level 0.

The virtual time code progression for basic time code and importance level 3 progresses in accordance with the frame rate of the captured landscape video, and therefore progresses at a speed of 1/30 second (approximately 0.033 seconds) per frame. Furthermore, the virtual time code progression for importance level 2 progresses at a speed of 1/60 second (approximately 0.017 seconds) per frame, the virtual time code progression for importance level 1 progresses at a speed of 1/90 second (approximately 0.011 seconds) per frame, and the virtual time code progression for importance level 0 progresses at a speed of 1/150 second (approximately 0.007 seconds) per frame.

In other words, importance level 3 assigns a virtual time code whose time passes at the same rate as the basic time code, importance level 2 assigns a virtual time code whose time passes at approximately twice the rate of the basic time code, importance level 1 assigns a virtual time code whose time passes at approximately three times the rate of the basic time code, and importance level 0 assigns a virtual time code whose time passes at approximately five times the rate of the basic time code.

In Figure 6, the average calculated importance level is 3 for all frames between 0 and 180 seconds elapsed since the start of shooting. Therefore, the virtual time code is assigned at the same rate as the basic time code, 1/30 second per frame.

Furthermore, the average calculated importance level between 180 and 300 seconds after the start of shooting is 1. Therefore, the virtual time code is assigned at a rate of 1/90 seconds per frame, which is one-third the speed of the basic time code. In other words, for the 120 seconds between 180 and 300 seconds after the start of shooting, the basic time code progresses at the same rate as the actual shooting time during this period, but the virtual time code is assigned at a rate of 40 seconds per frame, relative to the actual shooting time of 120 seconds during this period.

Furthermore, the average calculated importance level between 300 and 420 seconds after the start of shooting is 2. Therefore, the virtual time code is assigned at a rate of 1/60 seconds per frame, which is half the rate of the basic time code. In other words, for the 120 seconds between 300 and 420 seconds after the start of shooting, the virtual time code is assigned at a rate of 60 seconds per frame for the 120 seconds of actual shooting time during this period.

Furthermore, the average calculated importance level between 420 and 480 seconds after the start of shooting is 0. Therefore, the virtual time code is assigned at a rate of 1/150 seconds per frame, which is 1/5 the speed of the basic time code. In other words, for the 60 seconds between 420 and 480 seconds after the start of shooting, the virtual time code is assigned at a rate of 12 seconds per frame for each of the 60 seconds of actual shooting time during this period.

Furthermore, the average calculated importance between 480 and 600 seconds after the start of shooting is 2. The progression speed of the virtual time code during this time is the same as in the case of importance 2 above, so explanation is omitted.

In other words, in a 600-second landscape video shot at 30 fps by an in-car camera commercial, the basic time code is constant at 1/30 second per frame, so the basic time code is assigned a time code ranging from 0 to 599.967 seconds for frame numbers 1 to 18000. In contrast, the virtual time code's progression speed changes depending on the importance of each given time period, so the virtual time code is assigned a time code ranging from 0 to 351.967 seconds for frame numbers 1 to 18000.

Figure 7 shows the relationship between the frame numbers of the multiple frames that make up the information-containing video in the example of the information-containing video shown in Figure 6, the virtual time codes assigned to each frame number, and the relationship between the extracted thumbnail images.

The horizontal axis shows the frame numbers of the multiple frames that make up the information-containing video. It also shows the elapsed time from the start of shooting (basic time code) that corresponds to each frame number. The vertical axis also shows the virtual time code, which corresponds to the frame number on the horizontal axis.

In this example, we will explain the case where 10 thumbnail images are extracted from the information-containing video shown in Figure 6.

As shown in Figure 7, the section from frame numbers 1 to 5400 (0 to 180 seconds elapsed since the start of shooting) has a calculated importance of 3, so the virtual time code per frame is assigned with the same elapsed time as the basic time code.

The section between frame numbers 5401 and 9000 (180 to 300 seconds elapsed from the start of shooting) has a calculated importance of 1, so the virtual time code for each frame is assigned at 1/3 the elapsed time of the basic time code.

The section between frame numbers 9001 and 12600 (300 to 420 seconds elapsed from the start of shooting) has a calculated importance of 2, so the virtual time code for each frame is assigned at half the elapsed time of the basic time code.

The calculated importance for the section between frame numbers 12601 and 14400 (420 to 480 seconds elapsed from the start of shooting) is 0, so the virtual time code for each frame is assigned an elapsed time that is 1/5 of the basic time code.

The section from frame numbers 14401 to 18000 (480 to 600 seconds elapsed from the start of shooting) has a calculated importance of 2, so the virtual time code per frame is assigned with an elapsed time that is half the basic time code.

As a result, for the information-containing video in this embodiment, the relationship between the multiple frames that make up the information-containing video and the virtual time code is shown as the solid line in the figure.

The virtual time code is assigned so that the lower the importance, the faster the virtual time code progresses. Therefore, as shown in Figure 7, the lower the importance, the shorter the time that passes per frame, and the smaller the slope of the virtual time code.

The thumbnail generation unit 35 extracts frame images for each specified time period of the virtual time code as thumbnail images based on the virtual time code.

When generating 10 thumbnail images from a landscape video in this embodiment, the virtual time code of the information-holding video is assigned a time code of 0 to 351.967 seconds, so the thumbnail generation unit 65 extracts 10 frame images approximately every 35.2 seconds starting from 0 seconds on the virtual time code. In other words, the thumbnail generation unit 65 extracts frame images at regular intervals of approximately 35.2 seconds on the vertical axis in the diagram.

The frame images extracted as thumbnail images are selected at regular intervals of approximately 35.2 seconds from the virtual time code starting at 0 seconds, 35.2 seconds, 70.4 seconds, ..., 281.6 seconds, and 316.8 seconds. Because the virtual time code's progression speed changes depending on the importance of each specified time period, the basic time code and frame numbers of the extracted thumbnail images are not set at regular intervals.

Therefore, as shown in Figure 7, the lower the importance, the fewer thumbnail images are extracted per unit shooting time.

In other words, according to the present invention, the progression speed of the virtual time code is changed according to the importance of each predetermined time period, making it possible to change the number of thumbnail images extracted per unit shooting time according to that importance.

Figure 8 shows the time codes, frame numbers, and importance associated with the ten thumbnail images SM1 to SM10 extracted from the landscape video in this example.

The top row of Figure 8 shows, as a comparative example, information about thumbnail images generated from frame images extracted based on a conventional basic time code, while the bottom row of Figure 8 shows information about thumbnail images generated from frame images extracted based on a virtual time code according to the present invention. The bottom row of Figure 8 also shows the basic time code corresponding to the frame number of the thumbnail image extracted based on the virtual time code.

As shown in the upper part of Figure 8, when extracting thumbnail images based on the basic time code, the thumbnail generation unit 65 extracts frame images at fixed intervals of 60 seconds when the basic time code is 0 seconds, 60 seconds, 120 seconds, ..., 480 seconds, and 540 seconds as thumbnail images. The frame numbers of the extracted thumbnail images are 1, 1801, 3601, ..., 14401, 16201, for fixed intervals of 1 to 1800. The importance of the extracted frame images is 3 for the first to third frame images, 1 for the fourth and fifth frame images, 2 for the sixth and seventh frame images, 0 for the eighth frame image, and 2 for the ninth and tenth frame images.

Furthermore, as shown in the lower part of Figure 8, when extracting thumbnail images based on the virtual time code, the thumbnail generation unit 65 extracts frame images at regular intervals of 35.2 seconds when the virtual time code is 0 seconds, 35.2 seconds, 70.4 seconds, ..., 281.6 seconds, and 316.8 seconds as thumbnail images. The frame numbers of the extracted thumbnail images are frame numbers 1, 1056, 2112, 3168, 4224, 5280, 8211, 10586, 12845, and 15890. The importance levels corresponding to the extracted frame images are as follows: the importance level for the first to sixth frame images is 3, the importance level for the seventh frame image is 1, the importance level for the eighth frame image is 2, the importance level for the ninth frame image is 0, and the importance level for the tenth frame image is 2.

In other words, the thumbnail generation unit 65 extracts fewer thumbnail images from time segments with low importance determined to have few highlights within the captured landscape video, and extracts more thumbnail images from time segments with high importance determined to have many highlights.

Furthermore, as shown in the basic time code corresponding to the frame images of thumbnail images extracted based on the virtual time code, the intervals between the shooting times of extracted thumbnail images are longer in sections of low importance, and shorter in sections of high importance. In other words, the time intervals between extractions of thumbnail images or frame images used to generate those thumbnail images are longer in sections of low importance, and the time intervals between extractions of thumbnail images or frame images used to generate those thumbnail images are shorter in sections of high importance.

This allows many thumbnail images of highly important and desired scenes from the captured scenery video to be extracted, making it possible to efficiently search for desired scenes from the captured scenery video.

Instead of setting the importance of a scenic image according to the traveling speed of the automobile M, the video generation unit 63 may set the importance of a scenic image based on the shooting location, the scenic beauty associated with the position on the map information corresponding to the shooting location, the shooting direction based on the direction of movement of the automobile M, or the radio wave reception status of the satellite system. Alternatively, the video generation unit 63 may set the importance of a scenic image based on a combination of multiple elements from among the traveling speed, the shooting location, the scenic beauty associated with the position on the map information corresponding to the shooting location, the shooting direction based on the direction of movement of the automobile M, and the radio wave reception status of the satellite system.

Figure 9 shows a flowchart FC2 for generating thumbnail images using the image processing device 10 according to an embodiment of the present application.

First, the playback control unit 67 determines whether a command to play a landscape video has been issued. This determination may be made, for example, based on whether a command to play a landscape video has been accepted on the touch panel 17 (step S201).

Next, the playback control unit 67 reads the information-containing video for which playback has been instructed, which is recorded in the storage unit 50 (step S202).

In the interval determination process, the thumbnail generation unit 65 determines the extraction interval of the virtual time code for extracting thumbnail images from the virtual time code of the information-containing video read by the playback control unit 67 and the number of thumbnail images to be generated (step S203).

Next, as a frame image determination process, the thumbnail generation unit 65 determines the frame number to be used as the thumbnail image from the thumbnail image extraction interval of the determined virtual time code (step S204).

Next, as an extraction step, the thumbnail generation unit 65 extracts the frame image of the determined frame number as a thumbnail image (step S205).

The thumbnail generation unit 65 then arranges the extracted thumbnail images in chronological order and supplies them to the output unit 30 (step S206).

After completing the processing of step S206, the thumbnail generation unit 65 ends the processing, and the playback control unit 67 then performs the playback processing of the information-containing video.

The above process makes it possible to generate a group of thumbnail images that contains many thumbnail images of sections with high importance and few thumbnail images of sections with low importance.

In other words, the image processing method of the present invention is an image processing method in which the image processing device 10 extracts thumbnail images from an information-holding video to which a virtual time code has been assigned, and includes an interval determination step of determining an extraction interval for extracting thumbnail images based on the virtual time code and the number of thumbnail images to be generated, a frame image determination step of determining a plurality of frame images to be extracted as thumbnail images from a plurality of frame images constituting the information-holding video based on the extraction interval, and an extraction step of extracting a plurality of frame images to be extracted as thumbnail images from the information-holding video to generate a thumbnail group image.

Figures 10A and 10B show examples of the display on the touch panel 17 when the information-containing video shown in Figures 6, 7, and 8 is being played.

Figure 10A shows the operation acceptance display on the touch panel 17 when playing a section with an importance level of 3, and Figure 10B shows the operation acceptance display on the touch panel 17 when playing a section with an importance level of 1. The length of the arrows in the figures indicates the scrolling speed. The times displayed in the lower left corner of thumbnail images SM1 to SM10 are the playback times (basic time codes) corresponding to the frames of each of the displayed thumbnail images SM1 to SM10.

The playback control unit 67 controls the playback speed, the playback time display in the time display unit TM, and the scrolling speed of the thumbnail image sequence SR during playback based on the basic time code.

For example, the scrolling speed during playback of the section of thumbnail image SM3 in Figure 10A, which has an importance level of 3, is approximately 35 seconds, from the beginning of thumbnail image SM3, which has a basic time code of 70.4 seconds, to the beginning of thumbnail image SM4, which has a basic time code of 105.6 seconds.

Furthermore, the scrolling speed during playback in the section of thumbnail image SM6, which has an importance level of 1 in Figure 10B, is approximately 79 seconds from the beginning of thumbnail image SM7, which has a basic time code of 273.7 seconds, to the beginning of thumbnail image SM8, which has a basic time code of 352.9 seconds.

In other words, when the original frame image of thumbnail image SM4 is displayed as a still image that makes up the video being displayed on display 15, the scrolling speed of the thumbnail image series SR is controlled so that thumbnail image SM4 approaches the bar BA that indicates the current playback time position, and when the original frame image of thumbnail image SM8 is displayed as a still image that makes up the video being displayed on display 15, the scrolling speed of the thumbnail image series SR is controlled so that thumbnail image SM8 approaches the bar BA that indicates the current playback time position.

In other words, the playback time (basic time code interval) of a video corresponding to a thumbnail image with low importance is longer than the playback time corresponding to a thumbnail image with high importance. Therefore, the scrolling speed of the thumbnail image string SR is slower when playing a video with low importance, and the scrolling speed of the thumbnail image string SR is faster when playing a video with high importance.

Furthermore, in sections of low importance, the thumbnail images displayed in the thumbnail image string SR are displayed in small numbers at long intervals relative to the recording time of the captured scenery video, and in sections of high importance, the thumbnail images are displayed in large numbers at short intervals relative to the recording time of the captured scenery video.

When a user searches for a desired scene from a captured landscape video, the preferred shooting time interval between multiple thumbnail images displayed as the thumbnail image sequence SR will differ depending on whether the user has an idea of where in the landscape video the desired scene is recorded or not. In the former case, a thumbnail image sequence SR with thumbnail images from around the already-estimated shooting time and with a relatively short shooting time interval between thumbnail images will be effective. In the latter case, it is first necessary to have an idea of where in the landscape video the desired scene is recorded, so a thumbnail image sequence SR with a relatively long shooting time interval between thumbnail images will be effective.

To accommodate this, it is preferable that the image processing device 10 be configured to allow the user to change the shooting time interval between adjacent thumbnail images displayed in the thumbnail image string SR as part of the operation reception display.

Specifically, the image processing device 10 displays an operation icon that allows the user to specify the time scale of the thumbnail image sequence SR, included in the operation acceptance display on the touch panel 17, and the thumbnail generation unit 65 determines the extraction interval of the virtual time code for extracting thumbnail images to be displayed as the thumbnail image sequence SR, according to the time scale specified by the user's operation. This makes it possible to arbitrarily change the time scale of the thumbnail image sequence, thereby generating a thumbnail image sequence that is effective when searching for a desired scene within a limited time range, or when searching for a desired scene within a relatively long time range.

In the image processing device 10 of the above embodiment, a method was described in which frame images were extracted from an information-containing video after a playback instruction was given, and thumbnail images were generated. However, thumbnail image generation is not limited to this. For example, after shooting of a landscape video is completed and the video generation unit 63 generates an information-containing video by assigning a virtual time code to the landscape video (steps S101 to S106 in FIG. 5), the thumbnail generation unit 65 may then perform the processing of steps S202 to S205 in FIG. 9 on the information-containing video, thereby recording the information-containing video and thumbnail images in the storage unit 50. In this way, by generating and recording thumbnail images in advance, it is possible to quickly execute the playback process of the landscape video during playback.

In the embodiment, an image processing device 10 was described in which the importance of a scenic video captured by an onboard camera CM mounted on a vehicle M is determined based on the traveling speed and shooting position of the vehicle M, and the thumbnail generation unit 65 variably changes the number of thumbnail images generated per unit shooting time based on the determined importance. However, the image processing device according to the present invention is not limited to scenic videos captured from a vehicle.

For example, it is possible to generate videos by adding importance to videos shot with a smartphone or handheld camera. Furthermore, the shot information-containing video may be recorded on an external recording device such as a USB memory stick or HDD, and then processed and played back on a device such as a PC.

In addition, in this embodiment, a method has been described in which an average value of importance is calculated for each predetermined section of the basic time code and a virtual time code is assigned to each predetermined section, but the method of assigning virtual time codes is not limited to this. For example, for each of the multiple frames that make up a landscape video to which importance has been assigned, the passage of time for each frame may be assigned according to the importance. This makes it possible to efficiently generate thumbnail images of sections with high importance even if the importance changes significantly within a predetermined section of the basic time code.

Furthermore, in this embodiment, the image processing device 10 has been described as generating one piece of video data by assigning a virtual time code to a landscape video and performing video processing on the video data. However, the playback control method is not limited to generating one piece of video data. For example, a landscape video to which importance and a virtual time code have not been assigned may be generated, along with information data indicating frame numbers and virtual time codes corresponding to the frame numbers from the landscape video. The thumbnail generation unit 65 may determine the frame number to extract as a thumbnail image from the information data, and extract the frame image of that frame number from the landscape video as a thumbnail image. This makes it possible to reduce the file data size of the landscape video.

Furthermore, in this embodiment, a method has been described in which importance is determined from the traveling speed of automobile M and a virtual time code is generated from that importance, but the method of generating a virtual time code is not limited to this. For example, multiple thresholds may be set for the average traveling speed of automobile M for each specified time period, and virtual time codes may be set according to the multiple thresholds for the average traveling speed. This makes it possible to generate a virtual time code directly from the traveling speed of automobile M for each specified time period.

In other words, it would be possible to assign importance information indicating specific highlights to each of the multiple frame images that make up the captured image, and variably change the number of thumbnail images generated per unit of shooting time depending on that importance.

10 Image processing device 15 Display 17 Touch panel 20 Input unit 30 Output unit 40 Operation input unit 50 Storage unit 60 Video control unit 63 Video generation unit 65 Thumbnail generation unit 67 Playback control unit

Claims (12)

an acquisition unit that acquires video information to which importance information is assigned for each frame image that constitutes the video;
an extracting unit that extracts a part of the frame images from the video information as thumbnail images;
a thumbnail image generating unit that generates a group of thumbnail images by arranging the plurality of thumbnail images extracted by the extracting unit in time series;
Equipped with
The image processing device is characterized in that the extraction unit determines the number of thumbnail images to be generated, which is set in advance, based on user operation, or based on the recording time of the video, and the extraction interval for extracting the thumbnail images based on the importance information . 2. The image processing device according to claim 1,
The image processing device is characterized in that the extraction unit determines the extraction interval so that the extraction time interval of the thumbnail images corresponding to sections of the video with high importance indicated by the importance information is shorter than the extraction time interval of the thumbnail images corresponding to sections with low importance. 3. The image processing device according to claim 1,
The image processing device is characterized in that the extraction unit determines the extraction interval so that the higher the importance indicated by the importance information, the greater the number of thumbnail images extracted per unit shooting time of the video. 4. The image processing device according to claim 1,
The image processing device is characterized in that the extraction unit determines the extraction interval based on a virtual time code that indicates the passage of time, which is assigned to each frame image as the importance information and is different from the time code that indicates the actual passage of time when the video was shot. 5. The image processing device according to claim 4 ,
An image processing device characterized in that the virtual time code is assigned so that the lower the importance level indicated by the importance information in the video, the faster the time passes compared to the actual time passing when the video was shot. 6. The image processing device according to claim 1,
10. An image processing device according to claim 9, wherein the importance indicated by the importance information is determined based on a moving speed of a shooting device for the video. 6. The image processing device according to claim 1,
10. An image processing device according to claim 9, wherein the importance indicated by the importance information is determined based on location information of a shooting device for the video. 8. The image processing device according to claim 7,
10. An image processing device according to claim 9, wherein the importance indicated by the importance information is determined based on information corresponding to a position on a map compared with the position of the image capturing device. 6. The image processing device according to claim 1,
The image processing device according to claim 1, wherein the importance indicated by the importance information is determined based on a shooting direction relative to a moving direction of a shooting device for the video. An image processing method in which an image processing device generates a group of thumbnail images from video information in which importance information is assigned to each frame image that constitutes a video, the method comprising:
an extraction step of extracting a part of the frame images from the video information as thumbnail images;
a thumbnail image generating step of generating a group of thumbnail images by arranging the plurality of thumbnail images extracted in the extracting step in time series;
Including,
An image processing method characterized in that, in the extraction step, the number of thumbnail images to be generated is determined based on a predetermined setting, a setting based on a user operation, or a setting based on the recording time of the video, and an extraction interval for extracting the thumbnail images based on the importance information . An image processing program executed by an image processing device including a computer,
The computer
an acquisition means for acquiring video information to which importance information is assigned for each frame image constituting the video;
an extracting means for extracting a part of the frame images from the video information as thumbnail images; and
a thumbnail image generating unit that generates a group of thumbnail images in which the plurality of thumbnail images extracted by the extracting unit are arranged in time series;
The image processing program is characterized in that the extraction means determines the number of thumbnail images to be generated, which is set in advance, based on user operation, or based on the recording time of the video, and the extraction interval for extracting the thumbnail images based on the importance information . A computer-readable recording medium storing the image processing program described in claim 11.