JP7448629B2 - Image generation device, image generation method, image generation program, and recording medium - Google Patents

Description

The present invention relates to an image generation device, an image generation method, an image generation program, and a recording medium that generate one image based on a plurality of images.

2. Description of the Related Art A technique is known in which an image is presented as a clue to the user in order to specify a point where the user wants to start playing back a moving image.

For example, Patent Document 1 discloses a moving image search device that includes a temporal search means for searching moving image content in the temporal direction, extracts thumbnails from the moving image content using a time code string, and displays search results. is disclosed.

Furthermore, a technique is known in which photographed images are joined together to synthesize an image having a wider viewing angle than the viewing angle of the camera that took the photographs.

For example, Patent Document 2 discloses that by mounting two line cameras on a moving object and joining the images taken by each line camera while moving, a panoramic image that extends beyond the viewing angle of each of the two cameras is created. A method for creating it is disclosed.

Japanese Patent Application Publication No. 2002-281432 Patent No. 4551990

For example, if a panoramic image is presented as a clue to the user when playing a video of the scenery taken from the moving object by a camera mounted on the moving object, many parts of the image that are not needed will be displayed. One of the problems is that it is difficult to find the desired part of the video.

The present invention has been made in view of the above points, and an object of the present invention is to provide an image generation device that generates an image that allows efficient search when a user searches for a desired location in an image. There is only one.

The invention according to claim 1 includes: a video acquisition unit that acquires a landscape video in which a landscape is photographed from a moving body; a frame selection unit that selects a plurality of frame images from the landscape video; and a frame selection unit that selects a plurality of frame images from the landscape video. an index acquisition unit that acquires an importance index indicating the importance of each of the frame images; and an index acquisition unit that acquires an importance index indicating the importance of each of the selected frame images; a composite image generation unit that generates one composite image by reducing the plurality of images according to the importance indicated by the importance index associated with each of the plurality of images and combining them in the order in which they were taken. It is characterized by

The invention according to claim 8 is an image generation method in which an image generation device generates an image, the video acquisition unit comprising a video acquisition step of acquiring a landscape video in which a landscape is photographed from a moving object, and a frame selection step. a frame selection step in which the unit selects a plurality of frame images from the landscape video; and an index acquisition step in which the index acquisition unit acquires an importance index indicating the importance of each of the selected plurality of frame images. , a composite image generation unit converts each of the plurality of frame images selected in the frame selection step into a plurality of images such that a dimension in a first direction corresponding to the moving direction of the moving object is reduced. It is characterized by including a composite image generation step of reducing the size according to the importance indicated by the associated importance index and combining the images in the order in which they were photographed to generate one composite image.

The invention according to claim 9 is an image generation program executed by an image generation device including a computer, wherein a video acquisition unit is configured to cause the computer to generate a video that acquires a landscape video in which a landscape is photographed from a moving body. an acquisition step, a frame selection step in which a frame selection unit selects a plurality of frame images from the landscape video, and an index acquisition unit acquires an importance index indicating the importance of each of the selected plurality of frame images. In the index acquisition step, the composite image generation unit selects each of the plurality of frame images selected in the frame selection step so that the dimension in the first direction corresponding to the moving direction of the moving object is small. a composite image generation step of generating one composite image by combining each of the images reduced at a reduction rate according to the importance indicated by the importance index associated with each of the plurality of images in the order in which they were photographed; , is characterized by executing the following.

FIG. 1 is a perspective view of a car equipped with an image generation device according to an embodiment. FIG. 2 is a diagram schematically showing a display unit seen from the inside of a car equipped with an image generation device according to an example. FIG. 1 is a functional block diagram of an image generation device according to an embodiment. FIG. 2 is a diagram illustrating an example of data used in the image generation device according to the embodiment. FIG. 7 is a diagram illustrating an example of a reduction rate according to the importance of a frame image in the image generation device according to the embodiment. FIG. 3 is a diagram showing an example of a frame image selected by the image generation device according to the example. It is a figure showing an example of the operation screen of the image generation device concerning an example. 3 is a flowchart illustrating an example of a routine executed in the embodiment. 3 is a flowchart illustrating an example of a routine executed in the embodiment. It is a figure which shows an example of the degree of change based on a modification. It is a flowchart which shows an example of the routine performed in a modification.

Examples of the present invention will be described in detail below. In the following description and accompanying drawings, substantially the same or equivalent parts are designated by the same reference numerals.

FIG. 1 is a diagram schematically showing the configuration of an image generation device 10 according to an embodiment of the present invention. The image generation device 10 is a device that processes moving images captured by a movable imaging device. The image generation device 10 is mounted on a moving object, for example, and can capture a landscape video taken while the moving object is running by a camera mounted on the mobile object, play back the landscape video, and output video. It is composed of Furthermore, the image generation device 10 is configured to be able to generate a composite image from frame images that constitute the landscape video.

For example, the image generation device 10 outputs a composite image to a display device mounted on the moving body. Further, for example, the image generation device 10 outputs a video obtained by reproducing a landscape video to a display device mounted on the moving body.

For example, the image generation device 10 displays a composite image on a display device, receives a selection operation for selecting a position in the horizontal direction in the composite image, and cues from the frame image corresponding to the selected position. to output the video of the landscape video played.

In this embodiment, an example will be described in which the image generation device 10 is mounted on an automobile M as a moving body. As shown in FIG. 1, the image generation device 10 is mounted, for example, in the dashboard of an automobile M.

The camera 13 is, for example, a video camera attached to a roof portion or a door panel portion of the automobile M so as to be able to photograph the side of the automobile M. The camera 13 is communicably connected to the image generation device 10 and is configured to be able to transmit captured still images or moving images to the image generation device 10.

The moving image captured by the camera 13 is captured into the image generation device 10. For example, the camera 13 is set to start photographing when the car M starts its engine and starts traveling. Note that the automobile M may further be equipped with a video camera capable of photographing the front and rear of the automobile M, for example.

The first display 15 is a display device provided in a window portion of the automobile M. In this embodiment, a case will be described in which the first display 15 is a transparent display attached to the window glass of the automobile M.

For example, the first display 15 is attached to the rear door DR. Note that the first display 15 may be attached instead of the window glass of the automobile M, or may be attached to the inside of the window glass of the automobile M. Further, the first display 15 may be attached to, for example, the front door DF, or may be attached to the inside of the windshield, for example.

The first display 15 is, for example, a display made of transparent liquid crystal, organic EL, or inorganic EL. For example, the first display 15 is configured to have a visible light transmittance of a predetermined value or higher, at least when not displaying a moving image, so that external scenery can be seen with the naked eye.

The first display 15 is connected to the image generation device 10 by wire or wirelessly. The first display 15 has a display surface facing toward the interior (inside the cabin) of the vehicle M, and displays the moving image output from the image generation device 10 toward the passenger compartment of the vehicle M. For example, a user who is inside the car M can view a video of the scenery taken from the car M in a manner similar to looking out a window.

FIG. 2 is a diagram schematically showing a portion of the first display 15 (ie, the window of the vehicle M) and the rear door DR as viewed from the interior side of the vehicle M. FIG. 2 shows the rear door DR viewed from the rear seat.

The first display 15 has a moving image display surface 15A facing the inside of the automobile M. In FIG. 2, a moving image taken in the past from a car M is played back by the image generation device 10 and displayed on the moving image display screen 15A.

The touch panel 17 is a touch panel display attached to the inside of the door panel of the rear door DR. For example, as shown in FIG. 2, the touch panel 17 extends on the surface of the inner panel IP of the rear door DR along the upper end of the inner panel IP. The touch panel 17 has a rectangular shape whose longitudinal direction is along the upper end of the inner panel.

The touch panel 17 is a touch panel composed of a second display 17A and a touch pad 17B. The second display 17A is, for example, a display device such as a liquid crystal display device or an organic EL display device. The touch pad 17B is, for example, a pressure-sensitive or electrostatic transparent touch pad that accepts operations with a human finger or the like attached to the display surface of the second display 17A.

The touch panel 17 is communicably connected to the image generation device 10 and can display images output from the image generation device 10. Furthermore, the touch panel 17 is capable of accepting operations for reproducing, fast forwarding, or fast reversing the moving image displayed on the first display 15, and transmitting an instruction to the image generation device 10 based on the acceptance of the operation.

FIG. 3 is a functional block diagram of the image generation device 10. The image generation device 10 is, for example, a device in which each part cooperates via a system bus 18.

The input unit 19 is an interface that acquires data from equipment external to the image generation device 10. The input unit 19 is connected to a camera 13, which is a device included in the automobile M. For example, the image generation device 10 can capture a landscape video from the camera 13 via the input unit 19.

The input unit 19 also includes a speed sensor 21 capable of acquiring the speed of the vehicle M and a global positioning system (GNSS), which is one of the Global Navigation Satellite Systems (GNSS) that acquires position information of the vehicle M. It is connected to a GPS receiver 22 that can receive GPS (Global Positioning System) information. That is, the image generation device 10 is configured to be able to acquire the speed of the automobile M from the speed sensor 21 and to acquire GPS information from the GPS receiver 22.

The output unit 23 is an interface that outputs data to a device external to the image generation device 10. The output unit 23 is connected to the first display 15 and the second display 17A of the touch panel 17.

For example, the output unit 23 outputs a series of moving images reproduced by the image generation device 10 to the first display 15. For example, the output unit 23 outputs an operation image for playback operation to the second display 17A. Further, for example, the output unit 23 outputs the composite image generated by the image generation device 10 to the second display 17A.

The operation input unit 25 is an interface that obtains operation input signals via the touch pad 17B of the touch panel 17. The operation input unit 25 is connected to the touch pad 17B, and acquires an operation input signal for the operation image displayed on the second display 17A.

The storage unit 27 is configured with, for example, a hard disk device, an SSD (solid state drive), a flash memory, etc., and stores various programs related to image processing, video playback, etc. in the image generation device 10. The storage unit 27 also stores a map database in which map information is stored.

Note that various programs or map information may be obtained, for example, from another server device or the like via a network (not shown), or may be recorded on a recording medium and read via various drive devices. You can do it like this.

Various programs stored in the storage unit 27 can be transmitted via a network, or can be recorded on a computer-readable recording medium and transferred.

The control unit 29 includes a CPU (Central Processing Unit) 29A, a ROM (Read Only Memory) 29B, a RAM (Random Access Memory) 29C, and functions as a computer. The CPU 29A reads and executes various programs stored in the ROM 29B and the storage unit 27, thereby realizing various functions.

For example, the control unit 29 acquires a video captured by the camera 13 via the input unit 19, and functions as a video acquisition unit.

For example, the control unit 29 acquires the speed of the automobile M via the input unit 19, and acquires the cumulative travel distance from the speed and travel time.

For example, the control unit 29 acquires the current position information of the car M from the data of the GPS receiver 22 via the input unit 19, and calculates the cumulative travel distance of the car M from the position of the car M and the time.

The control unit 29 acquires the cumulative travel distance (i.e., cumulative travel distance) from when the vehicle M starts traveling until now based on the information acquired from the speed sensor 21 or the GPS receiver 22 via the input unit 19. In this case, it functions as a moving distance acquisition unit.

For example, the control unit 29 associates each of the frame images constituting the series of moving images obtained from the camera 13 via the input unit 19 with the cumulative mileage of the automobile M at the time when each of the frame images was photographed. It functions as an information-containing video generation unit that generates an information-containing video. Further, the control unit 29 functions as a recording control unit that stores the generated information-bearing video in the storage unit 27.

For example, the control unit 29 functions as a frame selection unit that selects a plurality of frame images from all the frame images that make up the information-bearing moving image.

For example, when selecting a plurality of frame images (hereinafter also referred to as selected frame images) from the information-bearing video, the control unit 29 extracts them according to the cumulative travel distance at the imaging position of each frame image.

For example, when selecting a plurality of frame images from the information-bearing video, the control unit 29 extracts the plurality of frame images such that the cumulative movement distances at the imaging positions of the respective frame images are equally spaced. In other words, the control unit 29 selects a plurality of frame images such that the points where each of the frame images is captured are equidistantly spaced from each other.

For example, when selecting a plurality of frame images at equal distance intervals, the control unit 29 always sets distance intervals between points where each of the plurality of frame images is taken to be a constant distance interval.

For example, when selecting a plurality of frame images at equal distance intervals, the control unit 29 arranges the distance intervals of the points at which each of the plurality of frame images is taken in the order in which the plurality of frame images are taken. The selected frame image is selected at a distance interval that causes an overlapping portion of the photographing range (photographing area) with the adjacent frame image.

For example, the control unit 29 functions as an index acquisition unit that acquires an importance index indicating the importance of each of the plurality of selected frame images.

For example, the degree of importance is determined based on the degree of scenery worth seeing at each location on the map. For example, the level of worthiness is associated in advance with map information in a map database for each location or area on the map, based on criteria such as the number of buildings and commercial facilities, and the presence or absence of tourist attractions. For example, the level of worth seeing is a predetermined numerical value, and is associated with a position on a map in a map database, for example, as a "sightseeing rank".

For example, a frame image photographed when the car M is located at a position that is highly worth seeing (for example, ranking of attractions) on the map database is determined to have a high degree of importance. On the other hand, a frame image photographed when the car M is at a position with a low ranking of attractions on the map database is determined to be of low importance.

In other words, the control unit 29 as an index acquisition unit determines the position of the car M at the time when each of the frame images constituting the landscape video or each of the selected frame images was photographed, and the view of the landscape at each position on the map. An importance index indicating the importance of each frame image is specified based on map data in which information regarding the frame image is stored in advance. For example, the control unit 29 obtains the degree of importance by referring to position information of the vehicle M based on GPS and a map database recorded in advance in the storage unit 27.

Furthermore, the degree of importance may be determined based on the degree of change by comparing each of a plurality of frame images or each selected frame image constituting a landscape moving image. For example, the control unit 29 serving as an index acquisition unit may determine that a frame image that has a large change in image compared to adjacent frame images in the order in which it was photographed has a high degree of change, and has a high degree of importance. good.

Furthermore, the degree of importance may be determined based on the degree of complexity of the image for each of a plurality of frame images or each selected frame image constituting the landscape moving image. For example, the control unit 29 as an index acquisition unit analyzes each of a plurality of frame images or selected frame images to calculate the degree of complexity of each image, and determines that the frame image having a higher degree of complexity has a higher degree of importance. You may.

Furthermore, for example, in mountainous areas with high altitudes, the scenery may be spectacular. Therefore, the control unit 29 as the index acquisition unit may determine that the importance level is high at the time when positioning in a predetermined position range is detected from positioning information such as GPS.

Further, for example, if the GPS receiver 22 cannot acquire GPS information for a predetermined period of time or more, there is a high possibility that the car M is traveling inside a structure such as a tunnel, and in such a case, the The content of the video will continue to be monotonous. Therefore, the control unit 29 serving as the index acquisition unit may determine that the importance of a landscape video shot in a situation where GPS information cannot be acquired for a predetermined period of time or more is low.

Further, in a scenery video in which the car M is traveling, the scenery video on the side of the car M changes greatly in a short period of time compared to the scenery video taken in front or behind the car M in the direction of travel. Therefore, the control unit 29 serving as the index acquisition unit may determine that a landscape video on the side in the direction of travel is more important than a landscape video on the front or rear side in the travel direction. That is, the degree of importance may be determined, for example, based on the photographing direction of the camera 13 based on the traveling direction of the automobile M.

The above criteria for determining the degree of importance can be set arbitrarily. In addition, the conditions for determining the degree of importance are the photographing position of the car M, the degree of visibility of the scenery associated with the position on the map information corresponding to the photographing position, the degree of change in the image between adjacent frame images, and the complexity of the frame image. The degree of importance may be determined based on a plurality of determination conditions, without being limited to any one of the following: the acquisition status of positioning information from the satellite system, or the photographing direction of the photographing unit with respect to the traveling direction of the automobile M.

In other words, the index acquisition unit determines the degree of importance by determining the photographing position of the car M, the visual appeal of the scenery associated with the position on the map information corresponding to the photographing position, the degree of image change between adjacent frame images, and the frame An importance index is acquired based on at least one of the complexity of the image, the acquisition status of positioning information from the satellite system, and the photographing direction of the photographing unit with respect to the traveling direction of the automobile M.

Note that the information-bearing video may be generated by associating the degree of importance with each of the frame images that make up the landscape video. In other words, each of the frame images constituting the information-bearing video may be associated with at least one of cumulative mileage and importance.

Further, for example, the control unit 29 functions as a composite image generation unit that generates a composite image from the information-bearing video. The control unit 29 as a composite image generation unit connects and composites each of the plurality of frame images (selected frame images) selected by the frame selection unit in the order in which they were photographed, that is, in the order corresponding to the moving direction of the automobile M. , generate a composite image. For example, a composite image is generated by seamlessly stitching together scenery in a plurality of selected frame images, resulting in a panoramic image.

For example, when generating a composite image, each of the selected frame images is reduced so that the dimension in the direction (first direction) corresponding to the moving direction of the automobile M is reduced according to the importance indicated by the importance index. Ru. For example, each of the selected frame images is reduced at a larger reduction ratio as the degree of importance is lower.

For example, the control unit 29 functions as a display control unit that controls display of the generated composite image on the second display 17A.

The control unit 29 serving as a display control unit may perform control to display the entire composite image, or may perform control to display a portion of the composite image, for example. For example, when displaying a part of the composite image, control may be performed such that the part of the composite image is scrolled.

Further, the control unit 29 functions as an operation reception unit that receives instruction operations regarding reproduction of a landscape video via the operation input unit 25. For example, the control unit 29 as an operation reception unit receives an instruction operation that instructs the playback control unit to play, fast-forward, or fast-reverse through an input operation on the touch pad 17B using an image for playback operation. .

The control unit 29 controls the reproduction of a moving image to be displayed on the first display 15 via the output unit 23 in response to the instruction operation. For example, the control unit 29 functions as a playback control unit that controls the playback of a landscape video or fast-forward or fast-reverse.

For example, when a selection operation for selecting one position in the direction (horizontal direction) corresponding to the moving direction of the car M in the composite image is received, the control unit 29 controls the control unit 29 to frame a frame corresponding to the position selected by the selection operation. A video obtained by cuing up the image and playing back a landscape video or an information-bearing video is supplied to the first display 15 via the output unit 23.

Note that in this embodiment, playback refers to playing back a photographed landscape video in the forward direction at a basic speed determined based on the time code attached to the video. Furthermore, fast-forwarding refers to playing back a photographed landscape video in the forward direction at a speed faster than the basic speed. Further, fast rewinding refers to playing back a captured landscape video in the reverse direction at a speed faster than the basic speed.

FIG. 4 shows that when an information-bearing video is generated, each of the frame images constituting the landscape video includes the distance traveled by the car M at the time when each frame image was photographed and the importance of the photographed point. It is a figure which shows table TB1 which is an example of the data with which the index was matched.

As shown in FIG. 4, in table TB1, the frame number, which is the number assigned to each frame image constituting the landscape video, the elapsed time from the start of shooting at the time when each frame image was shot, and the corresponding The cumulative travel distance from the shooting start point at the point in time is associated. In table TB1, a case is illustrated in which a landscape video is photographed by the camera 13 at a photographing frame rate of 30 fps.

For example, in table TB1, the frame image with frame number 0 taken at the start of shooting is associated with a travel distance of 0 (m) from the start of shooting. In other words, the time point at which photographing starts coincides with the time point at which cumulative travel distance starts. Note that time may be associated instead of elapsed time.

Further, as shown in FIG. 4, in table TB1, each frame number is associated with an importance index indicating the importance of the corresponding frame image at the point where the corresponding frame image was photographed.

For example, the importance index is a ranking of attractions that is associated in advance with map information in a map database, as described above. For example, the highlight rank is specified by comparing the location information of the car M based on GPS with the map information.

In the example shown in FIG. 4, a case is shown in which an importance index expressing the highlight ranking in four stages of 0, 1, 2, and 3 is given.

In this embodiment, when a composite image is generated, frame images are selected at predetermined intervals from among the frame images indicated by the frame numbers in FIG. The predetermined interval may be, for example, an interval based on the cumulative travel distance of the automobile M. For example, frame images may be selected at equidistant intervals such that the points at which corresponding frame images are photographed are equally spaced.

For example, the predetermined distance interval is a distance interval at which, when the selected frame images are arranged in the order in which they were photographed, there is an overlap in the photographing range or region between adjacent frame images.

Note that, for example, the predetermined interval when frame images are selected may be an interval based on time. For example, frame images may be selected at equal time intervals. In that case, the time interval at which frame images are selected may be adjusted depending on the speed of the automobile M.

For example, frame images taken during a time period when the car M was traveling at high speed may be selected at shorter time intervals than during a time period when the vehicle M was traveling at low speed. For example, the time interval may be adjusted so that a frame image in which a desired photographing range is photographed is selected.

FIG. 5 is a diagram showing a table TB2 that is an example of data in which the degree of importance associated with each frame image and the reduction rate according to the degree of importance are described when a composite image is generated. When the composite image is generated, each of the selected frame images is reduced in size in the direction (first direction) corresponding to the moving direction of the automobile M, depending on the degree of importance.

As shown in FIG. 5, four levels of importance, 0, 1, 2, and 3, are listed in table TB2. Table TB2 shows that frame images with lower importance are reduced at a higher reduction rate. Furthermore, table TB2 indicates that frame images associated with "3", which has the highest degree of importance, are not reduced.

FIG. 6 is a diagram showing a panoramic image FM in which selected frame images, which are a plurality of frame images selected from among frame images constituting an information-bearing video, are connected, and a composite image SY generated from the panoramic images FM. be. In FIG. 6, a panoramic image FM is shown at the top. The panoramic image FM is obtained by arranging selected frame images F1 to F8 in a direction corresponding to the moving direction of the automobile M (the X direction in the figure) and joining them together.

In FIG. 6, the importance index attached to each of the selected frame images F1 to F8 is shown in parentheses. For example, "F1(3)" indicates that the importance level assigned to the selected frame image F1 is "3".

FIG. 6 schematically shows how, for example, the selected frame image that shows more buildings etc. is assigned a higher importance index. Note that in the selected frame image F6 and the selected frame image F7, the car M was traveling in a tunnel, so the walls of the tunnel appear in the images.

The selected frame images F1 to F8 are selected at equal distance intervals, for example, and have portions in which the photographing ranges overlap with adjacent frame images in the photographing order. In FIG. 6, the overlapping portion OL, which is the overlapping portion, is hatched.

For example, in FIG. 6, the selected frame image F1 has an overlapping portion OL between it and the adjacent selected frame image F2. Similarly, the selected frame images F2 to F7 also have an overlapping portion OL between adjacent selected frame images.

For example, for two frame images that have an overlapping portion, by deleting the overlapping portion of one image and joining the two frame images together, a continuous landscape image can be generated. For example, when deleting the overlapping portion, a frame image with a small importance index is deleted, and the overlapping portion of a frame image with a high importance index is left.

In FIG. 6, after the overlapping portion OL is deleted from the panoramic image FM, portions originating from the selected frame images F1 to F8 are shown as portions FA1 to FA8 surrounded by broken lines. For example, the selected frame image F1 has a higher importance index than the selected frame image F2, so the overlapping portion OL is deleted from the selected frame image F2, and the remaining portion is the portion FA2.

When the composite image is generated, the portions FA1 to FA8 are reduced in the direction (X direction) corresponding to the moving direction of the automobile M at a reduction rate depending on the degree of importance, for example. For example, among the image parts FA1 to FA8, the lower the importance indicated by the importance index, the higher the reduction ratio in the X direction.

In the lower part of FIG. 6, a composite image SY is shown. The composite image SY is an image in which the parts FA1 to FA8 of the image from which the overlapping part OL has been deleted from the panoramic image FM are reduced and joined together according to the magnitude of the importance index. In FIG. 6, the reduced image portions are shown as FB1 to FB8. For example, image FB2 in composite image SY is a reduced image portion of image portion FA2 after deletion processing.

For example, in the example shown in FIG. 6, the portion FA4 with the importance index "2" is reduced in size in the X direction by half, and the portion with the importance index "1" is FA2 has a larger reduction ratio than part FA4, and its dimension in the X direction has been reduced to one-third.

Further, FA3, FA6, and FA7, which are assigned the lowest importance index "0", are reduced to one-fifth. FA1, FA5, and FA8 to which the highest importance index "3" is attached are not reduced, and the reduction rate is 0%.

In this way, the less important parts of the image FA1 to FA8 are reduced at a higher reduction rate in the X direction, so that in the composite image SY, the parts of the image where many buildings etc. , and the proportion of areas where buildings etc. are not reflected is decreasing. Therefore, when searching for a desired video location using the composite image SY as a clue, the proportion of portions that the user does not need is small, making it easier to find the desired video location.

FIG. 7 is a diagram illustrating an example of a reproduction operation image PC, which is an operation image for reproducing a moving image, displayed on the touch panel 17. In FIG. 7, a finger FN of a user performing an operation using the reproduction operation image PC is shown together with a reproduction operation image PC.

As shown in FIG. 7, the playback operation image PC includes a button area AR1, which is an area in which operation buttons are displayed, and a search area AR2, which is an area in which an image for searching a video part is displayed. include.

In the button area AR1, a play button PL, a stop button ST, a fast forward button FF, and a fast backward button RW are displayed. For example, the user can input operations for playing, stopping, fast-forwarding, or fast-reversing a video by touching the area where these buttons are displayed with the finger FN. For example, by touching the area where the play button PL is displayed with a finger FN, the user causes the image generation device 10 to play a landscape video and display it on the video display screen 15A of the first display 15 (see FIG. 2). can be done.

A composite image SY as a panoramic image is displayed in the search area AR2. The composite image SY extends along a direction parallel to the long side of the playback operation image PC. Further, in the search area AR2, a marker MK indicating which part of the video is reproduced by the image generation device 10 and currently displayed on the first display 15 indicates the corresponding part in the composite image SY. It is displayed as follows.

For example, the distance traveled by the car M corresponding to the point where the portion of the video indicated by the marker MK was photographed may be displayed. In this case, the image displayed on the first display 15 (FIG. 2) is the information-bearing video generated from the landscape video at the time when the cumulative travel distance of the car M is the travel distance indicated by the marker MK. , the video was taken at the location where car M was traveling. Note that the search area AR2 may display the time at which the portion of the video indicated by the marker MK was shot or the elapsed time from the start of shooting.

For example, the composite image SY is scroll-displayed. For example, the user can scroll the composite image SY by moving the finger FN in the direction along the long side of the composite image SY, that is, in the direction of the arrow in FIG. 7, and performing a drag operation or the like. Therefore, the composite image SY serves as a clue for the user to search for which part of the landscape video to start playing or to cue the scene.

For example, when the finger FN selects one position in the X direction on the composite image SY, a portion of the moving image cued from the frame image corresponding to the one position is displayed on the first display 15.

For example, an input operation for selecting the first position is accepted via the touch pad 17B when the finger FN contacts an area of the second display 17A of the touch panel 17 where the composite image SY is displayed. Note that the selection operation may be performed by aligning the image to be reproduced with the position of the marker MK.

FIG. 8 is a flowchart showing an information-bearing moving image generation routine RT1, which is an example of a routine executed by the control unit 29. For example, when the engine of the automobile M is turned on, the control unit 29 starts the information-bearing moving image generation routine RT1.

When the control unit 29 starts the information-holding video generation routine RT1, it determines whether a landscape video is stored in the storage unit 27 (step S11). If the control unit 29 determines that a landscape video is not stored (step S11: NO), it repeats step S11 and determines again whether a landscape video is stored.

When determining in step S11 that a landscape video is stored (step S11: YES), the control unit 29 reads out the landscape video and adds the cumulative information at the time of each shooting to each of the frame images constituting the landscape video. A moving distance is given (step S12).

For example, in step S12, the control unit 29 determines whether each of the frame images has been captured from the time when the shooting of the landscape video is started, based on the information acquired from the speed sensor 21 or the GPS receiver 22 via the input unit 19. Calculate the cumulative distance traveled up to the point in time. In step S12, for example, as shown in table TB1 shown in FIG. 4, each frame number of the frame image is associated with the elapsed time from the start of photography and the cumulative travel distance of the automobile M.

After executing step S12, the control unit 29 assigns an importance index indicating the importance of each frame image to each of the frame images constituting the landscape video (step S13). In step S13, which is an index acquisition step, for example, the control unit 29 refers to the map data, and specifies and assigns the degree of importance based on the highlight ranking at the point where each frame image was photographed. In step S13, for example, as shown in table TB1, each frame number is associated with the degree of importance of the corresponding frame image. In step S13, the control unit 29 functions as an index acquisition unit.

After executing step S13, the control unit 29 stores in the storage unit 27 an information-bearing video in which each frame image is associated with a cumulative mileage and an importance index (step S14). The control unit 29 ends the information-holding moving image generation routine RT1, returns to the beginning, and repeatedly executes the information-holding moving image generation routine RT1.

FIG. 9 is a flowchart showing a composite image generation routine RT2, which is an example of a routine executed by the control unit 29 when a composite image is generated in this embodiment. For example, when the control unit 29 receives an operation instructing to play a landscape video, it starts the composite image generation routine RT2.

When the control unit 29 starts the composite image generation routine RT2, it acquires the information-bearing video generated from the landscape video (step S21). In step S21 as a moving image acquisition step, for example, the control unit 29 generates an information-bearing moving image stored in the storage unit 27 in which a cumulative mileage and an importance index are associated with each of the frame images constituting the landscape moving image. Read out. In step S21, the control unit 29 functions as a moving image acquisition unit.

After executing step S21, the control unit 29 selects a plurality of frame images from among the frame images forming the information-bearing moving image (step S22). In step S22 as a frame selection step, for example, the control unit 29 selects a plurality of frame images at predetermined intervals. For example, in step S22, the points at which each of the frame images is photographed are selected to be equidistant apart. In step S22, the control unit 29 functions as a frame selection unit.

For example, in step S22, the control unit 29 may connect each of the selected frame images in the order in which they were photographed to generate the panoramic image FM (see FIG. 6).

After executing step S22, the control unit 29 reduces each of the selected frame images in a direction parallel to the moving direction of the automobile M at a reduction rate according to the importance index (step S23). In step S23, the control unit 29 reduces each of the selected frame images at a reduction rate that increases as the importance index decreases, as shown in FIG. 6, for example.

For example, in step S23, the control unit 29 deletes the portion where the photographing range overlaps between the frame images arranged in the order in which they were photographed, and reduces the remaining portion (see FIG. 6).

After executing step S23, the control unit 29 connects each of the reduced frame images to generate a composite image (step S24). In step S24, the control unit 29 arranges and connects each of the selected frame images reduced according to the importance index in the order in which they were photographed.

For example, as shown in FIG. 6, the overlapping portions of adjacent frame images are deleted, so that the scenery in the frame images is seamlessly connected. In step S23 and step S24 as the composite image generation step, the control unit 29 functions as a composite image generation unit.

After completing step S24, the control unit 29 ends the composite image generation routine RT2 and repeatedly executes the composite image generation routine RT2.

Note that when a user attempts to search for a desired scene from a captured landscape video, for example, the user may have an idea of where in the landscape video the desired scene is recorded; The preferable distance scale of the composite image SY to be displayed in the search area AR2 of the playback operation image PC is different depending on the case where there is no image PC. In the former case, the composite image SY of a limited range around the position corresponding to the cumulative travel distance that has already been estimated will be effective; in the latter case, the composite image SY will first be used to estimate where the desired scene is recorded. Since it is necessary to attach , the composite image SY is effective over a relatively wide range. In order to cope with this, it is preferable that the image generation device 10 is configured such that the movement distance range displayed in the composite image SY as part of the playback operation image PC can be changed by the user.

Specifically, the image generation device 10 includes and displays an operation icon for the user to specify the distance scale of the composite image SY in the playback operation image PC of the touch panel 17, and the control unit 29 displays the above-mentioned information. In step S21 of the retained video generation routine RT1, the range of cumulative movement distance corresponding to the distance scale specified by the user's operation is acquired from the information retained video. As a result, by making it possible to arbitrarily change the distance scale of the composite image SY, it is possible to search for a desired scene within a limited distance range, or when searching for a desired scene within a relatively long distance range. An effective composite image SY can be generated.

As described above, according to this embodiment, when playing back a video of a scene taken from a moving object by a camera mounted on the moving object, a composite image is presented as an image that serves as a clue to the user. can do. A composite image is an image created by selecting multiple frame images at predetermined intervals from among the frame images that make up a landscape video, reducing them horizontally according to their associated importance, and then joining them together. It is. Therefore, in the direction corresponding to the direction of movement of the moving body (lateral direction as viewed from the user), the range occupied by images that are less important and unnecessary is smaller, making it easier to find desired images. Therefore, it is possible to provide an image generation device that generates an image that allows a user to search for a desired location in an image with high efficiency and that allows the user to easily find the desired location. .

[Modification 1]
An image generation device 10 according to a first modification of the present embodiment will be described with reference to FIGS. 10 and 11. The image generation device 10 of this modification 1 is configured similarly to the image generation device 10 in the above embodiment.

In the present modification example 1, a plurality of frame images selected from frame images constituting a landscape video is provided with a degree of change index indicating the degree of change of the image from adjacent frame images in the order in which they were photographed, as an importance index. Granted. Specifically, among the plurality of selected frame images, if there is a large change in the image between one frame image and other frame images adjacent in the shooting order, that is, if there is a large change in the scenery, , since the degree of sight is high as the scenery changes, an importance index indicating that the importance is high is assigned to such a frame image. Also, among the selected multiple frame images, if there is a small change in the image between one frame image and other frame images adjacent in the shooting order, that is, if the change in the scenery is small, the same Since such a frame image is not worth seeing because it is a continuous landscape, an importance index indicating that the importance is low is assigned to such a frame image. In the first modification, when a composite image is generated, each of the selected frame images is reduced at a reduction rate according to the importance index.

For example, the degree of change can be determined by classifying the landscape recorded in each selected frame image into predetermined categories and comparing the landscape classification results with adjacent selected frame images. Ru. For example, the scenery recorded in each of the selected frame images may be classified into one of four classification items: "forest," "cityscape," "open road," and "other" based on the characteristics of each image.

FIG. 10 is a diagram showing a table TB3 in which an example of the degree of change index in Modification 1 is described. FIG. 10 shows an example in which the degree of change index is determined in two stages: "0" and "1". For example, as shown in FIG. 10, the change degree index may be specified based on whether the classification item of the scenery recorded in the frame image has changed.

In the example shown in FIG. 10, when the selected frame images are arranged in the order in which they were photographed, if the classification of the first frame image and the next frame image is the same, the next frame image will be changed. The degree is determined to be "0". For example, if a frame image following a frame image classified as "forest" is also classified as "forest", the degree of change is determined to be "0".

Further, if the classification of the first frame image and the next frame image are different items, the degree of change is determined to be "1" for the next frame image. For example, when a frame image following a frame image classified as "forest" is classified as "cityscape", the degree of change is determined to be "1".

That is, if there is no change that satisfies a predetermined standard between one frame image and the next frame image, the change degree index of the next frame image is specified as "0", and the specified change degree index is "0". 0'' is given to the next frame image as an importance index indicating that the importance is low. Furthermore, if there is a change that satisfies a predetermined standard between the first frame image and the next frame image, the change degree index of the next frame image is specified as "1", and the specified change degree index is "1". 1" is given to the next frame image as an importance index indicating that the importance is high. Note that the degree of change may be determined by comparing one frame image and the previous frame image when the selected frame images are arranged in the order in which they were photographed.

Further, the change index is not limited to being determined in two stages as illustrated in FIG. 10, but may be determined in three or more stages.

In this case, as an example of a method for classifying the scenery recorded in each of the selected frame images, each of the selected frame images is divided into a plurality of image pieces, and the results of analyzing the characteristics of each image piece are used. Good too. For example, classification may be performed by classifying each feature of the image fragment into multiple categories based on the analysis results of the image fragment, and quantifying each feature of the selected frame image based on the contribution rate of the image fragment in each category. . Such a method makes it possible to perform more detailed classification and to assign change indexes in more stages.

Specifically, for example, the image piece may be analyzed by color analysis, fractal dimension analysis, or artifact amount analysis. For example, color analysis analyzes the color properties of each pixel in each image segment of a frame image. For example, fractal dimension analysis yields a fractal dimension value for each pixel piece, which serves as an indicator of the complexity of the image, for example. For example, in artifact amount analysis, the amount of artifacts included in one pixel piece is quantified by counting the number of edge components of a predetermined length or more that continuously extend in the vertical direction in one image piece. It's okay. For example, based on these analysis results, each of the image pieces may be classified into classification items such as "plants", "sky", and "artificial object". Furthermore, the frame images may be classified by calculating the features of each frame image from the classification results of the image pieces.

FIG. 11 is a flowchart showing a composite image generation routine RT3 executed by the control unit 29 in the first modification. Note that a part of the description of the steps that are substantially equivalent to the composite image generation routine RT2 described in FIG. 9 will be omitted.

When the control unit 29 starts the composite image generation routine RT3, it acquires the information-bearing video generated from the landscape video (step S31), similar to step S21 of the composite image generation routine RT2 (FIG. 9). In step S31, the information-bearing video does not necessarily have to be associated with an importance index.

After executing step S31, the control unit 29 selects a plurality of frame images from among the frame images constituting the information-bearing video (step S32), similarly to step S22 of the composite image generation routine RT2.

After executing step S32, the control unit 29 acquires a change degree index and assigns it to one frame image as an importance degree index (step S33). For example, in step S33, the control unit 29 classifies the scenery recorded in each of the frame images selected in step S32 into predetermined classification items, and compares the scenery recorded in each of the frame images selected in step S32 with adjacent selected frame images in the order in which they were photographed. Determine the degree of For example, as described above, if the classification items of the first frame image and the next frame image are the same, the degree of change is determined to be "0", and if the classification items are different, the degree of change is determined to be "0". 1”. In step S33, the control unit 29 functions as an index acquisition unit.

After executing step S33, the control unit 29 reduces each of the selected frame images in a direction parallel to the moving direction of the automobile M at a reduction rate according to the importance index assigned in step S33 (step S34). ). In step S34, the control unit 29 reduces each of the selected frame images at a reduction rate that increases as the importance index decreases. Note that no matter how many levels the importance index is judged, for the importance index that indicates the highest degree of importance, the reduction rate is set to 0% (that is, no reduction is performed). Good too.

After executing step S34, the control unit 29 connects each of the reduced frame images to generate a composite image (step S35). In step S35, the control unit 29 connects each of the selected frame images reduced according to the importance index in the order in which they were photographed.

In this way, according to the present modification example 1, by employing the degree of change index as the degree of importance index, it is possible to reduce the continuous display of images with little change in the composite image. Therefore, there is little change from adjacent frame images, and in the direction corresponding to the moving direction of the moving body, the range occupied by unnecessary images is reduced, making it easier to find a desired image. Therefore, it is possible to provide an image generation device that generates an image that allows a user to search for a desired location in an image with high efficiency and that allows the user to easily find the desired location. .

[Modification 2]
An image generation device 10 according to a second modification of the present embodiment will be described. The image generation device 10 of this modification 2 is configured similarly to the image generation device 10 in the above embodiment. A flowchart showing a composite image generation routine executed by the control unit 29 in the second modification is shown in FIG. 11 similarly to the first modification.

This modification 2 employs an image complexity index as an importance index instead of the variation index that indicates the degree of change in an image as an importance index in modification 1. This modification differs from Modification 1 only in that respect. That is, the composite image generation routine of the second modification differs from the first modification only in the processing content of step S33 of the composite image generation routine RT3 shown in FIG. Therefore, the description of the steps that are substantially equivalent to the composite image generation routine RT3 described with reference to FIG. 11 will be omitted.

In step S33, the control unit 29 of the second modification analyzes each of the plurality of frame images selected from among the frame images constituting the information-bearing video, determines the complexity of each image, and calculates the complexity index. is specified and given to each frame image as an importance index.

Specifically, the control unit 29 calculates the complexity of each frame image by calculating the complexity of the frame image by a known analysis method using, for example, fractal dimension analysis and comparing this with a predetermined threshold. Identify indicators. Frame images with a high degree of complexity contain a large amount of visual information and are therefore highly impressive, so such frame images are considered to have a high degree of importance. The control unit 29 assigns the identified complexity index to each frame image as an importance index such that the higher the complexity indicated by the identified complexity index, the higher the importance indicated by the importance index. . Note that the complexity index may be specified in two or more arbitrary stages.

In other words, in this modification example 2, the control unit 29 serving as the index acquisition unit determines the characteristics shown by each of the plurality of frame images by analyzing each of the plurality of frame images, and determines the characteristics shown by the characteristics. Based on this, an importance index indicating the importance of each of the plurality of frame images can be obtained. In the present modification example 2, the control unit 29 as the index acquisition unit can employ the degree of complexity of the image as one of the characteristics shown by each of the plurality of frame images.

In this way, according to the second modification, by employing the complexity index as the importance index, it is possible to reduce the continuous display of images with a small amount of visual information in the composite image. can. Therefore, in the direction corresponding to the moving direction of the moving object, the range occupied by unnecessary images becomes smaller, making it easier to find a desired image. Therefore, it is possible to provide an image generation device that generates an image that allows a user to search for a desired location in an image with high efficiency and that allows the user to easily find the desired location. .

[Modification 3]
An image generation device 10 according to a third modification of the present embodiment will be described. The image generation device 10 of the present modification 3 is configured similarly to the image generation device 10 in the above embodiment. A flowchart showing a composite image generation routine executed by the control unit 29 in the third modification is shown in FIG. 11 similarly to the first and second modifications.

In this modification example 3, instead of using the image complexity index as the importance index in modification example 2, the importance index is set for each classification item of the scenery recorded in the image. This modification differs from modification 2 only in that it employs the importance level. That is, the composite image generation routine of Modification 3 differs from Modification 1 and Modification 2 only in the processing content of step S33 of composite image generation routine RT3 shown in FIG. Therefore, the description of the steps that are substantially equivalent to the composite image generation routine RT3 described with reference to FIG. 11 will be omitted.

In step S33, the control unit 29 of the present modification 3 determines the scenery recorded in each frame image, for example, "forest", " The frame images are classified into one of a plurality of classification items such as "cityscape", "open road", and "other", and a preset importance level is obtained for each classification item and given to each frame image as an importance index.

Specifically, as in Example 1, the control unit 29 divides each of the selected frame images into a plurality of image pieces, and uses the results of analyzing the characteristics of each image piece to record in each frame image. You may also classify the landscape. Further, the importance level for each category of landscapes may be predetermined for each category item, or the user himself/herself can assign multiple levels of importance level according to the user's degree of attention to each category item of landscapes. It may also be configurable.

As described above, in the present modification example 3, the control unit 29 as the index acquisition unit may employ a classification item that classifies the scenery recorded in the image as one of the characteristics shown by each of the plurality of frame images. can. As described above, the classification of the scenery can be performed, for example, by image analysis. In other words, in Modification 3, the control unit 29 serving as the index acquisition unit determines the characteristic shown by each of the plurality of frame images by analyzing each of the plurality of frame images, and determines the characteristic shown by the characteristic. Based on this, an importance index indicating the importance of each of the plurality of frame images can be obtained.

As described above, according to the third modified example, images with low importance are included in the composite image by using the importance set for each classification item of scenery recorded in the image as the importance index. Continuous display can be reduced. Therefore, in the direction corresponding to the moving direction of the moving object, the range occupied by unnecessary images becomes smaller, making it easier to find a desired image. Therefore, it is possible to provide an image generation device that generates an image that allows a user to search for a desired location in an image with high efficiency and that allows the user to easily find the desired location. .

Note that in the above embodiments and modifications, the case where the image generation device 10 is mounted on a moving body has been described, but the present invention is not limited to this. For example, the image generation device 10 may be installed in a terminal device carried by a user.

Further, in the above embodiments and modified examples, an example was described in which a camera for acquiring a moving image is provided in a moving body, but the present invention is not limited to this. At least, it is sufficient that the acquisition of the video is accompanied by movement. For example, the camera may be installed in a terminal device carried by a user. Furthermore, the photographing direction of the camera is not limited to being directly horizontal to the moving direction, and may be photographed in any direction as long as it is not along the moving direction (that is, the front-back direction). For example, the image may be taken in the upper direction or the lower direction. Furthermore, the image generation device 10 may be provided at a different location from the moving camera, as long as it can acquire a moving image shot by the camera.

Moreover, although the first display 15 has been described as an example provided in the window of the automobile M, the first display 15 is not limited thereto. The first display 15 may be provided so as to be able to display the video reproduced by the control section 29 as a reproduction control section.

Further, the first display 15 is not limited to being transparent, and may be semi-transparent or opaque.

Although an example has been described in which the second display 17A is provided separately from the first display 15, the present invention is not limited to this. For example, a part of the video display surface 15A of the first display 15 may be provided with an area where an image for operation can be displayed, and the playback operation image PC may be displayed.

Note that although an example has been described in which a cumulative movement distance is associated with an information-bearing video and frame images are selected at equal distance intervals based on the information-bearing video, the present invention is not limited to this. The information-bearing video does not need to be associated with distance.

The configurations in the embodiments and modified examples described above are merely examples, and can be selected and changed as appropriate depending on the purpose and the like.

10 Image generation device 13 Camera 15 First display 17 Touch panel 17A Second display 17B Touch pad 18 System bus 19 Input section 21 Speed sensor 22 GPS receiver 23 Output section 24 Speed sensor 25 Operation input section 27 Storage section 29 Control section

Claims (10)

An image generation device that generates a composite image in which a plurality of original images are combined,
an original image acquisition unit that acquires at least a part of continuous landscape images continuously photographed while moving by an imaging device as the plurality of original images ;
an index acquisition unit that acquires an importance index indicating the importance of each of the plurality of original images ;
Each of the plurality of original images is reduced in accordance with the importance index associated with each of the plurality of original images so that a dimension in a first direction corresponding to a moving direction of the imaging device is reduced. a composite image generation unit that generates one composite image by arranging and combining the images in the order in which they were photographed;
An image generation device comprising: 2. The composite image generation unit generates the composite image by reducing each of the plurality of original images at a reduction ratio that increases as the importance level indicated by the importance index decreases. Image generation device. The index acquisition unit determines the characteristics of the plurality of original images based on the characteristics shown by each of the plurality of original images determined by image analysis of each of the plurality of original images acquired by the original image acquisition unit. The image generation device according to claim 1 or 2, wherein an importance index indicating each importance is acquired. The index acquisition unit stores map data in which information regarding the location where each of the plurality of original images acquired by the original image acquisition unit was photographed and the scenery corresponding to the location on the map is stored in advance. The image generation device according to claim 1 or 2, wherein an importance index indicating the importance of each of the plurality of original images is acquired based on the following. The index acquisition unit determines a degree of change indicating the degree of change of each of the plurality of original images acquired by the original image acquisition unit from adjacent original images in the order in which the plurality of original images were photographed. The image generation device according to claim 1 or 2, wherein an index is acquired as the importance index. a movement distance acquisition unit that obtains a cumulative movement distance of the imaging device from the time when shooting of the continuous landscape images is started to the time when each of the frame images forming the continuous landscape images is shot;
an information-bearing video generation unit that generates an information-bearing video that is the continuous landscape image in which each of the frame images is associated with the cumulative travel distance at the time of shooting;
The original image acquisition unit acquires the plurality of original images from the frame images of the information-bearing video at predetermined distance intervals in the cumulative movement distance of the imaging device when the continuous landscape images were captured;
The predetermined distance interval is a distance interval that causes an overlap in the photographing range between each of the plurality of original images and an adjacent original image when the plurality of original images are arranged in the order in which they were photographed. The image generation device according to any one of claims 1 to 5. a reproduction control unit that outputs a video obtained by reproducing the continuous landscape images ;
an operation reception unit that accepts a selection operation for selecting one position in the first direction in the composite image,
7. The reproduction control unit outputs a video obtained by reproducing the continuous landscape images by cuing from the original image corresponding to the position selected via the operation reception unit. The image generation device according to any one of the above. An image generation method in which an image generation device generates a composite image in which a plurality of original images are combined ,
an original image acquisition step of acquiring at least a part of continuous landscape images continuously photographed while moving by an imaging device as the plurality of original images ;
an index acquisition step of acquiring an importance index indicating the importance of each of the plurality of original images ;
Each of the plurality of original images is reduced in accordance with the importance index associated with each of the plurality of original images so that a dimension in a first direction corresponding to a moving direction of the imaging device is reduced. and a composite image generation step of arranging and combining the images in the order in which they were photographed to generate one composite image. An image generation program that causes an image generation device including a computer to generate a composite image in which a plurality of original images are combined , the computer comprising:
an original image acquisition step of acquiring at least a part of continuous landscape images continuously photographed while moving by an imaging device as the plurality of original images ;
an index acquisition step of acquiring an importance index indicating the importance of each of the plurality of original images ;
Each of the plurality of original images is reduced in accordance with the importance index associated with each of the plurality of original images so that a dimension in a first direction corresponding to a moving direction of the imaging device is reduced. and a composite image generation step of arranging and composing the images in the order in which they were photographed to generate one composite image. A computer-readable recording medium storing the image generation program according to claim 9.

Images