JP7602951B2 - Imaging device, imaging method, imaging program, and recording medium - Google Patents

Description

The present invention relates to an imaging device, an imaging method, an imaging program, and a recording medium.

Technology is known that extracts a desired image, such as a best still image, from multiple images captured in advance.

For example, Patent Document 1 discloses an imaging device that, when converting video data into still images, creates a still image by arranging the image obtained at the shooting timing (the timing of the shutter operation) as the main frame, and arranging a predetermined number of frame images obtained before that shooting timing as other frame images.

Patent Publication 2007-318773

In imaging devices like the one described above, the timing of the user's instruction to shoot often does not match the timing when the desired image is actually captured, and one of the issues is that even if the image obtained at the time of the user's instruction is converted into a still image, the image that the user wanted to capture may not be extracted.

The present invention has been made in consideration of the above points, and one of its objectives is to provide an imaging device, imaging method, imaging program, and recording medium that enables an image capturing the scenery that the user wishes to capture to be obtained with a high probability in an imaging device that captures images according to imaging instructions from a user.

The invention described in claim 1 includes a storage unit that stores consecutive images captured recently by an imaging unit that moves with a moving object and captures video outside the moving object, a reception unit that receives an imaging instruction from a user, a setting unit that sets a reference timing within a period from the time the reception unit receives the imaging instruction to a time going back a first time based on a situation inside the moving object or a situation outside the moving object, and an extraction unit that extracts, from the storage unit, a predetermined number of images captured by the imaging unit at one or more timings based on the reference timing as captured images.

The invention described in claim 11 is an imaging method executed by an imaging device, and includes a storage step of storing in a storage unit consecutive images captured recently by an imaging unit that moves with a moving object and captures video outside the moving object; a reception step of receiving an imaging instruction by a user's operation; a setting step of setting a reference timing within the period based on a situation inside the moving object or a situation outside the moving object within a period from a point in time when the imaging instruction was received in the reception step to a point going back a first time; and an extraction step of extracting from the storage unit a predetermined number of images captured by the imaging unit at one or more timings based on the reference timing set in the setting step as captured images.

The invention described in claim 12 is an imaging program executed by an imaging device equipped with a computer, which causes the computer to execute a storage step of storing in a storage unit consecutive images captured recently by an imaging unit that moves with a moving object and captures video outside the moving object, a reception step of receiving an imaging instruction by a user's operation, a setting step of setting a reference timing within the period based on the situation inside the moving object or the situation outside the moving object within the period from the time when the imaging instruction was received in the reception step to a time going back a first time, and an extraction step of extracting from the storage unit a predetermined number of images captured by the imaging unit at one or more timings based on the reference timing set in the setting step as captured images.

1 is a diagram showing a front seat portion of a vehicle in which an imaging device according to an embodiment is mounted; 1 is a block diagram showing a configuration of an imaging device according to an embodiment. FIG. 4 is a diagram showing an example of an image outside a moving body according to an embodiment; 4 is a flowchart showing an example of a routine executed by the imaging device according to the embodiment. 10 is a flowchart showing an example of a subroutine executed by the imaging device according to the embodiment. 10 is a flowchart showing another example of a subroutine executed by the imaging device according to the embodiment. 10 is a flowchart showing another example of the routine executed by the imaging device according to the embodiment.

The following describes in detail an embodiment of the present invention. In the following description and the accompanying drawings, the same reference numerals are used to refer to substantially the same or equivalent parts.

The configuration of the imaging device 10 according to the embodiment will be described with reference to Figures 1 to 3. Figure 1 is a diagram showing the front seat portion of a vehicle M as an example of a moving body in which the imaging device 10 according to the embodiment is mounted.

As shown in FIG. 1, the imaging device 10 is disposed in the center console of the vehicle M. The imaging device 10 is a device that acquires video of the external scenery from the vehicle M and extracts an image from the video as a captured image in response to an imaging instruction from the user.

In the following description, the person issuing the image capture command is referred to as the user, and anyone in the vehicle M, including the user, is referred to as an occupant.

The exterior camera 11 serving as an imaging unit is provided on the back of the rearview mirror RM, i.e., on the surface facing the windshield FG. The exterior camera 11 captures images in the direction of travel of the vehicle M. When the vehicle M is traveling in the forward direction, the exterior camera 11 is positioned so that it can capture images of a range including the front, right front, and left front of the vehicle M. Therefore, the exterior camera 11 can capture images within the field of view of a passenger sitting in the front seat of the vehicle M when looking forward. The exterior camera 11 is connected to the imaging device 10 so that it can communicate with it, and can transmit a signal of the captured image to the imaging device 10.

The imaging device 10 acquires and stores video captured by the exterior camera 11. In other words, in this embodiment, the imaging device 10 has the same functions as a so-called drive recorder.

The exterior camera 11 may be installed anywhere on the vehicle M as long as it is capable of capturing an image of the area in front of the vehicle M. For example, the exterior camera 11 may be disposed on the dashboard DB, or may be installed on the exterior surface (exterior surface) of the vehicle M, such as on the hood or the front bumper.

The in-vehicle camera 13 serving as an imaging unit is provided near the exterior camera 11 above the windshield FG. The in-vehicle camera 13 is oriented to capture images of the interior of the vehicle M. The in-vehicle camera 13 is positioned so that it can capture images of passengers, including the user, sitting in the front seat of the vehicle M.

The in-vehicle camera 13 is communicably connected to the imaging device 10 and is capable of transmitting a signal of the captured image to the imaging device 10. The imaging device 10 acquires and stores the video captured by the in-vehicle camera 13.

The in-vehicle camera 13 may be installed anywhere in the vehicle M as long as it is capable of capturing images of occupants, including the user, inside the vehicle M. For example, the exterior camera 11 may be located on the dashboard DB.

The microphone 15 is a voice input device arranged to collect sounds inside the vehicle M. The microphone 15 is communicably connected to the imaging device 10 and can convert the collected sounds into electrical signals and transmit them to the imaging device 10.

The imaging device 10 receives a signal indicating the sound inside the vehicle M from the microphone 15, performs voice recognition on the voice based on the user's speech, and when it recognizes a specific word, it recognizes the voice as an imaging instruction from the user.

The imaging device 10 also stores information indicating sounds within the vehicle M based on the signal received from the microphone 15. For example, the sounds within the vehicle M include voices uttered by occupants within the vehicle M or sounds made by the occupants.

The touch panel display 17 is made up of a touch panel and a display. The touch panel display 17 is communicatively connected to the imaging device 10. The display of the touch panel display 17 displays images supplied from the imaging device 10. The touch panel of the touch panel display 17 transmits a signal indicating an input operation by touching the touch panel to the imaging device 10.

For example, the touch panel display 17 displays a button on the display that indicates an image capture instruction to the imaging device 10. For example, when the button indicating the image capture instruction is pressed, the imaging device 10 accepts the image capture instruction from the user via the touch panel of the touch panel display 17.

When the imaging device 10 receives an input operation indicating an imaging instruction from the user by voice operation or touch operation on the touch panel via the microphone 15 or the touch panel display 17, it captures a still image as a captured image using the stored video. Specifically, the imaging device 10 extracts a predetermined number of frame images from the many frame images that make up the stored video to generate a still image. The imaging device 10 transmits the extracted still images to a server (not shown), i.e., uploads them.

Figure 2 is a block diagram showing the configuration of the imaging device 10. As shown in Figure 2, the imaging device 10 is configured with each unit connected via a system bus 19.

The input unit 21 is an interface that communicatively connects the imaging device 10 to equipment provided in the vehicle M1.

The input unit 21 communicatively connects the imaging device 10 to the exterior camera 11 and the interior camera 13. The input unit 21 also communicatively connects the imaging device 10 to the microphone 15.

The input unit 21 communicatively connects the imaging device 10 to the touch panel of the touch panel display 17.

The input unit 21 is an interface that communicatively connects the imaging device 10 and the distance measurement sensor 23. The distance measurement sensor 23 is a sensor that measures the distance between the vehicle M and an object present in the imaging direction of the exterior camera 11 around the vehicle M.

In this embodiment, an object that is present in the imaging direction of the exterior camera 11 and is detected by the distance sensor 23 is considered to be an obstacle that obstructs the visibility of the scenery from the vehicle M. In this embodiment, when the distance between the vehicle M and the obstacle is short, it is considered that the obstacle has a large effect on the visibility of the scenery from the vehicle M. In addition, when the distance between the vehicle M and the obstacle is long, it is considered that the obstacle has a small effect on the visibility of the scenery from the vehicle M.

The distance sensor 23 is a sensor such as an ultrasonic sensor (SONAR; Sound Navigation And Ranging), LiDAR (Light Detection And Ranging), or millimeter wave radar, and is attached to the vehicle M so that it can detect the presence or absence of an obstacle in the imaging direction of the exterior camera 11 and measure the distance.

In this embodiment, the distance measurement sensor 23 is, for example, a millimeter wave radar. For example, the distance measurement sensor 23 is attached to the front bumper of the vehicle M. Note that the distance measurement sensor 23 may be any one of a sonar, a millimeter wave radar, and a LiDAR, or any combination of these.

The imaging device 10 receives a signal indicating distance from the distance measuring sensor 23 and stores distance information indicating the distance between the obstacle on the side of vehicle M and vehicle M.

The large-capacity storage device 25 is a storage device configured, for example, by a hard disk device, a solid state drive (SSD), a flash memory, etc. The large-capacity storage device 25 stores various programs executed in the imaging device 10. Note that the various programs may be obtained, for example, from another server device or the like via a network, or may be recorded on a recording medium and read via various drive devices.

For example, the large-capacity storage device 25 stores an imaging program that is executed by the imaging device 10 when extracting an image from a video and creating a still image. The large-capacity storage device 25 stores information necessary to execute various programs.

The large-capacity storage device 25 also stores information acquired by equipment installed in the vehicle M while the vehicle M is traveling.

Specifically, the large-capacity storage device 25 has an outside-vehicle image storage unit 25A. The outside-vehicle image storage unit 25A stores video images captured by the outside-vehicle camera 11. The outside-vehicle image storage unit 25A holds, for example, a certain period of video images. The outside-vehicle image storage unit 25A also stores, for example, a predetermined amount of video images.

The large-capacity storage device 25 has a temporary image storage unit 25B. In the temporary image storage unit 25B, for example, some images included in the video stored in the outside-vehicle image storage unit 25A are extracted and temporarily stored. For example, the temporary image storage unit 25B is a memory portion in which new images are stored from moment to moment, like a ring buffer.

The temporary image storage unit 25B can store, for example, a number of frame images from a video of several tens of seconds. For example, the temporary image storage unit 25B stores frame images extracted from the video at predetermined time intervals. In the temporary image storage unit 25B, old frame images are overwritten with new frame images, thereby updating the images from moment to moment.

For example, when the video images stored in the outside-vehicle image storage unit 25A are captured at a frame rate of about 15 to 30 fps (e.g., 27.5 fps), the temporary image storage unit 25B may store, for example, 10 frame images, one per second, for about 10 seconds. For example, about six images per second may be stored at equal time intervals for several tens of seconds.

In other words, the outside-vehicle image storage unit 25A and the temporary image storage unit 25B of the large-capacity storage device 25 store consecutive images recently captured by an imaging unit moving with the vehicle. Note that consecutive images refer to images that are arranged in chronological order.

The large-capacity storage device 25 has a distance information storage unit 25C. The distance information storage unit 25C stores distance information indicating the distance between the vehicle M and an obstacle ahead of the vehicle M, which is acquired by the distance measurement sensor 23. The distance information is stored, for example, in association with the distance between the vehicle M and the obstacle ahead of the vehicle M and the time at which the distance was measured.

The large-capacity storage device 25 has an in-vehicle information storage unit 25D. The in-vehicle information storage unit 25D stores information acquired about the situation inside the vehicle M. Specifically, the in-vehicle information storage unit 25D stores video images of the occupants inside the vehicle M captured by the in-vehicle camera 13. In other words, the in-vehicle information storage unit 25D stores video images of the inside of the moving body.

The in-vehicle information storage unit 25D holds, for example, video images for a certain period of time. The in-vehicle information storage unit 25D also stores, for example, video images of a predetermined capacity.

The in-vehicle information storage unit 25D also stores audio information generated based on the sound collected by the microphone 15. The in-vehicle information storage unit 25D stores, for example, audio information for a certain period of time. The in-vehicle information storage unit 25D also stores, for example, audio information of a predetermined capacity. The audio information may be stored, for example, in association with a video image captured of the interior of the vehicle M.

In addition, audio information about sounds collected by the microphone of the in-vehicle camera 13 along with the captured video images from inside the vehicle M may be stored in the in-vehicle information storage unit 25D in association with the captured video images.

The control unit 27 is configured with a CPU (Central Processing Unit) 27A, a ROM (Read Only Memory) 27B, a RAM (Random Access Memory) 27C, etc., and functions as a computer. The CPU 27A reads and executes various programs stored in the ROM 27B and the large-capacity storage device 25 to realize various functions.

The control unit 27 reads out and executes the imaging program stored in the mass storage device 25, and extracts the captured image based on the information acquired via the input unit 21 in response to an imaging instruction from the user.

The control unit 27 stores the video captured by the outside-vehicle camera 11 in the outside-vehicle image storage unit 25A of the large-capacity storage device 25 via the input unit 21. The control unit 27 also extracts some images included in the video stored in the outside-vehicle image storage unit 25A and temporarily stores them in the temporary image storage unit 25B.

In other words, the control unit 27 executes a storage step in which consecutive images captured recently by the imaging unit moving with the moving body are stored in the outside-vehicle image storage unit 25A or the temporary image storage unit 25B, which serve as a storage unit.

The control unit 27 acquires the distance to an obstacle present in the imaging direction of the exterior camera 11 of the vehicle M measured by the distance measuring sensor 23 via the input unit 21, and stores information indicating the distance in the distance information storage unit 25C.

In other words, the control unit 27 functions as a measurement unit that measures the distance between the moving body and an obstacle that exists in the imaging direction of the image outside the moving body captured by the exterior camera 11. The control unit 27 stores, for example, the distance between the obstacle in front of the vehicle M and the vehicle M in association with the time when the distance was measured.

The control unit 27 stores video images of the inside of the vehicle M captured by the in-vehicle camera 13 in the in-vehicle information storage unit 25D via the input unit 21.

The control unit 27 generates audio information from sounds inside the vehicle M collected by the microphone 15 via the input unit 21, and stores the information in the vehicle interior information storage unit 25D. The audio information is, for example, information in which the sound is associated with the time when the sound was collected. The audio information may be stored in association with a video image captured of the interior of the vehicle M, for example.

In addition, the control unit 27 may generate audio information indicating the sounds inside the vehicle M based on the audio recorded by the in-vehicle camera 13 together with the capture of moving images.

The control unit 27 acquires sounds inside the vehicle M collected by the microphone 15 via the input unit 21 and performs voice recognition. When the control unit 27 recognizes a specific voice based on the speech of a user inside the vehicle M, it accepts an image capture instruction from the user. For example, when the control unit 27 recognizes a predetermined wake word and voice command, it accepts an image capture instruction from the user. In other words, the control unit 27 accepts an image capture instruction by voice operation of the user.

The control unit 27 recognizes an input operation using the touch panel display 17 via the input unit 21. When the control unit 27 recognizes a specific contact operation performed on the touch panel of the touch panel display 17, it accepts an image capture instruction from the user. For example, when the control unit 27 recognizes that an image capture instruction button displayed on the display of the touch panel display 17 has been pressed, it accepts an image capture instruction from the user.

In this way, the control unit 27 accepts an image capture instruction from the user by recognizing a user's operation, such as a voice operation by the user's speech or a touch operation on a touch panel. The control unit 27 functions as a reception unit that executes a reception step of accepting an image capture instruction by a user's operation.

When the control unit 27 receives an image capture instruction from the user, it extracts a predetermined number of images as captured images from the images stored in the temporary image storage unit 25B of the large-capacity storage device 25.

More specifically, the control unit 27 sets a reference timing within a period from the time when the image capture instruction is received to a time a predetermined length of time (first time) prior to the time when the image capture instruction is received, and extracts a predetermined number of images captured at one or more timings based on the reference timing. The control unit 27 functions as a setting unit that executes a setting step that sets the reference timing.

However, after the user issues an image capture instruction, the control unit 27 of the imaging device 10 requires processing time to recognize the image capture instruction before the control unit 27 accepts the image capture instruction issued by the user. Furthermore, an arbitrary amount of time is also required from the time the user decides to capture an image until the user actually issues the image capture instruction. In particular, when the vehicle M is traveling at high speed or traveling around a curve, the delay due to these times may cause a large discrepancy between the scenery captured by the external camera 11 at the time the control unit 27 of the imaging device 10 recognizes the image capture instruction and the scenery that the user wanted to capture.

In order to address this problem, in this embodiment, for example, if the time when an image capture instruction is received is defined as time _t0 and the time going back from time _t0 by the first hour is defined as time _t1 , the control unit 27 sets time _t3 within the period from time _t1 to time _t0 as the reference timing, which is the reference time for extracting the captured image.

The first time may be set to a sufficient length of time, taking into consideration, for example, the time required for voice recognition or touch operation recognition to receive an image capture instruction from the user, as well as the time from when the user desires to capture an image to when the image capture instruction is actually given. The first time may be, for example, about 5 seconds, or may be an even longer time. The reference timing may be, for example, a time, or may be a time indicated by the elapsed time from the start of driving.

The control unit 27 sets the reference timing based on the situation inside the vehicle M (the situation inside the moving body) or the situation outside the vehicle M (the situation outside the moving body). The setting of the reference timing is described below.

[Setting the reference timing based on the situation inside the moving body]
The control unit 27 acquires the situation inside the vehicle from the image of the inside of the vehicle captured by the in-vehicle camera 13 .

The control unit 27 performs image analysis on the video of the inside of the vehicle M during the period from when the image capture instruction is received to the first time back, for example, and detects a predetermined movement or a predetermined change in facial expression of the occupant in the image. The control unit 27 sets the timing (e.g., the time) at which the predetermined movement or the predetermined change in facial expression is detected as a reference timing that serves as a basis for extracting the captured image.

The predetermined actions include, for example, actions that are expected when finding a noteworthy scene, such as pointing a finger outside the vehicle M. The predetermined changes in facial expression include, for example, facial expressions that are expected when witnessing a noteworthy scene, such as a surprised expression or a happy expression.

As a criterion for determining a predetermined action, for example, an image showing a person pointing a finger outside the vehicle may be stored in advance in the large-capacity storage device 25. Also, as a criterion for determining a predetermined facial expression, for example, an image showing a person with a surprised face may be stored in advance in the large-capacity storage device 25.

The control unit 27 may, for example, compare an image of a predetermined action or a predetermined facial expression that is stored in advance in the large-capacity storage device 25 with an image captured by the in-vehicle camera 13, and may determine that a change in the predetermined action or the predetermined facial expression has been detected when, for example, the degree of agreement is equal to or greater than a predetermined value. The control unit 27 may also detect a change in facial expression using AI or the like.

In this way, the control unit 27, which serves as a setting unit, sets the timing at which a specific movement of an occupant of the moving body or a change in the occupant's facial expression is detected in the video of the inside of the moving body as the reference timing.

By anticipating in advance the actions or facial expressions that a user is likely to make when encountering a noteworthy scene, and extracting a captured image using the timing at which such actions or facial expressions are detected as a reference timing, the probability of extracting an image captured at the timing the user wishes to capture can be increased.

The control unit 27 also acquires information about the situation inside the moving body from sounds inside the moving body collected by the microphone 15.

The control unit 27 performs voice recognition on sounds inside the vehicle M during a period from when the image capture instruction is received to a time going back a first time, for example, and detects a predetermined sound related to a voice uttered by an occupant inside the vehicle M or a sound made by an occupant. The control unit 27 sets the timing (e.g., time) at which the predetermined sound is detected as a reference timing that serves as a basis for extracting a captured image.

Examples of the predetermined sounds include sounds that are expected to be uttered by passengers when they encounter a noteworthy scene, such as a cheer like "Wow!" or a specific word like "Beautiful." Examples of the predetermined sounds include sounds that are expected to be uttered by passengers when they encounter a noteworthy scene, such as the sound of clapping.

As a criterion for determining a predetermined sound, for example, a model of the acoustic features when the predetermined sound is emitted may be stored in advance in the mass storage device 25. The control unit 27 may compare the acoustic features of the sound picked up by the microphone 15 with the acoustic features stored in advance, and may determine that the predetermined sound has been detected when, for example, a degree of match equal to or greater than a predetermined value is obtained.

In this way, the control unit 27, which serves as a setting unit, sets the timing at which a specific sound is detected among the sounds collected by the sound collection unit that moves together with the moving body, as the reference timing.

By anticipating sounds that are likely to indicate that the user wishes to capture an image, and using the timing at which such a sound is detected as the reference timing to extract a captured image, the probability of extracting an image captured at the timing at which the user wishes to capture an image can be increased.

[Setting the reference timing based on the situation outside the moving body]
The control unit 27 acquires the situation outside the vehicle from the video of the outside of the vehicle captured by the exterior camera 11 .

The control unit 27 performs image analysis on the image outside the vehicle M within a period from the time when the image capture instruction is received to the time going back by the first time, detects obstacles among the objects included in the image that may impede the view of the scenery, and detects the apparent size of the obstacle or the distance between the obstacle included in the image and the vehicle M. The control unit 27 sets the timing when the apparent size of the obstacle is smallest or the timing when the distance between the obstacle and the vehicle M is longest as the reference timing that serves as the basis for extracting the captured image.

An example of such an obstacle is an object such as another vehicle that is present in front of vehicle M and obstructs the view of the scenery from vehicle M. For example, when a mountain, building, or other feature is visible in front, to the right, or to the left of vehicle M, and another vehicle or the like is present in the vicinity of vehicle M and obstructs part of the scenery captured by vehicle exterior camera 11 from vehicle M, the other vehicle or the like becomes an obstacle.

For example, images of objects that are candidates for obstacles may be stored in advance in the mass storage device 25 in order for the control unit 27 to detect the obstacle. For example, images of a vehicle ahead, a vehicle on the left front, or an oncoming vehicle on the right front may be stored in advance in the mass storage device 25 as registered images that serve as models of obstacles. The control unit 27 may detect the obstacle by, for example, performing a matching process between the registered images stored in advance and an image captured by the exterior camera 11.

Figure 3 is a diagram showing an example of a frame image F included in the video outside the vehicle M captured by the exterior camera 11. As shown in Figure 3, the frame image F includes an obstacle OB, which is another vehicle traveling in front of the vehicle M.

When the control unit 27 detects an obstacle OB, the control unit 27 may determine, for example, the proportion of the obstacle OB that is occupied in the entire frame image F as the apparent size of the obstacle OB. For example, the control unit 27 identifies an area AR in which the obstacle OB is captured, and determines the proportion of the area of the area AR to the area of the entire frame image F as the apparent size of the obstacle OB. Then, the control unit 27 sets the timing at which the frame image in which the apparent size of the obstacle OB is smallest is captured as the reference timing.

Alternatively, when the control unit 27 detects an obstacle OB included in the frame image F, the control unit 27 may calculate the distance between the obstacle OB and the vehicle M. For example, the control unit 27 may set, as the reference timing, the timing at which a frame image in which the distance between the obstacle OB and the vehicle M is the longest, calculated based on the frame image F, is captured.

In this way, the control unit 27, which serves as a setting unit, detects the apparent size of an obstacle included in the image outside the moving body, or the distance between the obstacle and the moving body, and sets as the reference timing the timing at which the apparent size of the obstacle is smallest, or the distance between the obstacle and the moving body is longest, within the period from the time when the image capture instruction is received to the time going back by the first time.

This allows the captured image to be extracted using as the reference time the timing at which the view the user wishes to capture is least likely to be blocked by an obstacle. This therefore increases the probability of extracting an image captured at the time the user wishes to capture it.

The control unit 27 also obtains information about the situation outside the moving body from the distance between the vehicle M and an obstacle in front of the vehicle M measured by the distance sensor 23.

The control unit 27, for example, refers to the distance information storage unit 25C and sets the timing at which the distance to an obstacle ahead of the vehicle M is the longest within the period from the time when the image capture instruction is received to the time going back by the first time (hereinafter also referred to as the first period) as the reference timing that serves as the basis for extracting the captured image.

In this way, the control unit 27 as a measurement unit measures the distance between the moving body and an obstacle that exists in the imaging direction of the image outside the moving body captured by the exterior camera 11 via the distance sensor 23, and the control unit 27 as a setting unit sets the timing at which the distance between the obstacle and the moving body within the first period measured by the measurement unit is the longest as the reference timing.

This allows the captured image to be extracted using the time when the obstacle obstructing the view of the scenery the user wishes to capture is at its farthest as the reference time, thereby increasing the probability of extracting an image captured at the time the user wishes to capture it.

[Extraction of captured images]
The control unit 27 functions as an extraction unit that executes an extraction step of extracting a predetermined number of images captured by the outside camera 11 at one or more times based on a reference time.

Specifically, for example, the control unit 27 extracts one image captured at the reference timing as the captured image.

For example, the control unit 27 may extract images captured at multiple timings within a predetermined time before and after a reference timing as a center, and extract the images as a predetermined number of captured images. For example, the multiple timings may be equally spaced.

The control unit 27 extracts a predetermined number of images from the images stored in the temporary image storage unit 25B. In the temporary image storage unit 25B, a large number of frame images that make up the video are thinned out in advance and stored, so that the extraction of captured images is performed efficiently. The control unit 27 may also extract a predetermined number of images from the video stored in the outside-vehicle image storage unit 25A.

The output unit 29 is connected to the display of the touch panel display 17 provided in the vehicle M. The output unit 29 is an interface for supplying various information to be displayed on the display in accordance with commands from the control unit 27.

The output unit 29 outputs an image to be displayed on the display of the touch panel display 17 in response to an instruction from the control unit 27. For example, the output unit 29 supplies an image to the display for presenting to the user an image capture instruction button for accepting an image capture instruction.

The output unit 29 may be communicatively connected to a speaker (not shown) provided in the vehicle M, and may, for example, notify the user by voice that an image capture instruction has been received or that image capture has been completed.

The transmission/reception unit 31 is a network adapter such as a network interface card (NIC) connected to a wireless device (not shown). The transmission/reception unit 31 transmits and receives data between the imaging device 10 and the outside in accordance with commands from the control unit 27.

For example, the transmission/reception unit 31 functions as a transmission unit that transmits an image extracted as a captured image in the imaging device 10 to an external device such as a server.

With reference to Figures 4 to 7, the information processing executed by the control unit 27 when imaging is performed by the imaging device 10 in this embodiment will be described.

Figure 4 is a flowchart showing an imaging routine RT1, which is an example of a routine executed by the control unit 27. The control unit 27 starts the imaging routine RT1, for example, when the imaging device 10 is powered on. In addition, the control unit 27 starts storing moving images captured by the outside-vehicle camera 11 in the outside-vehicle image storage unit 25A, for example, when the imaging device 10 is powered on. In addition, when the control unit 27 starts storing moving images in the outside-vehicle image storage unit 25A, for example, it starts temporarily storing images included in the moving images stored in the outside-vehicle image storage unit 25A in the temporary image storage unit 25B.

In addition, for example, when the imaging device 10 is powered on, the control unit 27 starts storing distance information in the distance information storage unit 25C and storing video and audio information inside the vehicle M in the in-vehicle information storage unit 25D.

When the control unit 27 starts the imaging routine RT1, it determines whether or not an imaging instruction has been received from the user (step S101). In step S101, for example, the control unit 27 performs voice recognition on the voice input from the microphone 15, and when it recognizes, for example, a wake word followed by a predetermined voice command indicating an imaging instruction, it determines that an imaging instruction from the user has been received.

In step S101, for example, the control unit 27 may determine that an image capture instruction has been received if it recognizes that an image capture instruction button displayed on the display of the touch panel display 17 has been pressed by a contact operation on the touch panel.

In step S101, the control unit 27 functions as a reception unit that receives an image capture instruction from a user operation.

If the control unit 27 determines in step S101 that an image capture instruction has not been received (step S101: NO), it repeats step S101 and again determines whether an image capture instruction has been received.

When the control unit 27 determines in step S101 that an image capture instruction has been received (step S101: YES), it executes an image extraction subroutine and extracts the captured image from the images stored in the temporary image storage unit 25B (step S102).

In step S102, for example, the control unit 27 sets a reference timing within the period from the time when the image capture instruction is received to the time going back a first time based on the situation inside the vehicle M or the situation outside the vehicle M. The control unit 27 then extracts, as a predetermined number of captured images, images captured by the exterior camera 11 at one or more times based on the reference timing.

After executing step S102, the control unit 27 transmits the image extracted in step S102 to the outside (step S103). In step S103, for example, the control unit 27 transmits the extracted image to a preset server. In step S103, for example, the control unit 27 may transmit the extracted image to a terminal device owned by the user.

After executing step S103, the control unit 27 ends the imaging routine RT1 and starts a new imaging routine RT1.

Figure 5 is a flowchart showing an image extraction subroutine RT2, which is an example of an image extraction subroutine executed by the control unit 27 in step S102 of the imaging routine RT1.

When the control unit 27 starts the image extraction subroutine RT2, it refers to the situation inside the vehicle M during the period (first period) from the time when the image capture instruction was received to the time going back by the first time (step S201).

In step S201, the control unit 27 reads out the video captured by the in-vehicle camera 13 during the first period from the in-vehicle information storage unit 25D.

In addition, in step S201, the control unit 27 reads out audio information indicating the sounds collected by the microphone 15 during the first period from the in-vehicle information storage unit 25D.

As the in-vehicle situation in step S201, for example, either video or audio information about the inside of the vehicle may be referenced, or both may be referenced.

After executing step S201, the control unit 27 determines whether or not a predetermined event has occurred in the vehicle M within the first period (step S202). In step S202, for example, the control unit 27 performs image analysis on the video read out in step S201, and determines that a predetermined event has occurred if a predetermined movement or a predetermined change in facial expression of an occupant is detected in the image. In step S202, for example, if a pointing movement or a change to a surprised facial expression is detected, it is determined that a predetermined event has occurred.

In step S202, for example, the control unit 27 performs a voice recognition process on the voice information read out in step S201, and when a predetermined sound related to a voice uttered by an occupant in the vehicle M or a sound made by an occupant is detected, it is determined that a predetermined event has occurred. In step S202, for example, when a voice such as a cheer or a sound such as applause is detected, it is determined that a predetermined event has occurred.

When the control unit 27 determines in step S202 that a predetermined event has occurred (step S202: YES), it sets the time when the predetermined event has occurred as a reference timing (step S203). In step S203, for example, the time when an image including a predetermined posture or a predetermined facial expression of the occupant is captured is set as the reference timing. Also, in step S203, for example, the time when a predetermined sound is collected is set as the reference timing.

After executing step S203, the control unit 27 extracts one or more images as captured images from the images stored in the temporary image storage unit 25B of the large-capacity storage device 25 based on the reference timing (step S204). In step S204, for example, one image captured at the reference timing is extracted.

In step S204, for example, images captured at each of a plurality of timings including the reference timing may be extracted. For example, a predetermined number of images captured within a predetermined period before and after the reference timing may be extracted as the images captured at each of the plurality of timings.

If the control unit 27 determines in step S202 that no specific event occurred inside the vehicle (step S202: NO), it extracts an image captured a second time period prior to the time when the image capture instruction was received, which is set separately from the first time period (step S205).

For example, the second time is set to a length of time that is preset by statistically considering the time it takes for the control unit 27 to perform voice recognition or recognize a touch operation on the touch panel to receive an image capture instruction, as well as the time from when the user desires to capture an image to when the user issues an image capture instruction. For example, if a sufficiently long first time is set, the second time may be set to a length of time that is shorter than the first time.

After executing step S204 or step S205, the control unit 27 ends the image extraction subroutine RT2.

The image extraction subroutine RT2 detects a specific event inside the vehicle caused by the actions of the occupants, and extracts an image using the timing at which the detected event occurred as the reference timing, thereby increasing the probability of extracting an image captured at the timing desired by the user.

Furthermore, by step S205 of the image extraction subroutine RT2, for example, even if the user has not taken any action that would appear in the image or sound inside the vehicle at the time when the user wishes to capture an image, it is possible to extract an image captured at a point in time going back a second time, taking into consideration the time involved in voice recognition and the time from when the user wishes to capture an image until when the user issues an image capture instruction. In other words, even if the situation inside the vehicle does not satisfy the conditions for the control unit 27 as a setting unit to set the reference timing, it is possible to increase the probability of extracting an image captured at the time when the user wishes to capture an image.

Note that the image extraction subroutine RT2 extracts one or more images. However, for example, if only one image is extracted, the capacity required to save the extracted image can be reduced. Also, if only one image is extracted, it becomes easier for the user to handle the captured image.

Figure 6 is a flowchart showing an image extraction subroutine RT3, which is another example of an image extraction subroutine executed by the control unit 27 in step S103 of the imaging routine RT1. The image extraction subroutine RT3 proceeds in the same manner as the image extraction subroutine RT2 (see Figure 5) except for steps S301 and S302, and therefore some of the explanation will be omitted.

When the control unit 27 starts the image extraction subroutine RT3, it refers to the situation outside the vehicle during the period (first period) from the time when the image capture instruction was received to the time going back by one hour (step S301).

In step S301, the control unit 27 reads out the video captured by the exterior camera 11 during the first period from the exterior video storage unit 25A.

In addition, in step S301, the control unit 27 reads out distance information from the distance information storage unit 25C, the distance information indicating the distance between the vehicle M and an obstacle in front of the vehicle M measured by the distance measurement sensor 23 during the first period.

As the situation outside the vehicle in step S301, for example, either the image outside the vehicle captured by the exterior camera 11 or the distance information to an obstacle ahead of the vehicle M may be referenced, or both may be referenced.

After executing step S301, the control unit 27 determines whether there was a time during the first period when there were few obstacles ahead of the vehicle M (step S302). In step S302, for example, the control unit 27 performs image analysis on the video read out in step S301, and determines that there was a time when there were few obstacles in the image when, for example, the percentage of obstacles in the image was below a predetermined percentage.

In step S302, for example, if there is a time when the distance indicated by the distance information read in step S302 is a predetermined distance or greater, the control unit 27 determines that there was a time when there were few obstacles.

When the control unit 27 determines in step S302 that there was a time when there were few obstacles (step S302: YES), it sets a reference timing based on the situation outside the vehicle M in the first period (step S303). In step S303, for example, the time when there were the fewest obstacles is set as the reference timing.

In step S303, for example, the timing at which an image of the outside of vehicle M in which the proportion of the obstacle in the image is smallest is captured is set as the reference timing. In step S303, for example, the timing at which the distance between the obstacle and vehicle M is longest is set as the reference timing.

After executing step S303, the control unit 27 extracts one or more images as captured images from the images stored in the temporary image storage unit 25B of the large-capacity storage device 25 based on the reference timing (step S304). In step S304, for example, one image captured at the reference timing is extracted.

In step S304, for example, images captured at each of a plurality of timings including the reference timing may be extracted. For example, a predetermined number of images captured within a predetermined period before and after the reference timing may be extracted as the images captured at each of the plurality of timings.

If the control unit 27 determines in step S302 that there was no time when there were few obstacles (step S302: NO), it extracts an image captured at a second time before the time when the image capture instruction was received, which is set separately from the first time (step S305).

After executing step S304 or step S305, the control unit 27 ends the image extraction subroutine RT3.

According to the image extraction subroutine RT3, information about an obstacle in front of the vehicle M is detected as a situation outside the vehicle M, and an image can be extracted using as a reference timing the timing when the obstacle has the least effect on the visibility of the scenery from inside the vehicle M. This makes it possible to increase the probability of extracting an image captured at the timing desired by the user.

Furthermore, by using step S305 of the image extraction subroutine RT3, for example, even if an obstacle such as another vehicle is present near vehicle M at the time when the user wishes to capture an image, an image captured at a point in time going back a second time can be extracted, taking into consideration the time required for voice recognition and the time from when the user wishes to capture an image until when the user issues an image capture instruction. In other words, even if the situation outside the moving body does not satisfy the conditions for the control unit 27 as a setting unit to set the reference timing, the probability of extracting an image captured at the time when the user wishes to capture an image can be increased.

The image extraction subroutine RT2 described in FIG. 5 and the image extraction subroutine RT3 described in FIG. 6 may be executed in parallel or in combination. For example, when it is determined in the image extraction subroutine RT2 that a predetermined event has not occurred in the vehicle (step S202: NO), the image extraction subroutine RT3 may be executed.

Figure 7 is a flowchart showing an imaging routine RT4, which is another example of a routine executed by the control unit 27. The imaging routine RT4 differs from the imaging routine RT1 described in Figure 4 in that it includes steps S402 and S403, but proceeds in the same way in other respects, so some of the description will be omitted.

When the control unit 27 starts the imaging routine RT4, it determines whether or not an imaging instruction has been received from the user (step S401).

In step S401, the control unit 27 accepts an imaging instruction based on the user's speech or a touch operation by the user. For example, the control unit 27 performs voice recognition on the voice spoken by the user input from the microphone 15, and when the control unit 27 recognizes, for example, a wake word followed by a predetermined voice command indicating an imaging instruction, it determines that an imaging instruction from the user has been accepted. For example, the control unit 27 determines that an imaging instruction has been accepted when it recognizes an imaging instruction by a touch operation on the touch panel of the touch panel display 17.

When the control unit 27 determines in step S401 that an image capture instruction has been received (step S401: YES), the control unit 27 determines whether the received image capture instruction is an image capture instruction based on the user's speech (step S402). In step S401, for example, if the control unit 27 has performed voice recognition in step S401, the control unit 27 determines that the image capture instruction is based on the user's speech.

When the control unit 27 determines in step S402 that the imaging instruction is made by speech (step S402: YES), it sets the length of the first time period when referring to the situation inside the vehicle M or the situation outside the vehicle M going back a first time from the time when the imaging instruction was received to be longer than when the imaging instruction was received by the user's touch operation (step S403).

In step S403, the first time is set to be longer by the amount of time it takes for the user's image capture instruction to be recognized by voice recognition to exceed the time it takes for a touch operation on the touch panel to be recognized, for example.

If the control unit 27 determines in step S402 that the image capture instruction is not based on the user's speech (step S402: NO), or after executing step S403, it executes an image extraction subroutine and extracts an image as a captured image (step S404).

In step S404, a subroutine such as the image extraction subroutine RT2 shown in FIG. 5 or the image extraction subroutine RT3 shown in FIG. 6 is executed. In step S404, for example, when the image extraction subroutine RT2 is executed and the vehicle interior situation is referenced going back a first time (step S201), the first time that was set long in step S403 is applied.

In step S404, for example, when the image extraction subroutine RT3 is executed and the situation outside the vehicle is referenced going back a first time (step S301), the first time that was set long in step S403 is applied.

After executing step S404, the control unit 27 transmits the extracted image to an external device such as a server (step S405), ends the imaging routine RT4, and repeats a new imaging routine RT4.

According to the imaging routine RT4, the first time can be appropriately set according to the operation method used when the user issues an imaging instruction. Therefore, even if the time required for voice recognition is longer than the time required for touch operation recognition, for example, the reference timing can be appropriately set and an image can be extracted based on the reference timing. Therefore, it is possible to increase the probability of extracting an image captured at the timing desired by the user.

Note that, for example, if an image capture instruction based on the user's speech has been accepted in step S404 of the imaging routine RT4, the length of the second time in the subroutine (step S205 or step S305) may also be set to be longer than when an image capture instruction based on the user's touch operation has been accepted. This can increase the probability of extracting an image captured at the timing the user wishes to capture, even if the time required for voice recognition is longer than the time required for recognition of the touch operation.

As described above in detail, the imaging device 10 of this embodiment includes a storage unit that stores consecutive images captured recently by an imaging unit that moves with the moving object and captures video outside the moving object, a reception unit that receives an imaging instruction through a user's operation, a setting unit that sets a reference timing within the period based on the situation inside the moving object or the situation outside the moving object within the period from the time the reception unit receives the imaging instruction to a time going back a first time, and an extraction unit that extracts from the storage unit a predetermined number of images captured by the imaging unit at each of one or more timings based on the reference timing as captured images.

Accordingly, the imaging device 10 can appropriately set the reference timing based on the situation inside the moving body or the situation outside the moving body as the timing or a timing close to the timing when the user wishes to capture an image within the period prior to the time when the imaging instruction was received. Therefore, by extracting an image captured at a timing based on the reference timing, the probability of extracting an image captured at the timing when the user wishes to capture an image can be increased.

Accordingly, the imaging device 10 can provide an imaging device, imaging method, imaging program, and recording medium that can capture an image of the scenery that the user wishes to capture with a high probability in an imaging device that captures images according to imaging instructions from the user.

The first time period can be set to any length of time, for example, a sufficiently long time that takes into account the time required to accept the user's imaging instruction as well as the time between when the user desires to capture an image and when the imaging instruction is actually given.

In the above embodiment, the imaging device 10 is described as being capable of receiving imaging instructions from a user via the touch panel display 17, but this is not limiting. For example, the imaging device 10 does not need to be connected to the touch panel display 17. For example, the vehicle M does not need to be equipped with a touch panel display 17. For example, the imaging device 10 may receive imaging instructions only through voice recognition. Also, for example, the imaging device 10 may be configured to be able to receive imaging instructions by button operation using, for example, a push button that is not a touch panel display, in addition to voice recognition.

In the above embodiment, the imaging device 10 extracts captured images from images stored in the temporary image storage unit 25B of the large-capacity storage device 25, but this is not limiting. For example, the imaging device 10 may extract captured images from videos that are sequentially stored in the outside-vehicle image storage unit 25A and held for a certain period of time.

In the above embodiment, an example of generating a still image from an image extracted as a captured image has been described, but the present invention is not limited to this. For example, the imaging device 10 may generate a video made up of a plurality of frame images extracted as captured images.

In the above embodiment, the imaging device 10 transmits the extracted captured image to a server, but this is not limited to the above example. The imaging device 10 may transmit the extracted captured image to another external device, such as a terminal owned by the user, or may store the image in a storage unit. The imaging device 10 may also display the extracted captured image on a display provided on a moving object.

In the above embodiment, the imaging unit that captures images outside the moving body has an external camera 11 that captures images in a direction forward of the vehicle M, but this is not limited to the above. For example, the imaging device 10 may store consecutive images captured recently by an imaging unit that captures images to the side or rear of the moving body, and extract the captured images.

In addition, although an example has been described in which distance information is stored in the distance information storage unit 25C, the distance information may be stored in the outside-vehicle image storage unit 25A or the temporary image storage unit 25B, for example, in association with a frame image.

In the above description, the control unit 27 sets the reference timing based on the situation inside or outside the moving body during the period from the time point ( _t1 ) going back the first time from the time point when the image capture instruction was received to the time point ( _t0 ) when the image capture instruction was received, but the present invention is not limited to this. For example, the reference timing may be set within a predetermined period before or after the time point ( _t1 ) going back the first time.

The configurations and routines in the above-described embodiments are merely examples and can be selected, combined, or modified as appropriate depending on the application, etc.

REFERENCE SIGNS LIST 10 Imaging device 11 Exterior camera 13 Interior camera 15 Microphone 17 Touch panel display 19 System bus 21 Input unit 23 Distance measurement sensor 25 Large-capacity storage device 25A Exterior image storage unit 25B Image temporary storage unit 25C Distance information storage unit 25D Interior information storage unit 27 Control unit 29 Output unit 31 Transmitting/receiving unit

Claims (13)

a storage unit that stores a series of images captured recently by an imaging unit that moves together with the moving object and captures an image outside the moving object;
a reception unit that receives an image capture instruction by a user operation;
a setting unit that sets a reference timing within a period based on a situation inside the moving object or a situation outside the moving object within the period from a point in time when the reception unit received an image capturing instruction to a point in time going back a first time;
an extracting unit that extracts, as captured images, a predetermined number of images captured by the imaging unit at one or more timings based on the reference timing from the storage unit;
An imaging device comprising: The imaging unit captures an image of the inside of the moving body,
The imaging device according to claim 1 , wherein the setting unit sets, as the reference timing, a timing at which a predetermined motion of an occupant of the moving body or a change in a facial expression of the occupant is detected that is included in an image of the inside of the moving body. The imaging device according to claim 1, wherein the setting unit sets the reference timing to a timing at which a predetermined sound is detected among sounds collected by a sound collection unit that moves together with the moving body. The imaging device according to claim 1, wherein the setting unit detects the apparent size of an obstacle included in an image outside the moving body or the distance between the obstacle and the moving body, and sets the timing when the apparent size is smallest or the distance is longest within the period as the reference timing. a measurement unit that measures a distance between the moving body and an obstacle present in an imaging direction of an image outside the moving body captured by the imaging unit,
The imaging device according to claim 1 , wherein the setting unit sets, as the reference time, a time when a distance between the obstacle and the moving object within the period measured by the measurement unit is the longest. the reception unit receives the image capture instruction based on a speech and a touch operation of the user;
6. The imaging device according to claim 1, wherein the extraction unit sets the first time longer when the reception unit receives the imaging instruction based on the utterance than when the reception unit receives the imaging instruction based on the contact operation. The imaging device according to any one of claims 1 to 6, wherein the extraction unit extracts, from the storage unit, as captured images, a predetermined number of images captured by the imaging unit at a timing that is a second time prior to the time when the imaging instruction was received, when the situation inside the moving body or the situation outside the moving body does not satisfy the condition for the setting unit to set the reference timing within a period from the time when the reception unit received the imaging instruction to the time that is the first time prior. the reception unit receives the image capture instruction based on a speech and a touch operation of the user;
8. The imaging device of claim 7, wherein when the situation inside the moving body or the situation outside the moving body does not satisfy the condition for the setting unit to set the reference timing within a period from the time when the reception unit receives the imaging instruction based on the utterance, the extraction unit sets the second time longer than when the reception unit receives the imaging instruction based on the contact operation. The imaging device according to any one of claims 1 to 8, wherein the extraction unit extracts, as the captured image, one image captured by the imaging unit at one timing based on the reference timing from the storage unit. The imaging device according to any one of claims 1 to 8, further comprising a transmission unit that transmits the image extracted by the extraction unit to an external device. An imaging method performed by an imaging device, comprising:
a storage step of storing in a storage unit consecutive images captured recently by an imaging unit that moves together with the moving object and captures an image outside the moving object;
a receiving step of receiving an image capturing instruction by a user operation;
a setting step of setting a reference timing within a period based on a situation inside the moving body or a situation outside the moving body within the period from a point in time when the imaging instruction was accepted in the accepting step to a point going back a first time;
an extraction step of extracting, from the storage unit, a predetermined number of images captured by the imaging unit at each of one or a plurality of timings based on the reference timing set in the setting step, as captured images;
An imaging method comprising: An imaging program executed by an imaging apparatus including a computer, the computer comprising:
a storage step of storing in a storage unit consecutive images captured recently by an imaging unit that moves together with the moving object and captures an image outside the moving object;
a receiving step of receiving an image capturing instruction by a user operation;
a setting step of setting a reference timing within a period based on a situation inside the moving body or a situation outside the moving body within the period from a point in time when the imaging instruction was accepted in the accepting step to a point going back a first time;
an extraction step of extracting, from the storage unit, a predetermined number of images captured by the imaging unit at each of one or a plurality of timings based on the reference timing set in the setting step, as captured images;
An imaging program that executes the above. A computer-readable recording medium storing the imaging program according to claim 12.

Images