JP6980493B2 - Information processing equipment, information processing methods and programs - Google Patents

Description

The present invention relates to an information processing apparatus, an information processing method and a program.

In recent years, digitized cameras have appeared, and still images can be converted to JPEG and moving images can be converted to MPEG2 or H.D. on recording media such as flash memory cards (SD cards, CF cards, etc.). Products that encode and record / reproduce by the 264 method or the like have appeared.
Further, a method of correcting the shaking of the image has been proposed so that the shaking during shooting due to movement such as walking or camera shake during zooming makes it difficult to see the image.
For example, Patent Document 1 discloses the following techniques. That is, in an image pickup system consisting of a lens and an image pickup sensor, the light input from the lens is converted into a video signal, the video signal is displayed on the monitor device, and the video signal is also input to the image shake detection circuit. By correlating the image, the direction and size of the image shake are detected. Then, it is a technique for correcting the shake by driving the lens system in the direction of canceling the shake of the image.
Similarly, when shooting by tracking a subject, the subject in the shot image may be shaken, making it difficult to see the image.
For example, Patent Document 2 is a technique for correcting the fluctuation in such a case. Patent Document 2 discloses the following techniques. That is, when the subject is tracked and photographed, the direction and magnitude of the image shake are detected as a motion vector when the image shake is detected. Then, it is a technique of adding motion vectors of an image for a certain period of time, determining a region for correcting the shake using the added motion vector, and performing the shake correction so as to cancel the shake in the region.

Japanese Unexamined Patent Publication No. 61-248681 Japanese Patent No. 3199834

On the other hand, in recent years, the miniaturization of image pickup devices such as cameras has progressed, and cameras have been attached to human bodies, automobiles, motorcycles, bicycles and other vehicles for taking pictures.
When such an image is taken, the person who has the image pickup device or the inside of the automobile in which the image pickup device is installed may be reflected in the image taken by the image pickup device.
For example, when an image pickup device is installed in a car, the captured image includes a part of the inside of the car and the scenery through the glass of the car. The camera shakes due to the vibration of the car in which the device is installed. Therefore, different fluctuations occur in these two regions in the captured image. In this way, different fluctuations may occur in a plurality of regions in the image captured by the image pickup apparatus.
However, in the prior art, when different fluctuations occur in a plurality of regions in an image, the image cannot be properly corrected.

The information processing apparatus of the present invention includes a dividing means for dividing the video into a plurality of regions based on motion information which is information indicating the motion in the video detected from the video captured by the image pickup device. wherein the dividing means the region in which the image is included in the plurality of regions divided, on the basis of the motion information corresponding to the area, and chromatic correction means performs a process of correcting the shaking, the said The dividing means captures the image and the object in which the image pickup device is installed, which is a plurality of regions, based on the shaking information acquired through the sensor of the image pickup device and the motion information. Divide into the existing area and other areas .

According to the present invention, it is possible to appropriately correct an image when different fluctuations occur in a plurality of regions in the image.

It is a figure which shows an example of the hardware composition of an information processing apparatus. It is a figure explaining an example of the shake correction processing. It is a figure which shows an example of the installation situation of the image pickup apparatus. It is a flowchart which shows an example of the shake correction processing. It is a figure explaining an example of the motion vector detection processing. It is a figure which shows an example of the detected motion vector and the like. It is a figure which shows an example of the waveform of the detected shaking. It is a figure which shows an example of the photographed image and the like. It is a figure explaining an example of the interpolation processing of a motion vector. It is a figure which shows an example of the photographed image and the like.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.

<Embodiment 1>
FIG. 1A is a diagram showing an example of the hardware configuration of the image pickup apparatus 100 of the present embodiment. The image pickup device 100 is an image pickup device that includes a shooting unit having a wide-angle lens and controls shooting through the image pickup unit. The image pickup device 100 is an example of an information processing device.
The image pickup apparatus 100 includes a control unit 101, a memory 102, a recording medium 103, a photographing unit 104, an input I / F 105, and a sensor unit 106. The elements are communicably connected to each other via the system bus 107.
The control unit 101 is an arithmetic unit such as a Central Processing Unit (CPU) or a microcomputer that controls the image pickup device 100. The memory 102 is a storage device including a storage device such as a Read Only Memory (ROM) that stores various programs, various setting information, and the like. Further, the memory 102 includes a storage device such as a Random Access Memory (RAM) that functions as a work area of the control unit 101 and a temporary storage place for data. In the following, the portion of the memory 102 that temporarily stores the video signal is referred to as a frame memory.

The recording medium 103 is a recording medium that stores various programs, various setting information, video data captured by the image pickup apparatus 100, and the like. The recording medium 103 is a flash memory such as an SD memory card, a hard disk drive (HDD), a solid state drive (SSD), or the like. Further, in the present embodiment, the control unit 101 communicates data and various commands to the recording medium 103 via a known interface (IF) such as ATA and SDCARD I / F. Further, the recording medium 103 may be configured to be attached to and ejected from the image pickup device 100 by a recording medium attachment / discharge mechanism, or may be configured to be built in the image pickup device 100. good.
Further, when writing / reading data (file) to / from the recording medium 103, the control unit 101 reproduces the file system data from the recording medium 103 and stores it in the memory 102. This file system data is data indicating a file name, a file size, a data recording address, etc. of the data recorded on the recording medium 103, and is management information for managing the file. Then, the control unit 101 controls writing and reading of the file according to the read file system data. The control unit 101 updates the file system data stored in the memory 102 in response to writing the file to the recording medium 103. Then, the control unit 101 records the updated file system data on the recording medium 103.

The photographing unit 104 includes a lens, an image sensor, and the like, and is an photographing unit that photographs the surrounding environment. The photographing unit 104 acquires a video signal by converting the light entering from the lens via the image sensor. In the present embodiment, the photographing unit 104 includes a microphone, and also acquires an ambient audio signal via the microphone. The input I / F 105 is an interface used for connecting to an input unit such as a hard switch, a hard button, and an operation unit of a touch panel for instructing power on / off and video recording / playback. In the present embodiment, the input unit is built in the image pickup apparatus 100, but may be an external apparatus. By operating the input unit connected via the input I / F 105, the user can start and stop recording and playback of still images and moving images of the image pickup device 100, display a list of recorded images, and the like. You can instruct 100. The control unit 101 receives an input based on an operation of the input unit by the user via the input I / F 105. The sensor unit 106 is a sensor unit including sensors such as a gyro sensor and an acceleration sensor. The control unit 101 detects the vibration of the image pickup apparatus 100 based on the information detected by the sensor unit 106.
The control unit 101 executes processing according to a program stored in the memory 102, the recording medium 103, etc., so that the function of the image pickup apparatus 100 described later in FIG. 1B, the processing of the flowchart described later in FIG. 4 and the like can be performed. It will be realized.

FIG. 1B is a diagram showing an example of the functional configuration of the image pickup apparatus 100.
The image pickup apparatus 100 includes a coding unit 110, a recording medium control unit 111, a shake detection unit 112, a region determination unit 113, and a shake correction unit 114.
The coding unit 110 encodes / decodes video and audio. When recording a video, the coding unit 110 may use the video signal or audio signal acquired via the photographing unit 104 in the case of a moving image. 264, if it is a still image, it is encoded according to a coding format such as JPEG to reduce the amount of data.
The recording medium control unit 111 writes and reads various information such as video signals and audio signals to the recording medium 103. When recording video, the recording medium control unit 111 records video data and audio data stored in the frame memory of the memory 102 on the recording medium 103. Further, the recording medium control unit 111 reads video data and audio data from the recording medium 103 and stores them in the memory 102 when the video is reproduced. Further, the recording medium control unit 111 manages video data, audio data, and various information to be recorded on the recording medium 103 as files according to a file system such as FAT (File Allocation Table).

The shake detection unit 112 detects the shake of the image pickup apparatus 100 via the sensor unit 106. The shake detection unit 112 can detect the shake of the image pickup apparatus 100 according to the type of the sensor included in the sensor unit 106, and can detect not only the direction of the shake but also the magnitude of the shake.
The area determination unit 113 detects a motion vector from a plurality of images temporarily stored in the frame memory of the memory 102, and based on the detected motion vector and the vibration information detected by the vibration detection unit 112, Divide the area in the video. The motion vector detected from the video is an example of motion information indicating the motion in the video.
The shake correction unit 114 corrects the shake of the image based on the motion vector obtained by the area determination unit 113 and the shake information detected by the shake detection unit 112. The recording medium control unit 111 stores the video signal corrected by the shake correction unit 114 in the memory 102 again.
The shake detection unit 112, the area determination unit 113, and the shake correction unit 114 are always in operation when the image pickup apparatus 100 is recording an image, and also include an image acquired by the photographing unit 104 and stored in the memory 102. Works with. The coding unit 110 encodes an image that has been shake-corrected by the shake correction unit 114 and stored in the memory 102. Then, the recording medium control unit 111 records the encoded video on the recording medium 103.

FIG. 2 is a diagram illustrating an example of the shake correction process.
The images 200 to 202 of FIG. 2A show images taken through the photographing unit 104. The images 200 to 202 are images in which the flowerpots are continuously photographed in chronological order at predetermined intervals. The control unit 101 can shoot an image having a size larger than the image actually displayed or recorded on the display unit via the photographing unit 104. In the following, an image having a size larger than the image actually displayed or recorded on the display unit will be referred to as an unprocessed image. Further, in the following, an image that is actually displayed or recorded on the display unit will be referred to as a processed image.
The outer frame (solid line) in the images 200 to 202 indicates the range of the raw image taken through the photographing unit 104. Images 200 to 202 show that the flowerpot is shaking up and down in the image. If the images 200 to 202 are continuously played as a moving image, the flowerpot will shake up and down, and the image will be very difficult to see.

The images 210 to 212 in FIG. 2B are processed images partially cut out from the images 200 to 202, respectively. The dotted frame of the images 200 to 202 indicates the cutting position.
Looking at the position of the dotted frame line in the images 200 to 202, it can be seen that the cutting position is changed so as to cancel the shaking by the shaking of the flowerpot.
The control unit 101 can generate an image without shaking as shown in the images 210 to 212 by cutting out the image from the raw image so as to cancel the shaking in this way.
The above is an outline of an example of the shake correction process.

FIG. 3 is a diagram showing an example of the installation status of the image pickup apparatus 100.
FIG. 3A shows a situation in which the image pickup device 100 is installed in the center of the dashboard of an automobile. In the example of FIG. 3A, the scenery outside the vehicle spreads through the glass on the other side of the dashboard. In the present embodiment, the image pickup device 100 is installed in an automobile. In the following, the vehicle in which the image pickup device 100 is installed will be referred to as an installed vehicle.
FIG. 3B is an example of an image taken by the image pickup apparatus 100 in the situation of FIG. 3A. The image pickup apparatus 100 includes a wide-angle lens in the photographing unit 104, and captures an image on which a dashboard on which the image pickup apparatus 100 is installed is reflected.

Hereinafter, the processing of the image pickup apparatus 100 of the present embodiment will be described with reference to FIG.
FIG. 4 is a flowchart showing an example of the shake correction process.
When the power of the image pickup apparatus 100 is turned on, the process of FIG. 4 is started.
In S401, the control unit 101 shoots an image via the photographing unit 104 and the coding unit 110, acquires the image data of the unprocessed image which is the data of the photographed image, and stores the acquired image data in the frame memory. Remember. Further, the control unit 101 generates an image of the size of the processed image (for example, an image in which the pixel value of all pixels is 0). The control unit 101 generates a processed image by changing the pixel value in the generated image. In the following, this generated video will be referred to as the original video. Further, the control unit 101 periodically shoots video data at a predetermined interval Δt, and continues to store the video data of the shot raw video in the frame memory. It is assumed that the video data stored in the frame memory in S401 is the video data (t) which is the video data at the time t.

In S402, the control unit 101 detects a motion vector based on the video data stored in the memory 102 in S401. FIG. 5A is a diagram illustrating an example of motion vector detection processing by the control unit 101 in S402. Further, the video (video data at the time (tnΔt)) n frames before the video data (t) is referred to as video data (tnΔt). As shown in FIG. 5A, the control unit 101 has video data (t) and video data at the time (t−Δt) recorded in the frame memory (video data one frame before the video data at time t). ) And, to detect the motion vector. The control unit 101 uses a known technique such as a method of dividing an image into a plurality of predetermined blocks, matching each divided block, and obtaining the direction and size of each block. , Detect motion vector. The control unit 101 detects a motion vector by matching each block in, for example, the video data (t) and the video data (t−Δt).
FIG. 6A shows an example of the motion vector detected in S402. Each arrow in FIG. 6A is a motion vector. The direction and size of the arrows indicate the direction and size of the shake, respectively.

In S403, the control unit 101 detects the vibration of the image pickup device 100 with respect to the installed vehicle via the sensor unit 106 and the vibration detection unit 112. In the present embodiment, the control unit 101 detects the shaking in the direction perpendicular to the bottom surface of the vehicle in which the image pickup device 100 is installed. The control unit 101 stores the detected shaking information in the memory 102 as shaking information.
FIG. 7A shows a graph showing the shaking detected in S403. The vertical axis of the graph of FIG. 7A shows the magnitude of the shaking, and the horizontal axis shows the time.
In S404, the control unit 101 determines the region in the image via the region determination unit 113 based on the motion vector detected in S402 and the fluctuation information indicating the fluctuation of the image pickup device with respect to the installed vehicle detected in S403. Divide into multiple areas.
It is assumed that the motion vector detected in S402 is the motion vector shown in FIG. 6A. It is assumed that the shaking information indicating the shaking detected in S404 is the shaking information shown by the graph of FIG. 7 (a). In the present embodiment, the control unit 101 determines that the S403 detects only the shaking in the direction perpendicular to the bottom surface of the installed vehicle, but detects the shaking in other directions such as the horizontal direction with respect to the bottom surface of the installed vehicle. You may do it.

In S402, the video data at the time t−Δt and the time t is used for the detection of the motion vector as shown in FIG. 5 (a). Therefore, among the fluctuations detected in S403, the control unit 101 determines the fluctuations at the time t−Δt and the time t, the motion vector obtained from the video data (t) and the video data (t−Δt), and the motion vector. Will be compared. The control unit 101 detects the direction and magnitude of the sway in the direction perpendicular to the bottom surface of the installed vehicle from the change in the sway within the period from time t−Δt to t in the graph of FIG. 7A.
Further, the image pickup device 100 is installed in the installed vehicle. Therefore, it can be assumed that the size and direction of the motion vector in the portion of the installed vehicle on the image captured by the image pickup device 100 correspond to the size and direction of the shaking of the image pickup device 100 with respect to the installed vehicle. That is, when the image pickup device 100 swings in a certain direction, the area of the installed vehicle in the image captured by the image pickup device 100 swings in the opposite direction. Further, when the image pickup device 100 is shaken by a certain distance, the area of the installed vehicle in the image captured by the image pickup device 100 is shaken by a distance corresponding to the distance.
Therefore, in the present embodiment, the control unit 101 determines the portion of the image in which the shaking corresponding to the image pickup device 100 is generated as the portion where the installed vehicle is photographed, and the other portion is the scenery outside the vehicle. Is determined as the part being photographed.

For example, the control unit 101 divides the image captured by the image pickup apparatus 100 into a plurality of predetermined blocks, and the motion vector in the divided blocks is the fluctuation of the image pickup apparatus 100. Determine if it corresponds.
The shape and size of each of the plurality of blocks may be any shape and size, but it is desirable that the size is sufficient to divide the area. As the plurality of blocks, for example, there are blocks in which the entire area of the video is divided into rectangular areas of n × n (n: an arbitrary natural number).
For example, the control unit 101 performs the following processing for each block via the area determination unit 113. That is, the control unit 101 determines whether or not the direction of the motion vector in the block and the direction of the shaking of the image pickup apparatus 100 are opposite, and if it is determined that they are not opposite, the motion vector in the block and the motion vector 101 are determined. It is determined that the shaking of the image pickup apparatus 100 does not correspond.

Further, when the control unit 101 determines that the direction of the motion vector in the block and the direction of the shaking of the image pickup apparatus 100 are opposite to each other, the control unit 101 performs the following processing. That is, the control unit 101 has a difference between the magnitude of the motion vector in the block and the value obtained by multiplying the magnitude of the shaking of the image pickup device 100 with respect to the installed vehicle by a predetermined value, which is equal to or less than a predetermined threshold value. Performs the process of determining whether or not. This predetermined value is a value indicating the ratio of the fluctuation distance in the image captured by the image pickup device 100 to the fluctuation of the unit distance of the image pickup device 100 with respect to the actual installed vehicle. When the control unit 101 determines that this difference is larger than a predetermined threshold value, the control unit 101 determines that the motion vector in the block and the fluctuation of the image pickup apparatus 100 with respect to the installed vehicle do not correspond to each other. Further, when the control unit 101 determines that this difference is equal to or less than a predetermined threshold value, the control unit 101 determines that the motion vector in the block corresponds to the fluctuation of the image pickup apparatus 100 with respect to the installed vehicle.
Then, the control unit 101 determines the region indicated by each of the blocks determined to correspond to the motion vector in the block and the fluctuation of the image pickup device 100 with respect to the installed vehicle as the region B in which the installed vehicle is photographed. .. Further, the control unit 101 sets the area indicated by each of the blocks determined not to correspond to the motion vector in the block and the shaking of the image pickup device 100 with respect to the installed vehicle as the area A in which the scenery outside the vehicle is photographed. decide.
By the process of S404, the control unit 101 divides the region in the image captured by the image pickup apparatus 100 into the region A and the region B. FIG. 6B shows an example of how the image is divided into a region A and a region B.

In S405, the control unit 101 performs the vibration correction processing for the region A via the vibration correction unit 114. In the present embodiment, it is assumed that a portion (a dotted rectangular area in the image 200 of FIG. 2A) for making a processed image is defined in advance in the area in the image data which is an unprocessed image. For example, the control unit 101 performs the following processing to correct the fluctuation of the region A. That is, the control unit 101 specifies an area included in a predetermined area as a processed image in the area A in the image data (t) which is the unprocessed image stored in the frame memory in S401. The control unit 101 moves the specified region in the video data (t) by the distance indicated by the motion vector in the direction indicated by the motion vector detected in the region A in S402 in the video data (t). , It is specified as a region A in the processed image in which the shaking is corrected. Then, the control unit 101 sets the pixel value of each pixel to the same value as the image in the specified area for the area corresponding to the area A in the original image generated in S401.
In S406, the control unit 101 performs the vibration correction processing for the region B via the vibration correction unit 114. For example, the control unit 101 performs the following processing to correct the fluctuation of the region B. That is, the control unit 101 specifies an area included in a predetermined area as a processed image in the area B in the image data (t) which is the unprocessed image stored in the frame memory in S401. The control unit 101 moves the specified region in the video data (t) in the video data (t) by the distance indicated by the motion vector in the direction indicated by the motion vector detected for the region B in S402. It is specified as a region B in the processed image in which the shaking is corrected. Then, the control unit 101 sets the angular pixel value to the same value as the image in the specified region for the region corresponding to the region B in the original video generated in S401.

The original image generated in S401 by the processing of S405 to S406 is a processed image in which the shaking is corrected. The control unit 101 stores the processed image corrected for this shaking in the memory 102 and the recording medium 103.
In the present embodiment, the control unit 101 executes the process of S406 after the process of S405. However, the control unit 101 may execute the process of S405 after the process of S406, or may execute the process of S405 and the process of S406 in parallel.
Further, the control unit 101 may perform a process of correcting the shaking only on the designated one of the area A and the area B. For example, the control unit 101 sets the amount of the shake correction in the area B to 0 (equivalent to not performing the shake correction), so that the portion (dashboard portion) of the installed vehicle photographed by the image pickup apparatus 100 is shaken. It is possible to generate an image in which the scenery outside the car is not shaken. As a result, the image pickup apparatus 100 can produce an effect of being able to generate a realistic image.
In S407, the control unit 101 determines whether or not the power of the image pickup apparatus 100 has been turned off. When the control unit 101 determines that the power of the image pickup apparatus 100 has been turned off, the process of FIG. 4 is terminated, and when it is determined that the power supply of the image pickup apparatus 100 has not been turned off, the process proceeds to the process of S401.

As described above, in the present embodiment, the image pickup device 100 detects the motion vector from the image captured by the image pickup device 100, and detects the shaking information for the vehicle on which the image pickup device 100 is installed via the sensor unit 106. The image pickup apparatus 100 determines whether the region of each block in the image captured by the image pickup apparatus 100 is a region A or B based on the detected motion vector and the fluctuation information. Then, the image pickup apparatus 100 performs correction processing corresponding to each of the regions A and B. As a result, the image pickup apparatus 100 can appropriately correct the image when different fluctuations occur in a plurality of regions in the image.

<Embodiment 2>
In the first embodiment, the image pickup device 100 detects the shaking of the image pickup device 100 with respect to the installed vehicle. In the present embodiment, the image pickup device 100 detects the shake of the image pickup device 100 with respect to the installed vehicle and the shake of the installed vehicle via the sensor unit 106 and the shake detection unit 112.
The hardware configuration and functional configuration of the image pickup apparatus 100 of the present embodiment are the same as those of the first embodiment. Further, the processing of the image pickup apparatus 100 is the same as the processing shown in FIG. 4, but the processing of S403 to S404 is different from that of the first embodiment.

In the present embodiment, the image pickup apparatus 100 detects, in S403, the shaking of the image pickup device 100 with respect to the installed vehicle and the shaking of the installed vehicle itself, which is a vibration having a lower frequency than this shaking. The shaking of the installed vehicle itself can be seen as the shaking of the landscape portion outside the vehicle through the glass in the image taken by the image pickup device 100.
In the present embodiment, the processing when the image pickup apparatus 100 detects a plurality of shakings will be described as being different from the first embodiment.
The control unit 101 detects the vibration of the image pickup device 100 with respect to the installed vehicle and the vibration of the installed vehicle itself in S403 via the sensor unit 106 and the vibration detection unit 112. In the present embodiment, the control unit 101 detects the shaking of the image pickup device 100 in the direction perpendicular to the bottom surface of the installed vehicle and the shaking of the installed vehicle in the direction perpendicular to the bottom surface of the installed vehicle. The control unit 101 stores the detected information on the shaking of the image pickup device 100 and the installed vehicle in the memory 102 as the shaking information.
FIG. 7B shows a graph showing an example of the shaking detected in S403 of the present embodiment. The vertical axis of the graph of FIG. 7B shows the magnitude of the shaking, and the horizontal axis shows the time. FIG. 7B shows a waveform in which relatively low frequency fluctuations and relatively high frequency fluctuations are superimposed. Here, it is assumed that the relatively low frequency sway is the sway of the installed vehicle itself, and the relatively high frequency sway is the sway of the image pickup device 100 with respect to the installed vehicle.

The control unit 101, for example, converts the detected waveform data of the fluctuation into data in the frequency domain by performing a Fourier transform. Then, the control unit 101 acquires the fluctuation data indicating the fluctuation of the image pickup apparatus 100 with respect to the installed vehicle by performing inverse Fourier transformation of the data having a frequency equal to or higher than a predetermined threshold value among the converted data. Further, the control unit 101 acquires the fluctuation data indicating the fluctuation of the installed vehicle itself by performing the inverse Fourier transformation of the data having a frequency less than a predetermined threshold value among the converted data. As a result, the control unit 101 can separate the shaking data of FIG. 7B into the shaking data of the image pickup device 100 with respect to the installed vehicle and the shaking data of the installed vehicle itself.
FIG. 7C shows a graph showing an example of the shaking of the image pickup apparatus 100 with respect to the separated installed vehicle. The shaking of FIG. 7 (c) is the same as that of FIG. 7 (a). Further, FIG. 7 (d) shows a graph showing an example of the shaking of the separated installed vehicle itself. The sway in FIG. 7 (d) can be assumed to correspond to the sway in region A in FIG.

Further, in the present embodiment, the control unit 101 passes through the region determination unit 113 based on the motion vector detected in S402 in S404 and the vibration information indicating the vibration detected in S403, and the region in the image. Is divided into multiple areas.
More specifically, the control unit 101 divides, for example, an image captured by the image pickup apparatus 100 into a plurality of predetermined blocks. Then, the control unit 101 determines, for each divided block, whether or not the motion vector in the block corresponds to the shaking of the image pickup apparatus 100 with respect to the installed vehicle or the shaking of the installed vehicle itself.
For example, the control unit 101 performs the following processing for each block via the area determination unit 113. That is, the control unit 101 determines whether or not the direction of the motion vector in the block and the direction of the shaking of the image pickup device 100 with respect to the installed vehicle are opposite to each other. Then, when it is determined that the control unit 101 is not the opposite, it is determined that the motion vector in the block and the fluctuation of the image pickup apparatus 100 with respect to the installed vehicle do not correspond to each other.
Further, when the control unit 101 determines that the direction of the motion vector in the block and the direction of the shaking of the image pickup apparatus 100 with respect to the installed vehicle are opposite, the control unit 101 performs the following processing. That is, the control unit 101 has a difference between the magnitude of the motion vector in the block and the value obtained by multiplying the magnitude of the shaking of the image pickup device 100 with respect to the installed vehicle by a predetermined value, which is equal to or less than a predetermined threshold value. Performs the process of determining whether or not. When the control unit 101 determines that this difference is larger than a predetermined threshold value, the control unit 101 determines that the motion vector in the block and the fluctuation of the image pickup apparatus 100 with respect to the installed vehicle do not correspond to each other. Further, when the control unit 101 determines that this difference is equal to or less than a predetermined threshold value, the control unit 101 determines that the motion vector in the block corresponds to the fluctuation of the image pickup apparatus 100 with respect to the installed vehicle.

Further, the control unit 101 performs the following processing for each block, for example, via the area determination unit 113. That is, the control unit 101 determines whether or not the direction of the motion vector in the block and the direction of the shaking of the installed vehicle itself are opposite, and if it is determined that they are not opposite, the motion vector in the block and the motion vector 101 are determined. It is determined that the shaking of the installed vehicle itself does not correspond.
Further, when the control unit 101 determines that the direction of the motion vector in the block and the direction of the shaking of the installed vehicle itself are opposite to each other, the control unit 101 performs the following processing. That is, whether or not the difference between the magnitude of the motion vector in the block and the value obtained by multiplying the magnitude of the shaking of the installed vehicle itself by a predetermined value is equal to or less than the predetermined threshold value. Is performed. When the control unit 101 determines that this difference is larger than a predetermined threshold value, the control unit 101 determines that the motion vector in the block and the fluctuation of the installed vehicle itself do not correspond to each other. Further, when the control unit 101 determines that this difference is equal to or less than a predetermined threshold value, the control unit 101 determines that the motion vector in the block corresponds to the fluctuation of the installed vehicle itself.
Then, the control unit 101 determines the region indicated by each of the blocks determined to correspond to the motion vector in the block and the fluctuation of the image pickup device 100 with respect to the installed vehicle as the region B in which the installed vehicle is photographed. .. Further, the control unit 101 determines a region indicated by each of the blocks determined to correspond to the motion vector in the block and the shaking of the installed vehicle itself as the region A in which the scenery outside the vehicle is photographed.

As described above, by the process of this embodiment, the image pickup apparatus 100 can more appropriately divide the region in the image captured by the image pickup apparatus 100. As a result, the image pickup apparatus 100 can more appropriately correct the shaking.

<Embodiment 3>
In the present embodiment, the image captured image is based on a plurality of motion vectors detected by comparison of a plurality of images captured by the image pickup apparatus 100 at predetermined intervals (Δt) in time series. A process of dividing an area and correcting the fluctuation for each divided area will be described.
The hardware configuration and functional configuration of the image pickup apparatus 100 of the present embodiment are the same as those of the first embodiment. The processing of the image pickup apparatus 100 is the same as the processing shown in FIG. 4, but the processing of S402 and S404 is different from that of the first embodiment.

FIG. 8A shows a situation in which a person in the installed vehicle moves and his / her hand or the like is reflected in the image. For example, the motion vector detected when the hand moves upward in FIG. 8 (a) is shown in FIG. 8 (b). The arrow 801 in FIG. 8B is an upward motion vector detected by the movement of the hand of a person in the installed vehicle. It can be seen that the motion vector of the arrow 801 has a different direction from the other motion vectors in the area where the installed vehicle is photographed in the image. At this time, when the area determination process of S404 is performed as in the first embodiment, the area corresponding to the motion vector of the arrow 801 is not regarded as the area B because the shaking information and the motion vector are different, and is determined as the area A. Will be done.
If the vibration correction process is performed in this state, the region corresponding to the motion vector of the arrow 801 will be corrected as the region A, so that the image will not be corrected correctly and the image will have a sense of discomfort. Further, the area corresponding to the motion vector of the arrow 801 is the area of the person in the installed vehicle, and is determined to be the area B as the area equivalent to the area in which a part of the vehicle such as the dashboard is reflected. It is desirable to correct the shaking.
The period in which the motion vector of the arrow 801 does not match the motion vector in the other region B is a temporary period in which the hand is moving. On the other hand, when the hand is not moving, the same shaking as in the other area B occurs as in the dashboard. Therefore, the motion vectors corresponding to the region B at that time are all the same motion vector as shown in FIG. 6A.
Therefore, in the present embodiment, when there is a region where the motion vector temporarily fluctuates, the image pickup apparatus 100 performs a process of determining the region as a region corresponding to the motion vector before the fluctuation.

FIG. 5B is a diagram illustrating an example of motion vector detection processing by the control unit 101 in S402 in the present embodiment.
It is assumed that the video data (t) is acquired in the process of S401. Further, at the time of processing S402, the video data (t), the video data (t—Δt), ..., ~, The video data (t— (n-1) Δt), and the video data (t) are stored in the frame memory. It is assumed that n + 1 video data of −nΔt) are stored. It is assumed that n is a predetermined integer of 2 or more. The control unit 101 is S402, and the video data (t−kΔt), the video data (t− (k + 1) Δt), and ( From 0 <= k <= n-1 k ∈ Z), the motion vector in the video data (t−kΔt) is detected. In the following, the motion vector detected from the video data (t−kΔt) and the video data (t− (k + 1) Δt) will be referred to as a motion vector (t−kΔt). The control unit 101 performs this process for each of the cases where the value of k is {0, 1, 2, ..., N-2, n-1}, so that n motion vectors (motions) are performed. The vector (t) to the motion vector (t- (n-1) Δt) is detected. As a result, the control unit 101 has acquired not only the motion vector at time t but also the past n-1 motion vectors. That is, the control unit 101 has acquired n motion vectors before the time t.

The control unit 101 determines the area of each of the plurality of blocks preset in the video in S404 based on the plurality of motion vectors acquired in S402.
For example, when the motion vector corresponding to a certain block at a certain time point is oriented in the same direction for a predetermined period and causes a fluctuation of a magnitude equal to or less than a predetermined threshold value, the control unit 101 is described as follows. Process. That is, the control unit 101 determines the motion vector as the final motion vector corresponding to the block that can be placed at that time point. Further, when the motion vector is oriented in the same direction for a predetermined period and causes only a fluctuation of a magnitude equal to or less than a predetermined threshold value, the control unit 101 sets the motion vector. Not determined as the final motion vector.
This predetermined period is defined as th1. In this embodiment, th1 is 3Δt. In this case, the control unit 101 acquires a motion vector (t), a motion vector (t−Δt), a motion vector (t-2Δt), and a motion vector (t-3Δt) corresponding to a certain block from the frame. It is assumed that the motion vector (t−Δt), the motion vector (t-2Δt), and the motion vector (t-3Δt) are motion vectors having the same direction and magnitude, respectively. Then, the control unit 101 determines whether or not the directions of the motion vector (t) and the motion vectors (t−Δt) to (t-3Δt) match.

When the control unit 101 determines that the directions of the motion vector (t) and the motion vectors (t−Δt) to (t-3Δt) do not match, the motion vector (t) is determined as the final motion vector. Without doing, perform the following processing. That is, the control unit 101 acquires, for example, the average motion vector of the motion vectors (t−Δt) to (t-3Δt), and determines the acquired motion vector as the final motion vector of the block at time t. do.
When the control unit 101 determines that the directions of the motion vector (t) and the motion vectors (t−Δt) to (t-3Δt) match, the control unit 101 performs the following processing. That is, whether the difference between the magnitude of the motion vector (t) and the average value of the magnitudes of the motion vectors (t−Δt) to (t-3Δt) of the control unit 101 is equal to or less than a predetermined threshold value. Judge whether or not. When the control unit 101 determines that this difference is equal to or less than a predetermined threshold value, the control unit 101 determines the motion vector (t) as the final motion vector of the block at time t. When the control unit 101 determines that this difference is larger than a predetermined threshold value, the control unit 101 performs the following processing without determining the motion vector (t) as the final motion vector. That is, the control unit 101 acquires, for example, the average motion vector of the motion vectors (t−Δt) to (t-3Δt), and determines the acquired motion vector as the final motion vector of the block at time t. do.

FIG. 9 is a diagram illustrating an example of the processing of S404 of the present embodiment. The upper part shows the time-series transition of the motion vector detected by S402 corresponding to a certain block. The upper part of FIG. 9 shows how the direction of the motion vector fluctuates from the direction of the motion vector at the time up to that time at time t. Arrow 901 in FIG. 9 indicates a motion vector at time t. For example, as described with reference to FIG. 8, it can be assumed that the movement of a person's hand or the like has occurred.
In the lower part of FIG. 9, for example, the average motion vector of the motion vectors (t−Δt) to (t-3Δt) is finally displayed instead of the motion vector indicated by the arrow 901 by the processing of S404 of the present embodiment. It is shown that it is determined as a motion vector.

Further, the control unit 101 may determine, for example, the average of the n motion vectors acquired in S402 for each of the plurality of blocks as the final motion vector in the block.
Further, for example, the control unit 101 may determine the intermediate value of the n motion vectors acquired in S402 as the final motion vector in the block for each of the plurality of blocks.
Further, the control unit 101 specifies, for example, the direction of each of the n motion vectors acquired in S402 for each of the plurality of blocks, and specifies the direction in which the most motion vectors are oriented. Then, the control unit 101 identifies a motion vector facing the specified direction from the n motion vectors acquired in S402, and sets the average motion vector of the specified motion vector in the final block. It may be determined as a motion vector.
Then, the control unit 101 determines the region of each block by the same processing as in the first embodiment based on the determined final motion vector.

As described above, by the processing of the present embodiment, the image pickup apparatus 100 can correct the shaking in the image more stably even if the shaking occurs temporarily in the image.

<Embodiment 4>
In this embodiment, the processing of the image pickup device 100 when the image pickup device 100 is installed on a motorcycle, a bicycle, or the like will be described.
The hardware configuration and functional configuration of the image pickup apparatus 100 of the present embodiment are the same as those of the first embodiment. The processing of the image pickup apparatus 100 is the same as the processing shown in FIG. 4, but the processing of S403 is different from that of the first to third embodiments.
One of the characteristics of motorcycles and bicycles is that the vehicle body tilts when turning a curve.
FIG. 10A is a diagram showing an example of a photographed image taken by the image pickup device 100 when the image pickup device 100 is installed on the side surface of a tire of a motorcycle, a bicycle, or the like. In the example of FIG. 10A, the background is photographed including a part of the tire 1000. In this case, when the same processing of S404 as in the first embodiment is performed, the tire portion is determined to be the region B, and the other portions are determined to be the region A. In the present embodiment, the control unit 101 detects, in S403, the shaking of the image pickup device 100 with respect to the motorcycle, the bicycle, etc. in the direction perpendicular to the bottom surface of the motorcycle, the bicycle, or the like. Further, the control unit 101 may detect the shaking of the motorcycle, the bicycle, etc. in the direction perpendicular to the bottom surface of the motorcycle, the bicycle, etc. by S403.

FIG. 10B is a diagram showing an example of an image taken by the image pickup apparatus 100 when the vehicle body is tilted when a motorcycle, a bicycle, or the like turns a curve. Since the image pickup device 100 is installed in the vehicle, the position of the tire 1000 in the image does not change. Further, since the image pickup device 100 is tilted together with the vehicle, the image of the background portion is a tilted image. In this case, the motion vector detected from the image captured by the image pickup apparatus 100 by the process of S402 is as shown in FIG. 10 (c). On the other hand, depending on the type of sensor included in the sensor unit 106, the fluctuation information acquired by S403 may include fluctuation information in the rotation direction. In that case, since the image pickup device 100 and the vehicle of the motorcycle or bicycle on which the image pickup device 100 is installed are fixed, the fluctuation information in the rotation direction is unnecessary information for determining the region B. Further, if the region B is determined using this unnecessary information, it may cause an erroneous determination.
Therefore, the control unit 101 detects predetermined types of shaking information such as directions other than the rotation direction and other directions in S403, and based on the detected shaking information, each of the images in the image captured by the image pickup apparatus 100. The area of the block will be determined.

As described above, in the present embodiment, the image pickup apparatus 100 detects only a predetermined type of shake such as a direction other than the rotation direction, and the detected shake information is the region in the image captured by the image pickup device 100. I decided to use it for division. As a result, the image pickup apparatus 100 can reduce the possibility of erroneously determining the region.

<Other embodiments>
In the first to fourth embodiments, the image pickup device 100 is installed in an automobile or a motorcycle. However, the image pickup device 100 may be installed on other objects such as trains, ships, drones, elevators, buildings, people, and animals.
In the first to fourth embodiments, the image pickup apparatus 100 is determined to execute a process of correcting the fluctuation in the image captured by the image pickup apparatus 100. However, an information processing device different from the image pickup device 100 that has acquired the image captured by the image pickup device 100 may execute a process of correcting the fluctuation in the image captured by the image pickup device 100. For example, an information processing device (for example, a personal computer, a computer mounted on a car, a server device, etc.) connected to the image pickup device 100 may execute a process of correcting a shake in an image captured by the image pickup device 100. good. In that case, the information processing device acquires the image captured by the image pickup device 100 from the image pickup device 100, and executes the process of FIG. 4 based on the acquired image. That is, the control unit of the information processing apparatus executes the processing according to the program stored in the recording medium of the information processing apparatus, whereby the function of FIG. 1B, the processing of FIG. 4 and the like are realized. It becomes.

In the first to fourth embodiments, the image pickup apparatus 100 divides the region in the image captured by the image pickup apparatus 100 into a plurality of regions based on the shake detected via the sensor unit 106 and the shake detection unit 112. However, the control unit 101 does not use, for example, the vibration detected via the sensor unit 106 and the vibration detection unit 112, and the region in the image captured by the image pickup apparatus 100 is divided into a plurality of regions as follows. It may be divided into. That is, for example, the control unit 101 uses the k-means method, the shortest distance method, and the longest method to obtain a plurality of motion vectors detected for each of a plurality of predetermined blocks in S402 based on the direction and magnitude of the vectors. Clustering is performed using methods such as the distance method and the Ward method. Then, the control unit 101 identifies a region indicated by a block corresponding to a motion vector belonging to the same cluster as a region in which the same shaking occurs, so that a plurality of regions in the image captured by the image pickup apparatus 100 can be obtained. It may be divided into areas of. In that case, the control unit 101 executes a process of correcting the fluctuation in S405 to 406 based on the motion vector corresponding to each region.
In this case, the control unit 101 may cluster the plurality of motion vectors into a plurality of clusters of three or more. As a result, the control unit 101 can divide the region in the image captured by the image pickup apparatus 100 into three or more regions. As a result, even when three or more different shakes occur in the image captured by the image pickup device 100, the image pickup device 100 can appropriately correct the shakes.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.
For example, a part or all of the functional configuration of the image pickup device 100 described above may be mounted on the image pickup device 100 as hardware.
Although the preferred embodiment of the present invention has been described in detail above, the present invention is not limited to the specific embodiment. Each of the above-described embodiments may be arbitrarily combined.

100 Imaging device 101 Control unit 104 Imaging unit

Claims (8)

A dividing means for dividing the image into a plurality of areas based on the motion information which is information indicating the motion in the image detected from the image taken through the image pickup apparatus.
A correction means that performs a process of correcting a shake based on the motion information corresponding to the region for a region included in the plurality of regions in which the image is divided by the division means.
Have a,
The dividing means captures the image by the object in which the image pickup device is installed, which is a plurality of regions, based on the shaking information acquired through the sensor of the image pickup device and the motion information. An information processing device that divides an area into other areas. The dividing means divides the video into a plurality of regions based on the motion vector which is the motion information.
The first aspect of the present invention, wherein the correction means corrects a shake of a region included in the plurality of regions in which the image is divided by the division means based on the motion vector corresponding to the region. Information processing equipment. The dividing means divides the video into a plurality of regions based on the direction and magnitude of the motion vector.
The correction means corrects a region included in the plurality of regions in which the image is divided by the division means, based on the direction and magnitude of the motion vector corresponding to the region. The information processing apparatus according to claim 2. The dividing means is based on the shaking information, which is information about a predetermined type of shaking, and the motion information, and the moving image is displayed by an object in which the image pickup device, which is a plurality of regions, is installed. The information processing apparatus according to any one of claims 1 to 3, wherein the area is divided into a photographed area and another area. The correction means is claimed to perform a process of correcting a shake of a designated area included in the plurality of areas in which the image is divided by the dividing means, based on the motion information corresponding to the area. Item 4. The information processing apparatus according to any one of Items 1 to 4. The dividing means divides the video into the plurality of regions based on a plurality of motion information acquired based on the video and a plurality of videos captured via the image pickup device in the past. The information processing apparatus according to any one of claims 1 to 5. It is an information processing method executed by an information processing device.
A division step of dividing the image into a plurality of areas based on the motion information, which is information indicating the motion in the image detected from the image taken through the image pickup device, and
A correction step of performing a process of correcting a shake based on the motion information corresponding to the region for a region included in the plurality of regions in which the video is divided in the division step.
Including
In the division step, the image is captured by the object in which the image pickup device is installed, which is a plurality of regions, based on the shaking information acquired through the sensor of the image pickup device and the motion information. An information processing method that is divided into an area that is used and other areas. A program for making a computer function as each means of the information processing apparatus according to any one of claims 1 to 6.

Images