JP7707612B2 - Imaging device and control program - Google Patents

Description

The present invention relates to an imaging device and a control program.

Conventionally, there are cameras that capture images by controlling their attitude with an electronic gimbal using detection signals from an acceleration sensor or a gyro sensor (see, for example, Patent Document 1 below). However, the above-mentioned conventional technology does not take into consideration the automatic switching between a mode in which images are captured using an electronic gimbal and a mode in which images are captured without using an electronic gimbal.

JP 2013-214947 A

An imaging device according to one aspect of the invention disclosed in the present application is an imaging device that can be mounted on a moving body, and includes an imaging unit that images a subject, an acquisition unit that acquires first information regarding a change in the attitude of the imaging device using a sensor , a determination unit that identifies an invisible area where it is not possible to see outside the moving body based on image data of the subject generated by output from the imaging unit, and a determination unit that executes one of a first determination process that determines a variation parameter that varies the position of an extraction range of image data to be output from the image data based on the first information acquired by the acquisition unit, and a second determination process different from the first determination process, based on the first information and second information indicating a change in the attitude of the imaging device obtained from a motion vector in the invisible area identified by the determination unit, and the second determination process, wherein the second determination process is a process of invalidating the detection signal of the sensor, or a process of determining a variation parameter that varies the position of the extraction range based on the image data .

A control program which is one aspect of the invention disclosed in the present application is a control program that causes a processor to execute control of an imaging device that can be mounted on a moving body and captures images of a subject, and causes the processor to acquire first information regarding changes in the attitude of the imaging device using a sensor , identify an invisible area where it is not possible to see outside the moving body based on image data of the subject captured by the imaging device, and execute one of a first determination process that determines a variation parameter that varies the position of an extraction range of image data to be output from the image data based on the first information, and a second determination process different from the first determination process, based on the first information and second information indicating a change in attitude of the imaging device obtained from a motion vector in the invisible area, and the second determination process is a process of invalidating the detection signal of the sensor, or a process of determining a variation parameter that varies the position of the extraction range based on the image data .

FIG. 1 is an explanatory diagram showing an image capturing example 1 using an image capturing device mounted in an automobile. FIG. 2 is an explanatory diagram showing an image capturing example 2 by an image capturing device mounted in an automobile. FIG. 3 is an explanatory diagram showing an image capturing example 3 using an image capturing device mounted in an automobile. FIG. 4 is a block diagram showing an example of the hardware configuration of the imaging apparatus. FIG. 5 is a block diagram showing an example of the functional configuration of the imaging apparatus. FIG. 6 is an explanatory diagram illustrating an example (first half) of generating spherical image data by the image processing unit. FIG. 7 is an explanatory diagram showing an example (second half) of generating spherical image data by the image processing unit. FIG. 8 is an explanatory diagram showing the visible area and the invisible area in the combined equirectangular converted image data. FIG. 9 is an explanatory diagram showing the motion vectors of the visible area in the combined equirectangular converted image data. FIG. 10 is an explanatory diagram showing the amount of movement related to the position of the extraction range. FIG. 11 is an explanatory diagram showing the movement angle obtained by the movement detection unit. FIG. 12 is an explanatory diagram showing an example of detection of the subject distance in the overlapping region. FIG. 13 is a flowchart illustrating an example of a manual mode switching process performed by the imaging apparatus. FIG. 14 is a flowchart showing an example of an imaging process procedure by the imaging device in the sensor use mode. FIG. 15 is a flowchart showing an example of a processing procedure for automatic mode switching in the imaging apparatus. FIG. 16 is a flowchart showing an example 1 of an imaging process procedure by the imaging device in the image detection mode. FIG. 17 is a flowchart showing an example 2 of an imaging process procedure by the imaging device in the image detection mode. FIG. 18 is a flowchart showing an example 3 of an imaging process procedure by the imaging device in the image detection mode. FIG. 19 is a flowchart illustrating an example of a playback process procedure for spherical image data. FIG. 20 is an explanatory diagram showing an example of imaging by an imaging device mounted on a motorcycle.

The imaging device according to this embodiment can be mounted on a moving body or on a passenger of the moving body, and has an omnidirectional camera (also called a 360-degree camera) that captures an image of a subject and captures omnidirectional image data. Such an imaging device is equipped with an electronic gimbal. This electronic gimbal detects the tilt of the imaging device and corrects the position of an extraction range for extracting image data to be output from the omnidirectional image data so as to cancel the tilt.

When an imaging device is mounted on a moving body or on a passenger of a moving body, the imaging device is accelerated by the acceleration/deceleration and turning of the moving body. As a result, the electronic gimbal corrects the captured image data as if the imaging device were tilted, even though in reality the imaging device is not tilted. In this embodiment, by controlling so that the electronic gimbal does not make such corrections, erroneous position changes of the extraction range for extracting output target image data due to the acceleration/deceleration and turning of the moving body are suppressed, based on image data outside the moving body where the subject is not the moving body. This makes it possible to provide image data suitable for the actual running of the moving body.

The present embodiment will be described in detail below. Examples of moving objects include vehicles (e.g., automobiles, motorcycles, bicycles, and trains), ships, flying objects (e.g., airplanes, helicopters, and drones), and robots. In the following embodiment, an automobile will be used as an example of a moving object. When the moving object is an automobile, the imaging device is removably fixed inside the vehicle or to the passenger, but in the following embodiment, a case will be described where the imaging device is fixed to any position inside the vehicle.

An omnidirectional camera is a camera in which the back sides of two imaging elements are placed facing each other, and a fisheye lens with an angle of view of 180 degrees or more is provided in front of each imaging element. The optical axes of both fisheye lenses are assumed to be aligned. The omnidirectional camera generates omnidirectional image data that can be rotated 360 degrees in any direction around the imaging device by synthesizing image data from both imaging elements. The omnidirectional camera is equipped with an electronic gimbal. The electronic gimbal detects the tilt of the omnidirectional camera using an acceleration sensor and a gyro sensor, and corrects the image data to cancel out the tilt.

The spherical camera may be configured with imaging elements and lenses in six locations: top, bottom, front, back, left and right. Also, instead of a spherical camera, it may be a camera equipped with one imaging element and one fisheye lens. Also, instead of a fisheye lens, it may be an ultra-wide-angle lens with an angle of view of, for example, 170° or more and less than 180°. Examples are described below.

<Example of imaging using an imaging device installed in a car>
1 to 3 are explanatory diagrams showing imaging examples 1 to 3 by an imaging device mounted inside an automobile. Fig. 1 shows imaging example 1 when the automobile 100 accelerates, Fig. 2 shows imaging example 2 when the automobile 100 decelerates, and Fig. 3 shows imaging example 3 when the automobile 100 turns right in the traveling direction. The imaging example when the automobile 100 turns left in the traveling direction is the turning direction opposite to imaging example 3, so a description thereof will be omitted. Note that the omnidirectional camera, which is the imaging device 101, is fixed inside the automobile 100 with the imaging element and fisheye lenses 102a and 102b facing respectively in the traveling direction and the reversing direction of the automobile 100.

In FIG. 1, (A) shows the type of acceleration acting on the imaging device 101 when the automobile 100 accelerates. The optical axis OA of the first fisheye lens 102a on the forward direction side and the second fisheye lens 102b on the backward direction side coincide. The imaging direction CD1 indicates the direction of the optical axis OA in which the imaging element on the forward direction side and the first fisheye lens 102a are facing. When the automobile 100 accelerates, the imaging device 101 is subjected to gravitational acceleration g in the vertical direction and acceleration a1 in the backward direction of the automobile 100. The resultant acceleration ga1 is the acceleration obtained by combining the gravitational acceleration g and acceleration a1.

(B) shows an example of tilt detection of the imaging device 101 when the automobile 100 accelerates. The electronic gimbal of the imaging device 101 determines that the traveling direction side of the imaging device 101 has tilted upward in the pitch direction (tilt angle α) due to the resultant acceleration ga1 (however, in reality, the imaging device 101 does not tilt because it is fixed). The electronic gimbal changes the imaging direction CD1 to the imaging direction CD2 so that the first fisheye lens 102a on the traveling direction side faces directly ahead in the traveling direction, i.e., to cancel out the tilt angle α. Note that since the imaging device 101 is fixed, the optical axis OA does not actually shift like the imaging direction CD2.

(C) shows an example of image data of a subject output by the imaging device 101 when acceleration a1 is not applied to the imaging device 101. Specifically, for example, (C) shows an extraction range 110 of the omnidirectional image data when the imaging direction CD1 is set as the projection direction of the stitched image data, and output target image data 111 extracted from the omnidirectional image data in the extraction range 110. The omnidirectional image data is image data in which the image data after stitching is pasted onto a virtual spherical surface. Of the output target image data 111, image data 111a is image data of the scenery outside the vehicle captured through the windshield, and image data 111b is image data of the interior of the vehicle.

(D) shows an example of image data of a subject output by the imaging device 101 when acceleration a1 is applied to the imaging device 101. Specifically, for example, (D) shows an extraction range 120 of the omnidirectional image data when the imaging direction CD2 is set as the projection direction of the stitched omnidirectional image data, and output target image data 121 extracted from the omnidirectional image data in the extraction range 120. Of the output target image data 121, image data 121a is image data of the scenery outside the vehicle captured through the windshield, and image data 121b is image data of the interior of the vehicle.

As shown in (D), each time acceleration a1 is applied to the imaging device 101, the extraction range 110 is changed to extraction range 120, resulting in a blurring of the output target image data 111 to output target image data 121. As a result, at least a portion of the area that is actually desired to be output (in this example, image data 121a of the scenery outside the vehicle) falls outside the extraction range 110 and is not output, and an unnecessary area (in this example, image data 121b of the interior of the vehicle) is included in the extraction range 110.

In FIG. 2, (A) shows the type of acceleration applied to the imaging device 101 when the automobile 100 decelerates. When the automobile 100 decelerates, the imaging device 101 is subjected to gravitational acceleration g in the vertical direction and acceleration (negative acceleration) a2 in the traveling direction of the automobile 100. The composite acceleration ga2 is the combination of the gravitational acceleration g and acceleration a2.

(B) shows an example of tilt detection of the imaging device 101 when the automobile 100 decelerates. The electronic gimbal of the imaging device 101 determines that the traveling direction side of the imaging device 101 has tilted downward in the pitch direction (tilt angle α) due to the resultant acceleration ga2 (however, in reality, the imaging device 101 does not tilt because it is fixed). The electronic gimbal changes the imaging direction CD1 to the imaging direction CD3 so that the first fisheye lens 102a on the traveling direction side faces directly ahead in the traveling direction, i.e., to cancel out the tilt angle α. Note that since the imaging device 101 is fixed, the optical axis OA does not actually shift like the imaging direction CD2.

(C) shows an example of capturing an image of a subject by the imaging device 101 when acceleration a2 is not applied to the imaging device 101. Specifically, for example, like (C) in FIG. 1, (C) shows an extraction range 110 of the omnidirectional image data when the imaging direction CD1 is set as the projection direction of the stitched omnidirectional image data, and output target image data 111 extracted from the omnidirectional image data in the extraction range 110.

(D) shows an example of capturing an image of a subject by the imaging device 101 when acceleration a2 is applied to the imaging device 101. Specifically, for example, (D) shows an extraction range 130 of the omnidirectional image data when the imaging direction CD3 is set as the projection direction of the stitched omnidirectional image data, and output target image data 131 extracted from the omnidirectional image data in the extraction range 130. The output target image data 131 is image data of the scenery outside the vehicle captured through the windshield.

As shown in (D), each time acceleration a2 is applied to the imaging device 101, the extraction range 110 is changed to extraction range 130, causing the output target image data 111 to be blurred into the output target image data 131, reducing visibility.

In FIG. 3, (A) shows the type of acceleration applied to the imaging device 101 when the automobile 100 turns to the right. When the automobile 100 turns to the right, the imaging device 101 is subjected to gravitational acceleration g in the vertical direction and acceleration in the turning direction of the automobile 100 (centripetal acceleration). The composite acceleration ga3 is the combination of the gravitational acceleration g and the centripetal acceleration.

(B) shows an example of tilt detection of the imaging device 101 when the automobile 100 turns to the right. The electronic gimbal of the imaging device 101 determines that the traveling direction side of the imaging device 101 has tilted in the right yaw direction (tilt angle α) due to the resultant acceleration ga3 (however, in reality, the imaging device 101 does not tilt because it is fixed). The electronic gimbal changes the imaging direction CD1 to the imaging direction CD4 so that the first fisheye lens 102a on the traveling direction side faces directly ahead in the traveling direction, i.e., to cancel out the tilt angle α. Note that since the imaging device 101 is fixed, the optical axis OA does not actually shift like the imaging direction CD4.

(C) shows an example of capturing an image of a subject by the imaging device 101 when no centripetal acceleration is applied to the imaging device 101. Specifically, for example, like (C) in FIG. 1, (C) shows an extraction range 110 of the omnidirectional image data when the imaging direction CD1 is set as the projection direction of the stitched omnidirectional image data, and output target image data 111 extracted from the omnidirectional image data in the extraction range 110.

(D) shows an example of capturing an image of a subject by the imaging device 101 when centripetal acceleration is applied to the imaging device 101. Specifically, for example, (D) shows an extraction range 140 of the omnidirectional image data when the imaging direction CD4 is set as the projection direction of the stitched omnidirectional image data, and output target image data 141 extracted from the omnidirectional image data in the extraction range 130. Of the output target image data 141, image data 141a is image data of the scenery outside the vehicle captured through the windshield, and image data 141b is image data of the interior of the vehicle.

As shown in (D), each time acceleration a2 is applied to the imaging device 101, the extraction range 110 is changed to extraction range 130, causing the output target image data 111 to be blurred into the output target image data 131, reducing visibility.

In this way, the imaging device 101 according to this embodiment implements two modes that can be switched manually or automatically in order to suppress the above-mentioned decrease in visibility. One is a sensor use mode in which tilt detection is performed by an electronic gimbal using detection signals from an acceleration sensor and a gyro sensor as usual.

When the sensor use mode is applied, the electronic gimbal performs tilt detection regardless of whether the imaging device 101 is mounted on the automobile 100, and changes the extraction range set by the projection direction according to the detected tilt. Therefore, when the imaging device 101 is mounted on the automobile 100, the extraction range and output target image data will be changed from (C) to (D) in Figures 1 to 3 due to the tilt detection of the electronic gimbal.

The other mode is a mode different from the sensor use mode, that is, a suppression mode that suppresses tilt correction by the sensor. Specifically, there are three suppression modes, for example, a stop mode, a non-use mode, and an image detection mode. The stop mode is a mode in which the output of detection signals from the acceleration sensor or gyro sensor is stopped. The non-use mode is a mode in which detection signals from the acceleration sensor or gyro sensor are accepted but not used. As a result, in the stop mode or non-use mode, the electronic gimbal does not use detection signals from the acceleration sensor or gyro sensor, and the extraction range of the image data to be output described above is not changed.

The image detection mode detects movement inside the vehicle from image data without correction based on detection signals from the acceleration sensor or gyro sensor. In image detection mode, the image data used for detection is not omnidirectional image data, but image data after stitching before being attached to a sphere. Since movements inside the vehicle are minute, even if tilt occurs, it does not affect visibility as much as in the sensor use mode.

In this way, when the imaging device 101 is mounted inside the automobile 100, the user can set the imaging device 101 to the suppression mode in advance so that the extraction range 110 is not changed, or the imaging device 101 can automatically switch to the suppression mode when tilt is detected. This allows the extraction range 110 to be maintained during playback even if the electronic gimbal detects tilt, improving the visibility of image data captured while the automobile 100 is traveling.

<Example of Hardware Configuration of Imaging Apparatus 101>
4 is a block diagram showing an example of a hardware configuration of the imaging device 101. The imaging device 101 includes a processor 401, a storage device 402, an operation device 403, a sensor 404, an LSI (Large Scale Integration) 405, an omnidirectional camera 406, and a communication IF (Interface) 407. These are connected by a bus 408. The processor 401 controls the imaging device 101. The storage device 402 serves as a working area for the processor 401.

The storage device 402 is a non-temporary or temporary recording medium that stores various programs and data. Examples of the storage device 402 include a ROM (Read Only Memory), a RAM (Random Access Memory), a HDD (Hard Disk Drive), and a flash memory. The operation device 403 operates data. Examples of the operation device 403 include a button and a touch panel.

The sensor 404 is an acceleration sensor or gyro sensor for generating information related to the attitude of the imaging device 101. The acceleration sensor is a sensor 404 that detects acceleration. In an XYZ coordinate system (origin O is the center of gravity of the imaging device 101) spanned by the X, Y, and Z axes, for example, the X axis is the traveling direction of the automobile 100, the Z axis is the vertical direction, and the Y axis is a direction parallel to the X and Z axes. In this case, the acceleration sensor detects acceleration v = (vx, vy, vz). vx is the acceleration in the X-axis direction, vy is the acceleration in the Y-axis direction, and vz is the acceleration in the Z-axis direction.

The gyro sensor is a sensor 404 that detects angular velocity ω = (ωy, ωp, ωr). ωy is the angular velocity of the yaw angle θy around the Z axis, ωp is the angular velocity of the pitch angle θp around the Y axis, and ωr is the angular velocity of the roll angle θr around the X axis. The acceleration v is converted to angular velocity ω by the processor 401 or LSI 405.

The LSI 405 is an integrated circuit that executes specific processing such as image processing and compression/expansion processing. The omnidirectional camera 406 captures an image of a subject and generates omnidirectional image data. The omnidirectional camera 406 is a camera in which the back sides of two image sensors are opposed to each other and fisheye lenses 102a and 102b with a field angle of 180 degrees or more are provided in front of each image sensor. The communication IF 407 is connected to the terminal 400 via a network and transmits and receives data. The terminal 400 can receive, store, and play back the omnidirectional image data output from the imaging device 101.

<Example of Functional Configuration of Imaging Apparatus 101>
5 is a block diagram showing an example of a functional configuration of the imaging device 101. The imaging device 101 includes an imaging unit 501, an acquisition unit 502, a specification unit 503, a determination unit 504, a motion detection unit 505, a luminance detection unit 506, a distance detection unit 507, and a calculation unit 508.

The imaging unit 501 captures an image of a subject inside the automobile 100. Specifically, the imaging unit 501 is, for example, the spherical camera 406 shown in FIG. 4. The imaging unit 501 has the first fisheye lens 102a, the second fisheye lens 102b, the first imaging element 501a, the second imaging element 501b, and the image processing unit 501c described above. The first fisheye lens 102a collects light from a first subject in front of it and outputs it to the first imaging element 501a. The second fisheye lens 102b collects light from a second subject in front of it and outputs it to the second imaging element 501b.

The first imaging element 501a receives light of the first subject incident through the first fisheye lens 102a, performs photoelectric conversion, and generates first circular fisheye image data. In this example, the first circular fisheye image data is circular fisheye image data obtained by capturing an image of the first subject from inside the vehicle as a subject facing the direction of travel. The second imaging element 501b receives light of the second subject incident through the second fisheye lens 102b, performs photoelectric conversion, and generates second circular fisheye image data. In this example, the second circular fisheye image data is circular fisheye image data obtained by capturing an image of the second subject from inside the vehicle as a subject facing the direction of travel.

The image processing unit 501c converts the first circular fisheye image data and the second circular fisheye image data into first equirectangular converted image data and second equirectangular converted image data, respectively, divides the second equirectangular converted image data, combines (stitches) it with the first equirectangular converted image data, and outputs the combined equirectangular converted image data (see Figure 8).

The image processing unit 501c attaches the combined equirectangular converted image data to the virtual spherical surface of the virtual sphere, thereby generating spherical image data that can be rotated 360 degrees in any direction, with the center of the virtual sphere, which corresponds to the center of gravity of the imaging device 101, as the center of rotation. The image processing unit 501c stores the combined equirectangular converted image data and the spherical image data in the storage device 402.

The image processing unit 501c is realized by causing the processor 401 to execute a program stored in the storage device 402 shown in FIG. 4, or by the LSI 405.

Figure 6 is an explanatory diagram showing an example (first half) of the generation of spherical image data by the image processing unit 501c. In Figure 6, (A) shows the state of the subject image formed by the first fisheye lens 102a and the second fisheye lens 102b. (B) shows the first circular fisheye image data 601a and the second circular fisheye image data 601b input from the first image sensor 501a and the second image sensor 501b by the image formation of (A) and subjected to Debayer processing. (C) shows equirectangular conversion. The first equirectangular converted image data 602a and the second equirectangular converted image data 602b are image data that have been equirectangular converted from the first circular fisheye image data 601a and the second circular fisheye image data 601b, respectively.

Figure 7 is an explanatory diagram showing an example (second half) of the generation of spherical image data by the image processing unit 501c. In Figure 7, (D) shows the latter image division, that is, the divided image data 602b1, 602b2 obtained by dividing the second equirectangular converted image data 602b of (C). (E) shows stitching. The combined equirectangular converted image data 701 is image data obtained by stitching the first circular fisheye image data 601a and the two divided image data 602b1, 602b2.

(F) shows the process of pasting the combined equirectangular converted image data 701 onto the virtual spherical surface 703 of the virtual sphere 702. (G) shows parallel projection onto the virtual spherical surface 703. The combined equirectangular converted image data 701 is pasted onto the virtual spherical surface 703, and output target image data 710 is extracted from the virtual spherical surface 703 within the extraction range TA by parallel projection from the projection direction (for example, the direction of the optical axis OA). The extraction range TA in (G) is set when the combined equirectangular converted image data 701 is reproduced after (F).

Returning to FIG. 5, the acquisition unit 502 acquires first information related to a change in the attitude of the imaging device 101. Specifically, for example, the acquisition unit 502 acquires the first information based on a detection signal from the sensor 404. If the sensor 404 is a gyro sensor, the detection signal is angular velocity, and therefore the acquisition unit 502 acquires the detection signal of angular velocity as the first information. Also, if the sensor 404 is an acceleration sensor, the detection signal is acceleration, and therefore the acquisition unit 502 converts the acceleration into angular velocity.

Specifically, for example, the acquisition unit 502 integrates the acceleration and calculates the amount of movement (x2-x1, y2-y1, z2-z1) per unit time in the XYZ coordinate system. The coordinate value (x1, y1, z1) is the position before the movement, and the coordinate value (x2, y2, z2) is the position before the movement. The acquisition unit 502 determines the movement angle from the coordinate value (x1, y1, z1) to the coordinate value (x2, y2, z2) using the coordinate values (x1, y1, z1), (x2, y2, z2), and the origin O, and by differentiating by unit time, calculates the angular velocity (ωy, ωp, ωr) as the movement angle per unit time as the first information.

The acquisition unit 502 is specifically realized, for example, by causing the processor 401 to execute a program stored in the storage device 402, or by the LSI 405.

The identification unit 503 identifies an invisible area where the outside of the moving body cannot be seen based on the image data of the subject generated by the output from the imaging unit 501. The image data referred to here is the combined equirectangular converted image data 701 in (E) of FIG. 7. The invisible area is specifically an area where nothing other than the moving body on which the imaging device 101 is mounted cannot be seen. The moving body includes the interior space of the moving body, such as the driver's seat, passenger seat, and rear seat, and the exterior surface of the moving body that is visible from the interior space (for example, the bonnet of a car). In other words, the invisible area where the outside of the moving body cannot be seen is an area that is blocked by the interior space of the moving body and the exterior surface of the moving body, and the scenery outside the moving body cannot be seen from inside the moving body. On the other hand, the visible area is an area outside the moving body that is visible from the interior space of the moving body on which the imaging device 101 is mounted (for example, scenery).

Figure 8 is an explanatory diagram showing the visible and non-visible areas in the combined equirectangular converted image data 701. Examples include the inner surfaces that make up the interior space of the vehicle, the backsides of the door panels, the dashboard, armrests, center console, console box, glove box, instrument panel, and seats. The hood seen through the windshield is also a non-visible area. On the other hand, the view outside the vehicle seen through the windshield is a visible area. The door windows are visible areas because the view outside the vehicle is visible regardless of whether the window glass is open or closed.

Figure 9 is an explanatory diagram showing the motion vectors of the visible area in the combined equirectangular converted image data 701. When the automobile 100 is moving, the scenery in the visible area changes, but the scenery in the invisible area remains the same. Therefore, the identification unit 503 detects the motion vector mv from the time-series group of combined equirectangular converted image data 701, and identifies areas where the motion vector mv is equal to or larger than a predetermined size as visible areas, and identifies areas where the motion vector mv is smaller than the predetermined size as invisible areas.

The identification unit 503 is specifically realized, for example, by causing the processor 401 to execute a program stored in the storage device 402, or by the LSI 405.

The determination unit 504 executes either a first determination process or a second determination process different from the first determination process, based on the invisible area identified by the identification unit 503. The first determination process is a process for determining a variation parameter for varying the position of the extraction range TA of the output target image data 710 from the image data, based on the first information acquired by the acquisition unit 502.

Specifically, for example, the first determination process is a process in the sensor use mode described above. The determination unit 504 integrates the angular velocity, which is the first information, to obtain an angle. The direction of the angular velocity and the obtained angle become fluctuation parameters that fluctuate the position of the extraction range TA of the output target image data 710. The first determination process of the acquisition unit 502 and the determination unit 504 functions as an electronic gimbal.

Figure 10 is an explanatory diagram showing the amount of movement of the position of the extraction range TA. The projection line PD1 is a line segment that passes through the center of gravity of the extraction range TA1 in the combined equirectangular converted image data 701, and is the projection direction of (G) in Figure 7. α is the angle obtained by integrating the angular velocity, which is the first information, that is, the amount of fluctuation related to the position of the extraction range TA1. The amount of fluctuation, angle α, and the direction of the angular velocity become a fluctuation parameter. The projection line PD2 is a line segment that transitions the projection line PD1 by angle α from the origin O in the direction of the angular velocity, and as a result, the extraction range TA1 moves to become the extraction range TA2. The projection line PD2 passes through the center of gravity of the extraction range TA2.

Returning to FIG. 5, the second determination process of the determination unit 504 is a mode different from the sensor use mode, that is, a suppression mode that suppresses tilt correction by the sensor 404. More specifically, for example, there is a stop mode that stops the output of the detection signal from the acceleration sensor or gyro sensor, a non-use mode that does not use the detection signal, and an image detection mode that determines a variation parameter that varies the position of the extraction range TA based on the combined equirectangular converted image data after stitching in FIG. 7(E).

In the image detection mode, the image data to be detected is not the omnidirectional image data, but the combined equirectangular converted image data 701 after stitching in FIG. 7(E). Since the movement inside the vehicle, i.e., the amount of fluctuation (angle) of the fluctuation parameter, is small, even if tilt occurs, it does not affect the fluctuation of the position of the extraction range TA as compared to the sensor use mode.

The determination unit 504 is specifically realized, for example, by causing the processor 401 to execute a program stored in the storage device 402, or by the LSI 405.

The motion detection unit 505 detects a motion vector mv based on a time-series group of image data generated by output from the imaging unit 501. This image data is the combined equirectangular converted image data 701 after stitching in FIG. 7(E). Specifically, for example, the motion detection unit 505 detects the motion vector mv based on the movement angle of the region of interest in two pieces of combined equirectangular converted image data that are consecutive in time.

Figure 11 is an explanatory diagram showing the movement angle obtained by the motion detection unit 505. Point P1 is a point on the attention area of the preceding combined equirectangular converted image data of two temporally consecutive combined equirectangular converted image data, and point P2 is a point on the attention area of the succeeding combined equirectangular converted image data. In other words, it shows that point P1 has moved to point P2. In the equirectangular coordinate system, point P1 is a point specified by radius r, yaw angle θ1, and pitch angle φ1, and point P2 is a point specified by radius r, yaw angle θ2, and pitch angle φ2.

The radius r is the subject distance to the subject inside the vehicle that is in the invisible area (for example, the inner wall of the vehicle interior space). The movement angle α is the angle between the line segments OP1 and OP2. The vector that passes through the imaginary sphere 703 with the point P1 as the start point and the point P2 as the end point is the motion vector mv.

In this case, the identification unit 503 identifies the above-mentioned invisible area based on this motion vector mv. For example, if the differences in the direction and magnitude of each motion vector mv in the group of motion vectors mv are both within an allowable range, the identification unit 503 identifies a visible area and an invisible area for each area in which the group of motion vectors mv exists. Specifically, for example, if the direction and magnitude of each motion vector mv are the same and the magnitude of each motion vector mv is equal to or greater than a threshold value, the identification unit 503 identifies the area in which the group of motion vectors mv exists as a visible area.

That is, the identification unit 503 identifies the area as one that shows the changing scenery seen through the windshield or window glass. Also, if the direction of the motion vectors mv is different and the magnitude is less than a threshold, the identification unit 503 identifies the area in which the group of motion vectors mv exists as an unrecognizable area. That is, the identification unit 503 identifies the area as one that shows the interior of the vehicle, which remains almost unchanged.

The motion detection unit 505 may detect the above-mentioned motion vector mv using a detection signal from the sensor 404 (angular velocity if it is a gyro sensor, or acceleration if it is an acceleration sensor). If the detection signal is from a gyro sensor, the motion detection unit 505 can obtain a movement angle by dividing the angular velocity by the frame rate of the combined equirectangular transformed image data 701. The direction and movement angle of the angular velocity are detected as the motion vector mv. In the case of an acceleration sensor, the motion detection unit 505 converts it into the direction and movement angle of the angular velocity by polar coordinate conversion similar to that of the acquisition unit 502.

When determining the direction of angular velocity and the angle of movement using the combined equirectangular transformed image data 701 or a gyro sensor, conversion from the XYZ coordinate system to the polar coordinate system is not required, so higher accuracy can be achieved than when determining the direction and angle of movement using an acceleration sensor.

Movement detection unit 505 is specifically realized, for example, by having processor 401 execute a program stored in storage device 402, or by LSI 405.

The brightness detection unit 506 detects the brightness of the combined equirectangular converted image data 701. In the imaging state of the imaging device 101, the outside of the vehicle is considered to have a higher brightness than the inside of the vehicle. Therefore, the brightness detection unit 506 detects the brightness of the combined equirectangular converted image data 701 in order to divide the combined equirectangular converted image data 701 into a visible area and an invisible area.

In this case, the identification unit 503 identifies the invisible region based on the luminance detected by the luminance detection unit 506. Specifically, for example, the identification unit 503 divides the combined equirectangular converted image data 701 into regions, detects the luminance for each divided region, and obtains the average luminance. The identification unit 503 then identifies regions with a luminance lower than the average luminance as invisible regions, and regions with a luminance equal to or higher than the average luminance as visible regions.

In addition, instead of the average brightness, a threshold brightness may be set in advance, and the identification unit 503 may identify areas with brightness lower than the threshold brightness as invisible areas and areas with brightness equal to or higher than the threshold brightness as visible areas.

The brightness detection unit 506 is specifically realized, for example, by having the processor 401 execute a program stored in the storage device 402, or by the LSI 405.

The distance detection unit 507 detects the subject distance of the image data. That is, the distance detection unit 507 detects the subject distance from the imaging device 101 to the subject captured as the combined equirectangular converted image data. The imaging unit 501 has a first fisheye lens 102a, a first imaging element 501a, a second fisheye lens 102b, and a second imaging element 501b. The angle of view of both the first fisheye lens 102a and the second fisheye lens 102b is 180 degrees or more.

Here, if the angle of view of the first fisheye lens 102a and the second fisheye lens 102b are both 200 degrees, the first image data generated by the output from the first imaging element 501a and the second image data generated by the output from the second imaging element 501b overlap in the range from 160 degrees to 200 degrees.

The overlapping area is overlapped by stitching as shown in FIG. 7(E). By using triangulation on this overlapping area, the distance detection unit 507 can detect the subject distance in the overlapping area. For the angle of view outside the overlapping area, for example, a distance measurement sensor such as an ultrasonic sensor or LiDAR (Light Detection and Ranging) may be installed as the sensor 404 to detect the subject distance outside the overlapping area.

In this case, the identification unit 503 identifies the invisible area based on a preset reference distance and the subject distance detected by the distance detection unit 507. The reference distance is the maximum distance to the inner wall of the vehicle when the imaging device 101 is fixedly installed at any position inside the vehicle, and is set in advance.

If the detected subject distance is equal to or greater than the reference distance, the identification unit 503 determines that the subject is outside the vehicle, and identifies the area in which the subject image is present in the combined equirectangular converted image data 107 as a visible area. If the detected subject distance is less than the reference distance, the identification unit 503 determines that the subject is inside the vehicle, and identifies the area in which the subject image is present in the combined equirectangular converted image data 107 as a non-visible area.

Figure 12 is an explanatory diagram showing an example of detecting the subject distance in the overlapping region. In Figure 12, the imaging device 101 calculates the subject distance D to the subject 1200 that exists in the overlapping region where the angles of view of the first fisheye lens 102a and the second fisheye lens 102b overlap. Here, the known distance on the optical axis OA of the first fisheye lens 102a and the second fisheye lens 102b is B, the intersection of the optical axis OA with the perpendicular line from the subject 1200 to the optical axis OA (the length of the perpendicular line is the subject distance D), and X is the unknown distance on the optical axis OA from the installation position of the second fisheye lens 102b to the intersection point C.

γ is the angle of view of the first fisheye lens 102a with respect to the subject 1200, and α = 180 - γ/2. β is the half angle of view of the second fisheye lens 102b with respect to the subject 1200 that corresponds to γ, and is uniquely determined once γ is determined. The subject distance D is expressed by the following formulas (1) and (2).

D=Qtanα (1)
D=(Q+B)tanβ...(2)

By eliminating the unknown distance Q from the above equations (1) and (2), the subject distance D can be calculated using the following equation (3).

D=B×(tanα×tanβ)/(tanα−tanβ)...(3)

Therefore, when the overlapping area is a visible area, the imaging device 101 can determine the subject distance to a subject outside the vehicle. For example, if the imaging device 101 is installed inside the vehicle so that the overlapping area is a visible area of the windshield and rear windshield, the subject-to-subject distance between each of the subjects in front of and behind the automobile 100 can be determined.

The distance detection unit 507 is specifically realized, for example, by causing the processor 401 to execute a program stored in the storage device 402, or by the LSI 405.

The calculation unit 508 calculates second information regarding the change in the posture of the image capture device 101 based on the invisible region. Specifically, for example, the calculation unit 508 calculates the angular velocity from the motion vector mv of the invisible region as the second information. As for this angular velocity, as described above, the amount of movement per unit time in the XYZ coordinate system obtained from the motion vector mv (x2-x1, y2-y1, z2-z1) is calculated.

The coordinate values (x1, y1, z1) are the position before the movement, and the coordinate values (x2, y2, z2) are the position before the movement. The calculation unit 508 calculates the movement angle from the coordinate values (x1, y1, z1) to the coordinate values (x2, y2, z2) using the coordinate values (x1, y1, z1), (x2, y2, z2), and the origin O, and calculates the angular velocity (ωy, ωp, ωr) as the movement angle per unit time as the second information by differentiating by unit time. The angular velocity of the second information represents the movement inside the automobile 100. In other words, the angular velocity of the second information is a small value compared to the angular velocity of the first information because it is the shaking of the automobile 100 itself.

The calculation unit 508 is executed, for example, when an automatic switching function between the first decision process and the second decision process is set in the imaging device 101. Therefore, when the calculation unit 508 calculates the angular velocity of the second information, the decision unit 504 switches to and executes one of the decision processes based on the angular velocity of the first information and the angular velocity of the second information calculated by the calculation unit 508.

For example, if the angular velocity difference (which may be a differentiated angular difference) is equal to or greater than a predetermined threshold value, the difference between the motion vector mv inside and outside the vehicle is large, and the determination unit 504 determines that the automobile 100 is moving. Therefore, in order to suppress fluctuations in the extraction range TA as shown in (D) of Figures 1 to 3, if the current determination process is the second determination process, the determination unit 504 continues to execute the second determination process, and if the current determination process is the first determination process, it switches to and executes the second determination process. This makes it possible to automatically switch between the first and second determination processes, eliminating the hassle of the user having to manually switch. This improves convenience.

The calculation unit 508 is specifically realized, for example, by causing the processor 401 to execute a program stored in the storage device 402, or by the LSI 405.

<Example of manual mode switching procedure>
13 is a flowchart showing an example of a processing procedure for manual mode switching by the imaging device 101. The imaging device 101 waits for a setting input by manual operation from the operation device 403 (step S1301: No). If a setting input has been received (step S1301: Yes), the imaging device 101 determines whether the input mode is a sensor use mode (step S1302).

If the current mode is the sensor use mode (step S1302: Yes), the imaging device 101 switches the current mode to the sensor use mode (step S1303). If the current mode is the sensor use mode, the imaging device 101 maintains the sensor use mode. On the other hand, if the current mode is not the sensor use mode (step S1302: No), the imaging device 101 switches the current mode to the image detection mode (step S1304). If the current mode is the image detection mode, the imaging device 101 maintains the image detection mode. This causes the imaging device 101 to end manual mode switching.

In this way, by implementing manual mode switching, for example when the user holds the device in his/her hand, the sensor use mode can be set manually, and the orientation of the imaging device 101 according to the detection signal from the sensor 404 is used to change the extraction position of the extraction range TA to extraction ranges 120, 130, and 140 according to the inclination of the imaging device 101, as shown in (D) of Figures 1 to 3.

On the other hand, for example, when the imaging device 101 is mounted on the automobile 100, the imaging device 101 can be manually set to the image detection mode, and the imaging device 101 will maintain the extraction position of the extraction range TA of the imaging device 101 as shown in (C) of Figures 1 to 3 within the extraction range 110, without using the attitude of the imaging device 101 according to the detection signal from the sensor 404. In other words, the imaging device 101 can ignore the effects of acceleration/deceleration and turning of the automobile 100, and capture images as if the automobile 100 had not accelerated/decelerated or turned.

<Example of imaging process procedure by imaging device 101 in sensor use mode>
14 is a flowchart showing an example of an imaging process procedure by the imaging device 101 in the sensor use mode. The imaging device 101 waits for a start trigger (e.g., input of an imaging start button) from the operation device 403 (step S1401: No). When the start trigger is received (step S1401: Yes), the acquisition unit 502 acquires an angular velocity based on a detection signal from the sensor 404, and calculates an angle by multiplying the angular velocity by a unit time (e.g., a frame rate of the omnidirectional image data). As a result, the acquisition unit 502 acquires a variation parameter for varying the position of the extraction range TA, which is configured by the direction and angle indicated by the angular velocity (step S1402).

The imaging device 101 also acquires image data from the first imaging element 501a and the second imaging element 501b, and generates spherical image data as shown in Figs. 6 and 7 (step S1403). The imaging device 101 then associates the spherical image data generated in step S1403 with the variable parameters acquired in step S1402, and stores the data in the storage device 402 (step S1404).

The imaging device 101 waits for an end trigger (for example, input of an imaging end button) from the operation device 403 (step S1405). If there is no end trigger (step S1405: No), the process returns to step S1402. By repeatedly executing steps S1402 to S1404, it is possible to store time-series spherical image data, i.e., video data, in the storage device 402. If there is an end trigger (step S1405: Yes), the imaging device 101 ends imaging of the spherical image data.

By storing the spherical image data in association with the variation parameters, the position of the extraction range TA during playback varies depending on the variation direction (angular velocity direction) and variation amount (angle) indicated by the variation parameters. Therefore, it is possible to display an image of the extraction range TA according to the change in posture of the imaging device 101.

<Example of automatic mode switching procedure>
Fig. 15 is a flowchart showing an example of a processing procedure for automatic mode switching of the imaging device 101. In Fig. 15, the sensor use mode is set as the default mode. The imaging device 101 waits for a start trigger (for example, input of an imaging start button) from the operation device 403 (step S1501: No), and when the start trigger is received (step S1501: Yes), as shown in Fig. 7E, acquires combined equirectangular converted image data Et (step S1502). The subscript t indicates the chronological number of the combined equirectangular converted image data E at the time of acquisition in step S1502.

The imaging device 101 acquires a first angular velocity as first information related to the attitude of the imaging device 101 from the detection signal from the sensor 404 by the acquisition unit 502 (step S1503). The imaging device 101 detects a motion vector mv using the acquired combined equirectangular converted image data Et and the combined equirectangular converted image data Et-1 acquired immediately before (step S1504). The imaging device 101 identifies a visible area and a non-visible area in the combined equirectangular converted image data Et from the detected motion vector mv by the identification unit 503 (step S1505).

The imaging device 101 calculates the second angular velocity from the motion vector mv of the identified non-visible area by the calculation unit 508 (step S1506). Then, the imaging device 101 obtains the angular velocity difference between the first angular velocity and the second angular velocity, and determines whether the absolute value of the angular velocity difference is equal to or greater than a threshold value (step S1507). Here, the angular velocity difference is used, but the first angular velocity and the second angular velocity may be multiplied by a unit time (for example, the frame rate of the omnidirectional image data) to convert them into a first angle and a second angle, and the angular difference may be used.

If the absolute value of the angular velocity difference (or angle difference) is less than the threshold value (step S1508: No), the automobile 100 is not moving, and the imaging device 101 switches to the sensor use mode (step S1508). If the current mode is the sensor use mode, the sensor use mode is maintained.

On the other hand, if the absolute value of the angular velocity difference (or angle difference) is equal to or greater than the threshold value (step S1507: Yes), the automobile 100 is moving, and the imaging device 101 switches to the image detection mode (step S1509). If the current mode is the image detection mode, the image detection mode is maintained.

The imaging device 101 waits for an end trigger (for example, input of an imaging end button) from the operation device 403 (step S1510). If there is no end trigger (step S1510: No), the process returns to step S1502. By repeatedly executing steps S1502 to S1510, the behavior of the automobile 100 can be constantly monitored and automatic switching between the sensor use mode and the image detection mode can be performed. If there is an end trigger (step S1510: Yes), the imaging device 101 ends the automatic mode switching.

In this way, automatic mode switching can automatically switch between sensor use mode and image detection mode, eliminating the hassle of users having to manually switch between modes, improving convenience.

<Example of imaging process procedure by imaging device 101 in image detection mode>
Fig. 16 is a flowchart showing an example of an imaging process procedure 1 by the imaging device 101 in the image detection mode. The imaging process procedure 1 of Fig. 16 is an example of a process procedure when a motion vector mv is detected by the motion detection unit 505. The imaging device 101 waits for a start trigger (for example, input of an imaging start button) from the operation device 403 (step S1601: No), and when the start trigger is received (step S1601: Yes), as shown in (E) of Fig. 7, acquires combined equirectangular converted image data Et (step S1502). The subscript t indicates the chronological number of the combined equirectangular converted image data E at the time of acquisition in step S1602.

The imaging device 101 detects a motion vector mv using the acquired combined equirectangular converted image data Et and the combined equirectangular converted image data Et-1 acquired immediately before in time by the motion detection unit 505 (step S1603). The imaging device 101 identifies the visible and invisible areas in the combined equirectangular converted image data Et from the detected motion vector mv by the identification unit 503 (step S1604).

The imaging device 101 calculates the second angular velocity from the motion vector mv of the non-visible region in the combined equirectangular converted image data Et by the calculation unit 508 (step S1605). The imaging device 101 calculates the second angle by multiplying the calculated second angular velocity by a unit time (for example, the frame rate of the omnidirectional image data) (step S1606). This second angle is an angle derived from the motion vector mv of the non-visible region, and is therefore a very small angle caused by vibration of the automobile 100, for example. This second angle is the amount of variation of the variation parameter that varies the position of the extraction range TA.

Then, the imaging device 101 associates the spherical image data Ft with the variation parameters (the direction of the second angular velocity and the second angle) and stores them in the storage device 402 (step S1404). The imaging device 101 waits for an end trigger (for example, input of an imaging end button) from the operation device 403 (step S1608). If there is no end trigger (step S1608: No), the process returns to step S1602. By repeatedly executing steps S1602 to S1607, imaging in the image detection mode can be performed. If there is an end trigger (step S1608: Yes), the imaging device 101 ends the processing in the image detection mode.

In this way, in the image detection mode, the detection signal from the sensor 404 is not used, and therefore there is no influence from the angular velocity or angle derived from the detection signal from the sensor 404. Therefore, it is possible to display an image without positional fluctuation, as shown in (C) of Figures 1 to 3, without operating the electronic gimbal.

Figure 17 is a flowchart showing an example of an imaging process procedure 2 by the imaging device 101 in image detection mode. The example of the imaging process procedure 2 in Figure 17 is an example of a process procedure when luminance is detected by the luminance detection unit 506. Note that the same processes as the example of the imaging process procedure 1 in Figure 16 are given the same step numbers, and their explanations are omitted.

The imaging device 101 detects the luminance from each area of the combined equirectangular converted image data Et acquired in step S1602 using the luminance detection unit 506 (step S1703). The imaging device 101 identifies the visible area and the invisible area in the combined equirectangular converted image data Et from the detected luminance using the identification unit 503 (step S1704).

The imaging device 101 calculates a motion vector mv from the combined equirectangular converted image data Et, Et-1 using the calculation unit 508, and calculates a second angular velocity from the motion vector mv of the non-visible area in the combined equirectangular converted image data Et (step S1705).

In this way, in the image detection mode, the detection signal from the sensor 404 is not used, and therefore there is no influence from the angular velocity or angle derived from the detection signal from the sensor 404. Therefore, it is possible to display an image without positional fluctuation, as shown in (C) of Figures 1 to 3, without operating the electronic gimbal.

In addition, the interior of a vehicle generally has a lower brightness than the exterior of the vehicle. For example, during the day, sunlight is blocked by the ceiling of the vehicle interior space, so the interior of the vehicle has a lower brightness than the exterior of the vehicle. Also, at night, the headlights of the automobile 100 and the lighting outside the vehicle make it brighter than the interior of the vehicle, so the interior of the vehicle has a lower brightness than the exterior of the vehicle. By utilizing such properties, it is possible to identify visible and invisible areas using only the combined equirectangular converted image data Et without using motion vector mv detection, thereby improving processing speed.

Figure 18 is a flowchart showing an example 3 of the imaging process procedure by the imaging device 101 in the image detection mode. The example 3 of the imaging process procedure in Figure 17 is an example of the process procedure when the luminance is detected by the distance detection unit 507. Note that the same processes as the example 1 of the imaging process procedure in Figure 16 are given the same step numbers, and the description thereof will be omitted.

The imaging device 101 detects the subject distance from the overlapping area during stitching in the combined equirectangular converted image data Et using the distance detection unit 507 (step S1803). The imaging device 101 compares the subject distance with a preset reference distance to identify visible and invisible areas within the overlapping area (step S1804).

The imaging device 101 calculates a motion vector mv from each overlapping area of the combined equirectangular converted image data Et, Et-1 using the calculation unit 508, and calculates a second angular velocity from the motion vector mv of the non-visible area within the overlapping area in the combined equirectangular converted image data Et (step S1805).

In this way, in the image detection mode, the detection signal from the sensor 404 is not used, and therefore there is no influence from the angular velocity or angle derived from the detection signal from the sensor 404. Therefore, it is possible to display an image without positional fluctuation, as shown in (C) of Figures 1 to 3, without operating the electronic gimbal.

In addition, since the subject distance of the overlapping region is calculated, it is possible to spatially identify whether the subject is inside or outside the vehicle. In addition, since the range of the combined equirectangular converted image data Et to be processed to calculate the variation parameters is limited to the overlapping region, there is no need to use the entire range of the combined equirectangular converted image data Et, and the processing efficiency of the image detection mode can be improved.

<Example of playback process for spherical image data>
19 is a flowchart showing an example of a procedure for playing back omnidirectional image data. The playing back process for the omnidirectional image data is executed, for example, by the terminal 400 to which the omnidirectional image data and the variation parameters are transferred. The terminal 400 waits for a playback start trigger (step S1901: No). If there is a playback start trigger (step S1901: Yes), the terminal 400 reads the omnidirectional image data Fi at the i-th playback position and the associated variation parameters (step S1902).

The terminal 400 updates the extraction range TA with the spherical image data Fi and the associated variation parameters (step S1903). The terminal 400 displays the output target image data 710 of the extraction range TA after the update in step S1903 on the display screen of the terminal 400. The terminal 400 increments the playback position i (step S1905) and determines whether an end trigger has been accepted (step S1906).

If the end trigger has not been accepted (step S1906: No), the process returns to step S1902. As a result, steps S1902 to S1904 are executed for the next spherical image data Fi. If the end trigger has been accepted (step S1906: Yes), the terminal 400 ends the playback process.

In this way, since the omnidirectional image data and the variation parameters are associated with each other, the output target image data 710 can be displayed while updating the extraction range TA during playback. Therefore, for omnidirectional image data captured in the sensor use mode, the position of the extraction range TA varies to extraction ranges 120, 130, and 140 as shown in (D) of FIGS. 1 to 3, and the output target image data 710 is displayed, and for omnidirectional image data captured in the image detection mode, the position of the extraction range TA is maintained at extraction range 110 as shown in (C) of FIGS. 1 to 3, and the output target image data 710 is displayed.

<Application to motorcycles>
20 is an explanatory diagram showing an example of imaging by the imaging device 101 mounted on a motorcycle. (A) is a side view of the motorcycle 2000 mounted with the imaging device 101, (B) is a plan view of the motorcycle 2000 mounted with the imaging device 101, and (C) is a front view of the motorcycle 2000 mounted with the imaging device 101. (C) shows a state in which the motorcycle 2000 is leaning due to centrifugal force c of turning.

As shown in (A), the imaging device 101 is subjected to gravitational acceleration g, and as shown in (B), when turning left, the imaging device 101 is subjected to a centrifugal force c of turning. When turning, the motorcycle 2000 leans, so a resultant acceleration gc of the gravitational acceleration g and the centrifugal force c of turning is obtained, as shown in (C).

(D) shows an example of capturing an image of a subject by the imaging device 101 when centrifugal force gc due to turning is not applied. Specifically, for example, (D) shows an extraction range 2001 of the omnidirectional image data when the imaging direction CD1 is set as the projection direction of the stitched omnidirectional image data, and output target image data 2010 extracted from the omnidirectional image data in the extraction range 2001. The output target image data 2010 includes image data 2011 of a stationary motorcycle 2000 and image data 2012 of a moving landscape.

(E) shows an example of capturing an image of a subject by the imaging device 101 when centrifugal force gc due to turning is applied. Specifically, for example, (E) shows an extraction range 2002 of the omnidirectional image data when the imaging direction CD1 is set as the projection direction of the stitched omnidirectional image data, and output target image data 2020 extracted from the omnidirectional image data in the extraction range 2002. The output target image data 2020 includes image data 2021 of a stationary motorcycle 2000 and image data 2022 of a moving landscape.

In (D), when the imaging device 101 is in sensor use mode, if the bike 2000 tilts as in (C), the electronic gimbal of the imaging device 101 corrects the tilt using the resultant acceleration gc, generating a variation parameter that varies the position of the extraction range TA. In other words, although the bike 2000 is tilted, the electronic gimbal corrects the position of the extraction range TA as if it were not tilted. Therefore, the imaging device 101 becomes the extraction range 2001 shown in (D) due to the correction of the electronic gimbal.

In (D), when the imaging device 101 is in image detection mode, if the motorcycle 2000 tilts as in (C), the electronic gimbal of the imaging device 101 does not perform tilt correction due to the resultant acceleration gc, and generates a fluctuation parameter that hardly changes the position of the extraction range TA. Therefore, since the imaging device 101 is not corrected by the electronic gimbal, the extraction range becomes 2002 as shown in (E). In this way, when applied to the motorcycle 2000, output target image data 710 that takes into account the tilt of the motorcycle 2000 due to turning is obtained, making it possible to display dynamic touring images.

(1) As described above, the imaging device 101 according to this embodiment can realize automatic switching between a first determination process (e.g., processing in a sensor use mode) that determines a variation parameter for varying the position of the extraction range TA of the output target image data 710 from the image data based on first information (e.g., a first angular velocity) regarding a change in the posture of the imaging device 101, and a second determination process that is different from the first determination process.

(2) In addition, by calculating second information (e.g., a second angular velocity) regarding a change in the attitude of the imaging device 101 based on the invisible area and executing one of the decision processes, automatic switching between the first decision process and the second decision process can be realized depending on the behavior of the moving object.

(3) Furthermore, by making the second decision process a process for stopping the output of data detected by the sensor 404, it is possible to forcibly block data from the sensor 404 and execute the second decision process.

(4) In addition, by determining a variation parameter for varying the position of the extraction range TA based on image data, the second determination process can be performed without using data from the sensor 404.

(5) Furthermore, the second determination process can be executed, for example, in an image detection mode by determining a variation parameter that varies the position of the extraction range TA of the output target image data 710 from the image data without using the first information (for example, the first angular velocity) regarding the change in posture of the imaging device 101.

(6) Furthermore, it is possible to identify the motion vector mv and the invisible area without using data obtained from the sensor 404, and it is possible to suppress fluctuations in the position of the extraction range due to data obtained from the sensor 404.

(7) In addition, it is possible to identify an invisible area without detecting a motion vector mv, thereby speeding up the identification process.

(8) In addition, by identifying the invisible area based on the reference distance to the inner wall of the moving body and the subject distance, it is possible to identify the invisible area without detecting the motion vector mv, thereby speeding up the identification process.

(9) Furthermore, by calculating second information related to changes in the posture of the imaging device 101 and determining a variation parameter for varying the position of the extraction range TA, it is possible to identify behavior within the moving body (e.g., vibration) via the invisible area, rather than the behavior of the main body of the moving body (e.g., acceleration/deceleration or turning), and reflect this in the extraction range TA. Therefore, if there is no vibration, there will be no variation in the extraction range TA, and if there is vibration, the extraction range TA will vary in accordance with the vibration.

(10) In addition, by identifying an invisible area based on the direction and magnitude of the motion vector mv, it is possible to improve the accuracy of identifying the invisible area.

(11) Furthermore, by detecting the motion vector mv based on the movement angle of the area of interest from the combined equirectangular transformed image data Et-1 to the combined equirectangular transformed image data Et, the calculation is performed in a polar coordinate system, thereby improving the detection accuracy of the motion vector mv.

(12) Moreover, by detecting the subject distance D based on the overlapping range of the first equirectangular converted image data 602a and the second equirectangular converted image data 602b, it is possible to spatially identify whether the subject is inside or outside the moving body. Furthermore, since the range to be processed to obtain the variation parameters is limited to the overlapping region, it is not necessary to use the entire range of the combined equirectangular converted image data, and the efficiency of the second determination process can be improved.

(13) Furthermore, according to the control program of this embodiment, automatic switching between a first determination process that determines a variation parameter for varying the position of the extraction range TA of the output target image data 710 from the image data based on information related to changes in the posture of the imaging device 101 (e.g., angular velocity), and a second determination process that is different from the first determination process can be realized by software. Therefore, even if the control program is not installed in an imaging device, by installing the control program, the imaging device can perform automatic switching between the first determination process and the second determination process.

In addition, when image data (e.g., spherical image data) captured by the imaging device 101 described above is associated with a variable parameter, and a playback device (e.g., terminal 400) specifies an extraction range TA from the spherical image data to play back image data to be output, the position of the extraction range TA is controlled by the variable parameter. That is, if the variable parameter is obtained by the first determination process, the position of the extraction range TA varies, but if the variable parameter is obtained by the second determination process, the position of the extraction range TA hardly varies.

In this way, when the imaging device 101 is not mounted on a moving body, the first determination process is executed. On the other hand, when the imaging device 101 is mounted on a moving body, the first determination process is automatically switched to the second determination process, and the electronic gimbal is stopped from being overcorrected, thereby making it possible to display omnidirectional image data with a stable imaging direction.

100 Automobile, 101 Imaging device, 102a First fisheye lens, 102b Second fisheye lens, 107, 701 Combined equirectangular converted image data, TA, 110, 120, 130, 140, 2001, 2002 Extraction range, 111, 121, 131, 141, 710, 2010, 2020 Output target image data, 400 Terminal, 401 Processor, 402 Storage device, 403 Operation device, 404 Sensor, 406 Omnidirectional camera, 501 Imaging unit, 501a First imaging element, 501b Second imaging element, 501c Image processing unit, 502 Acquisition unit, 503 Identification unit, 504 Determination unit, 505 Detection unit, 506 Luminance detection unit, 507 Distance detection unit, 508 Calculation unit, 1200 Subject: 2000 bikes

Claims (14)

An imaging device that can be mounted on a moving object,
An imaging unit that captures an image of a subject;
an acquisition unit that acquires first information regarding a change in a posture of the imaging device by a sensor ;
a specifying unit that specifies an invisible area where it is not possible to see outside the moving body based on image data of the subject generated by output from the imaging unit;
a determination unit that executes one of a first determination process that determines a variation parameter for varying a position of an extraction range of output target image data from the image data based on first information acquired by the acquisition unit and a second determination process different from the first determination process, based on second information indicating a change in posture of the imaging device obtained from a motion vector in a non-visible area identified by the identification unit and the first information ,
An imaging apparatus , wherein the second determination process is a process of invalidating a detection signal of the sensor, or a process of determining a variation parameter for varying a position of the extraction range based on the image data . 2. The imaging device according to claim 1,
The determination unit executes one of the determination processes based on a difference between a value of the first information and a value of the second information. 2. The imaging device according to claim 1,
The imaging device, wherein the process of invalidating the detection signal of the sensor is a process of stopping output of data detected by the sensor. 2. The imaging device according to claim 1,
The process of invalidating the detection signal of the sensor determines a variation parameter for varying a position of an extraction range of output target image data from the image data , without using the first information . 2. The imaging device according to claim 1 ,
a motion detection unit that detects the motion vector based on a time-series image data group generated by an output from the imaging unit;
The identification unit identifies the non-visible area based on a motion vector detected by the motion detection unit . 2. The imaging device according to claim 1 ,
a brightness detection unit for detecting the brightness of the image data ;
The identification unit identifies the invisible area based on the luminance detected by the luminance detection unit . 2. The imaging device according to claim 1 ,
a distance detection unit for detecting a subject distance of the image data ;
The identification unit identifies the invisible area based on a reference distance to an inner circumferential wall of the moving body and a subject distance detected by the distance detection unit . 6. The imaging device according to claim 5 ,
The determination unit determines a variation parameter for varying a position of the extraction range using the second information in the second determination process . 6. The imaging device according to claim 5 ,
The identification unit is configured to identify the non-visible area based on a direction and a magnitude of the motion vector . 6. The imaging device according to claim 5 ,
An imaging device, wherein the motion detection unit detects the motion vector based on a movement angle of a region of interest from first image data in the time-series image data group to second image data that is temporally later than the first image data . 8. The imaging device according to claim 7 ,
the imaging unit includes a first fisheye lens, a first imaging element configured to image a first subject through the first fisheye lens, a second fisheye lens arranged such that its concave surface faces the first fisheye lens, and a second imaging element configured to image a second subject through the second fisheye lens;
An imaging device, wherein the distance detection unit detects the subject distance based on an overlapping range of first image data generated by output from the first imaging element and second image data generated by output from the second imaging element. 2. The imaging device according to claim 1,
The imaging device, wherein the process of invalidating the detection signal from the sensor is a process of not using the detection signal from the sensor. 13. The imaging device according to claim 12,
The imaging device, wherein the process of not using the detection signal from the sensor is a process of stopping output of the detection signal from the sensor, or a process of accepting the detection signal from the sensor but not using the detection signal. A control program for causing a processor to execute control of an imaging device that can be mounted on a moving body and captures an image of a subject,
The processor,
acquiring first information regarding a change in the attitude of the imaging device by a sensor;
Identifying an invisible area where it is not possible to view anything outside the moving body based on image data of the subject captured by the imaging device;
executes one of a first determination process that determines a variation parameter for varying a position of an extraction range of output target image data from the image data based on the first information and a second determination process that is different from the first determination process, based on the first information and second information indicating a change in attitude of the imaging device obtained from a motion vector in the non-visible area;
The second determination process is a process of invalidating a detection signal of the sensor, or a process of determining a variation parameter for varying a position of the extraction range based on the image data.