JP7487464B2 - IMAGE PROCESSING APPARATUS, IMAGING APPARATUS, VIDEO PLAYBACK SYSTEM, METHOD, AND PROGRAM - Google Patents

Description

This disclosure relates to an image processing device, an imaging device, a video playback system, a method, and a program.

In recent years, spherical cameras that can capture video in all directions from a shooting location have become widespread. In addition to capturing images in a fixed position, such spherical cameras are also expected to be used by users who hold the camera in their hands while moving around to capture video. In such cases, correction techniques are known that correct the camera's camera shake and tilt to produce videos with less blur.

A spherical video captured by an omnidirectional camera usually has an initial front direction when the viewer views the spherical video. In general, the front direction when viewing follows the front direction of the spherical camera when capturing the video (for example, the front direction of one of the lenses in a twin-lens camera). However, if the front direction when viewing follows the movement of the spherical camera when capturing the video, the displayed image changes significantly depending on the shooting conditions of the photographer, and there is a risk that the viewer may feel sick when viewing the video.

For example, when a photographer shoots with a spherical camera fixed to his/her head, if the front direction when viewing follows the movement of the spherical camera during shooting, the spherical video will rotate significantly when the photographer turns his/her head to look around, which can cause nausea in the viewer. Furthermore, the large rotation of the spherical video may make the video itself very difficult to watch. The same applies when the spherical camera is fixed to the head or when it is handheld, as long as the front direction changes significantly during shooting.

JP 2017-147682 A (Patent Document 1) discloses a configuration for generating a spherical image by performing coordinate transformation on image data with correction that removes minute shaking components from shaking in the horizontal plane in global coordinates. However, although high-frequency components of shaking are corrected for rotation around a vertical axis, the correction follows intentional movements such as the photographer turning his or her head to look around, which is not sufficient in that it can cause nausea in the viewer when viewing the image.

This disclosure has been made in consideration of the above points, and aims to provide an image processing device that can reduce the viewer's nausea caused by rotation around a reference axis during filming when watching a video.

In order to address the above-mentioned problems, the present disclosure provides an image processing device having the following characteristics for processing a video including continuous images expressed in a coordinate system including at least angular coordinates around a predetermined axis. The image processing device includes a video data acquisition means for acquiring video data, and a sensor data acquisition means for acquiring sensor data corresponding to the video data, the sensor data including first rotation correction data and second rotation correction data. The image processing device further includes a rotation correction means for applying correction to the image to cancel rotational changes around a reference axis based on the first rotation correction data and the second rotation correction data , so that a reference for displaying the image is fixed to a specific direction during shooting across multiple frames of the video. The image processing device further includes the sensor data corresponding to the video including data output by an acceleration sensor and data output by an angular velocity sensor. The image processing device further includes the first rotation correction data being time series data of a first angle change amount in a front direction of the image relative to a reference value, and the second rotation correction data being time series data of a second angle change amount in a traveling direction relative to the reference value. This image processing device further defines a direction in a plane perpendicular to a reference axis based on the attitude of the camera capturing video in a reference frame at a specified time as a specific direction during shooting, and the amount of rotational change is based on the difference between the direction based on the attitude in the reference frame and the direction in a plane perpendicular to the reference axis based on the attitude of the camera in the frame to be corrected.

The above configuration can reduce the viewer's feeling of nausea caused by rotation around a reference axis during filming when watching a video.

FIG. 2 is a cross-sectional view of the omnidirectional camera constituting the omnidirectional video system according to the present embodiment. 1A and 1B are diagrams illustrating a case where an omnidirectional camera according to an embodiment of the present invention is fixed to a head. FIG. 1 is a hardware configuration diagram of a spherical video system according to an embodiment of the present invention. FIG. 2 is a block diagram of main functions related to a spherical video playback function realized on the spherical video system according to the present embodiment. FIG. 1A is an image data flow diagram for generating a spherical image, and FIG. 1B is a diagram for explaining the data structure of the spherical image when it is represented on a plane and when it is represented on a spherical surface. 5A to 5C are diagrams for explaining zenith correction and rotation correction of a spherical image according to the present embodiment. 1A to 1C are diagrams illustrating a celestial sphere image obtained by zenith correction and rotation correction of the celestial sphere image according to the present embodiment. FIG. 11 is a sequence diagram illustrating a process from capturing a video to viewing the video in the spherical video system according to a specific embodiment. FIG. 11 is a sequence diagram illustrating a process from capturing a video to viewing the video in an omnidirectional video system according to another specific embodiment. 10 is a flowchart showing a rotation process for frame synthesis data executed by the omnidirectional camera constituting the omnidirectional video system according to the present embodiment. FIG. 1 is a diagram for explaining, with an example, the relationship between the moving direction and the front direction of shooting when a photographer moves while wearing an omnidirectional camera on his/her head; 12A and 12B are diagrams (1/2) for explaining a celestial sphere image before rotation correction and a celestial sphere image after rotation correction captured when a photographer performs the action shown in FIG. 11 . 12A and 12B are diagrams (2/2) for explaining a celestial sphere image before rotation correction and a celestial sphere image after rotation correction captured when a photographer performs the action shown in FIG. 11 . 13 is a flowchart showing a rotation process for frame synthesis data executed by an omnidirectional camera constituting an omnidirectional video system according to another embodiment.

The present embodiment will be described below, but the embodiment is not limited to the embodiment described below. Note that in the following embodiment, a spherical camera and a spherical video system will be used as examples of an image processing device (and an imaging device) and a video playback system, respectively.

The overall configuration of the spherical video system according to this embodiment will be described below with reference to Figures 1 to 3.

Figure 1 is a cross-sectional view of an omnidirectional camera 110 that constitutes an omnidirectional video system according to this embodiment. The omnidirectional camera 110 shown in Figure 1 includes an imaging body 12, a housing 14 that holds components such as the imaging body 12, a controller 15, and a battery 16, and a shutter button 18 provided on the housing 14.

The imaging body 12 shown in FIG. 1 includes two imaging optical systems 20A and 20B and two imaging elements 22A and 22B such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor. Each imaging optical system 20 is configured as, for example, a fisheye lens with 6 groups and 7 lenses. In the embodiment shown in FIG. 1, the fisheye lens has a total angle of view greater than 180 degrees (=360 degrees/n; the number of optical systems n=2), preferably has an angle of view of 185 degrees or more, and more preferably has an angle of view of 190 degrees or more. A combination of such a wide-angle imaging optical system 20 and one imaging element 22 is referred to as a wide-angle imaging optical system. In the embodiment described, a case where the number of optical systems is two is described as an example, but the number of optical systems may be one or three or more as long as it is capable of capturing a wide-angle area. In addition, it is not limited to a fisheye lens having an angle of view of 180 degrees or more.

In the embodiment shown in FIG. 1, the imaging optical systems 20A and 20B are of the same specification and are combined in the opposite directions so that their optical axes coincide. The image pickup elements 22A and 22B convert the distribution of received light into an image signal and sequentially output image frames to an image processing means on the controller. As will be described in detail later, the images captured by the image pickup elements 22A and 22B are synthesized to generate an image with a solid angle of 4π steradians (hereinafter referred to as a "spherical image"). The spherical image captures all directions that can be seen from the shooting point. A spherical video is then composed of successive frames of the spherical image.

Here, in the embodiment described, a celestial sphere image and a celestial sphere video are generated, but the celestial sphere image and the celestial sphere video may be a so-called panoramic image and a panoramic video in which only the horizontal plane is captured at 360 degrees, or a hemispherical image or a semicylindrical image captured at a 180-degree angle of view. Any image data having spherical or cylindrical image information may be used. That is, the video and the images included therein according to this embodiment are images represented in a coordinate system including angular coordinates around a predetermined axis. In the case of the celestial sphere image/video, hemispherical image/video, and partial spherical image, the radius is set to a constant (e.g., 1) and the image is represented in a spherical coordinate system including angular coordinates representing the angle between a predetermined axis and the radius and angular coordinates around the predetermined axis. In the case of the panoramic image/video or a partial image thereof, the radius is set to a constant (e.g., 1) and the image is represented in a cylindrical coordinate system including coordinates in the axial direction of the cylinder and angular coordinates around the axis.

2 illustrates a case where the omnidirectional camera 110 in FIG. 1 is fixed to the head 50 of the photographer. FIG. 2 illustrates an example where the omnidirectional camera 110 is fixed to a hat-shaped fixture 52. By attaching the omnidirectional camera 110 to the head 50 of the photographer in this manner, the front direction of the image data captured by the omnidirectional camera 110 follows the head swing H of the photographer. The front direction of the image is a preset direction, and is a reference direction displayed when playing back video data. FIG. 2 also illustrates the moving direction T of the photographer. The photographer may move forward while looking around, and the front direction of the image captured by the head swing H does not necessarily match the moving direction T. In addition to fixing the omnidirectional camera 110 to a hat-shaped fixture 52 and attaching it to the head 50 as shown in FIG. 2, the omnidirectional camera 110 itself may be shaped so as to be attachable to the head 50 (a helmet or hat with an omnidirectional camera function embedded in it).

Figure 3 (A) shows the hardware configuration of the omnidirectional camera 110 that constitutes the omnidirectional video system according to this embodiment. The omnidirectional camera 110 corresponds to the image processing device or imaging device in the embodiment to be described.

The omnidirectional camera 110 includes a CPU (Central Processing Unit) 112, a ROM (Read Only Memory) 114, an image processing block 116, a video compression block 118, a dynamic random access memory (DRAM) 132 connected via a DRAM interface 120, and a sensor 136 connected via an external sensor interface 124. Here, the CPU 112, the ROM 114, the image processing block 116, and the video compression block 118 are mounted on the controller 15 shown in FIG. 1.

The CPU 112 controls the operation of each part of the omnidirectional camera 110 and the overall operation. The ROM 114 stores an operating system (OS), control programs, and various parameters written in code that the CPU 112 can decipher. The image processing block 116 is connected to two image sensors 130A and 130B (image sensors 22A and 22B in FIG. 1), and receives image signals of images captured by each of them. The image processing block 116 includes an ISP (Image Signal Processor) and performs shading correction, Bayer interpolation, white balance correction, gamma correction, and the like on the image signal input from the image sensor 130.

The video compression block 118 is a codec block that performs video compression and decompression such as MPEG-4 AVC/H.264. The DRAM 132 provides a storage area for temporarily storing data when various signal processing and image processing are performed.

In the embodiment shown in FIG. 3A, the sensor 136 includes an angular velocity sensor 136A and an acceleration sensor 136B. The angular velocity sensor 136A is a sensor that detects three-axis angular velocity components. The acceleration sensor 136B is a sensor that detects three-axis acceleration components. The detected acceleration components and angular velocity components are used to perform zenith correction of the omnidirectional image in the direction of gravity and rotation correction around the direction of gravity, as described below. The sensor 136 may further include other sensors, such as a geomagnetic sensor (also called an electronic compass) for determining an azimuth angle, and the azimuth angle may be used to perform rotation correction around the direction of gravity.

The omnidirectional camera 110 further includes an external storage interface 122, a USB (Universal Serial Bus) interface 126, and a serial block 128. An external storage 134 is connected to the external storage interface 122. The external storage interface 122 controls reading and writing from and to the external storage 134, such as a memory card inserted into a memory card slot. A USB connector 138 is connected to the USB interface 126. The USB interface 126 controls USB communication with an external device, such as a smartphone, connected via the USB connector 138. The serial block 128 controls serial communication with an external device, such as a smartphone, and is connected to a wireless NIC (Network Interface Card) 140.

When the power switch is operated to turn the power on, the OS and control programs are loaded into the main memory. The CPU 112 controls the operation of each part of the device according to the program loaded into the main memory, and temporarily stores data required for control in the memory. This realizes each of the functional parts and processes of the spherical camera 110, which will be described later.

Figure 3 (B) shows the hardware configuration of an information terminal 150 that constitutes the spherical video system according to this embodiment. The information terminal 150 corresponds to the information processing device in the embodiment to be described.

The information terminal 150 shown in FIG. 3(B) includes a CPU 152, a RAM 154, an internal storage 156, an input device 158, an external storage 160, a display 162, a wireless NIC 164, and a USB connector 166.

The CPU 152 controls the operation of each part of the information terminal 150 and the overall operation. The RAM 154 provides a working area for the CPU 152. The internal storage 156 stores control programs, such as an operating system and applications that handle processing on the information terminal 150 side according to this embodiment, written in code that the CPU 152 can decipher. The CPU 152 may be made up of multiple CPUs.

The input device 158 is an input device such as a touch screen, and provides a user interface. The input device 158 accepts various instructions from an operator, such as correction of the omnidirectional video. The external storage 160 is a removable recording medium attached to a memory card slot or the like, and records various data such as image data and still image data in video format. The display 162 displays the corrected omnidirectional video on the screen in response to a user operation. The wireless NIC 164 establishes a wireless communication connection with an external device such as the omnidirectional camera 110. The USB connector 166 makes a USB connection with an external device such as the omnidirectional camera 110.

When the information terminal 150 is powered on, a control program is read from the internal storage 156 and loaded into the RAM 154. The CPU 152 controls the operation of each part of the device according to the control program loaded into the RAM 154, and temporarily stores data required for control in memory. This realizes each of the functional parts and processes of the information terminal 150 described below.

The spherical video playback function of the spherical video system according to this embodiment will be described below with reference to Figures 4 to 9.

Figure 4 shows the main functional blocks 200 related to the omnidirectional video playback function realized on the omnidirectional video system according to this embodiment. As shown in Figure 4, the functional blocks of the omnidirectional camera 110 include an image processing unit 210 and an output unit 240. The image processing unit 210 includes a first imaging unit 212, a second imaging unit 214, an angular velocity detection unit 216, an acceleration detection unit 218, an image synthesis unit 220, a zenith correction amount calculation unit 222, a first rotation amount calculation unit 224, a second rotation amount calculation unit 226, a storage unit 228, and an image rotation unit 230. The processing of each unit is performed by the CPU 112 and the image processing block 116 of the controller 15 executing various programs.

In contrast, the functional blocks of the information terminal 150, as also shown in FIG. 4, include a receiving unit 252 and a display control unit 254. Processing of each unit is performed by the CPU 152 provided in the information terminal 150 executing various programs.

First, the functional blocks on the spherical camera 110 side will be explained below.

The first imaging unit 212 sequentially captures images using the imaging element 130A to generate a chronologically continuous fisheye image A. The second imaging unit 214 sequentially captures images using the imaging element 130B to generate a chronologically continuous fisheye image B. The image synthesis unit 220 performs stitching processing on the chronologically continuous fisheye image A (first frame data) and the chronologically continuous fisheye image B (second frame data) to generate frame synthesis data. The frame synthesis data is video data that is the main body of the omnidirectional video data, including a plurality of frames that respectively constitute a omnidirectional video from the start to the end of shooting. The fisheye image A and the fisheye image B are partial images for constituting the omnidirectional image, and the first and second frame data are also video data that can constitute a omnidirectional video, including frames of partial images from the start to the end of shooting. The image synthesis unit 220 outputs the generated frame synthesis data to the image rotation unit 230. The first imaging unit 212, the second imaging unit 214, and the image synthesis unit 220 may constitute the imaging means in this embodiment.

Below, the generation of a celestial sphere image and the generated celestial sphere image will be described with reference to FIG. 5. FIG. 5(A) describes the data structure of each image and the data flow of the image in the generation of the celestial sphere image. First, the image directly captured by each image sensor 130 is an image that covers roughly a hemisphere of the celestial sphere in its field of view. Light incident on the imaging optical system is imaged on the light receiving area of the image sensor 130 according to a predetermined projection method. The image captured here is captured by a two-dimensional image sensor whose light receiving area forms a planar area, and becomes image data expressed in a planar coordinate system. Typically, the obtained image is configured as a fisheye image that includes the entire image circle onto which each shooting range is projected, as shown by "fisheye image A" and "fisheye image B" in FIG. 5(A).

The multiple fisheye images of each frame captured by the multiple image sensors 130 are corrected for distortion and synthesized to generate one omnidirectional image corresponding to each frame. In the synthesis process, first, each hemispherical image including complementary hemispherical portions is generated from each fisheye image configured as a planar image. Then, the two hemispherical images including each hemispherical portion are stitched (image synthesis) based on matching of overlapping regions to generate a omnidirectional image including the entire omnidirectional sphere.

Fig. 5(B) is a diagram illustrating the data structure of image data of a celestial sphere image used in this embodiment, expressed on a plane. Fig. 5(C) is a diagram illustrating the data structure of image data of a celestial sphere image, expressed on a spherical surface. As shown in Fig. 5(B), the image data of the celestial sphere image is expressed as an array of pixel values whose coordinates are a vertical angle φ made with respect to a predetermined axis and a horizontal angle θ corresponding to a rotation angle around the predetermined axis. The vertical angle φ is in the range of 0 degrees to 180 degrees (or -90 degrees to +90 degrees), and the horizontal angle θ is in the range of 0 degrees to 360 degrees (or -180 degrees to +180 degrees).

As shown in FIG. 5(C), each coordinate value (θ, φ) in the omnidirectional format is associated with each point on a sphere that represents all directions centered on the shooting point, and the all directions are associated with the omnidirectional image. The planar coordinates of a fisheye image captured with a fisheye lens and the coordinates on the spherical surface of the omnidirectional image are associated with each other in a predetermined conversion table. The conversion table is data that is created in advance by a manufacturer or the like according to a predetermined projection model based on the design data of each lens optical system, and is data that converts a fisheye image into a omnidirectional image.

In the embodiment described below, the fisheye image is converted into a celestial sphere image, and the image rotation unit 230 rotates the frame synthesis data after the stitching process. However, the order of the conversion process, synthesis process, and rotation process is not particularly limited. The image rotation unit 230 may rotate (correct) each of the partial images, fisheye image A and fisheye image B (two hemispherical images including complementary hemispherical parts converted from the partial images), and then perform stitching. In addition to performing rotation processing on an image in a celestial sphere format, the rotation processing can be reflected in the conversion table that converts a fisheye image into a celestial sphere image, and a celestial sphere image after rotation processing can be obtained from the fisheye image A and the fisheye image B based on the conversion table in which the rotation processing is reflected. In this embodiment, performing correction on a moving image means obtaining a moving image including an image in which a predetermined correction is finally reflected, regardless of the order of the conversion process, synthesis process, and rotation process. In addition, in this embodiment, a spherical image is created by stitching two fisheye images, but this is not limited to the above, and in other embodiments, for example, a spherical image or a panoramic image may be created by stitching three or more images.

Now, refer to FIG. 4 again. The angular velocity detection unit 216 is an angular velocity detection mechanism including the angular velocity sensor 136A, and outputs angular velocity data obtained by measuring angular velocity components in three axial directions using the angular velocity sensor 136A during shooting. The angular velocity data is time series data of the angular velocity generated around the three axes of the omnidirectional camera 110 (the angular velocity sensor 136A therein) from the start to the end of shooting. The angular velocity data does not need to be recorded in one-to-one correspondence with the frames, and can typically be recorded at a rate faster than the frame rate. In this case, the correspondence with each frame can be found later using the timestamp as a clue. Alternatively, the angular velocity may be recorded in one-to-one correspondence with each frame of the omnidirectional video. In this embodiment, the angular velocity detection unit 216 constitutes a recording means for recording sensor data in association with each frame of the video, and the angular velocity data can be included in the sensor data in this embodiment.

The acceleration detection unit 218 is an acceleration detection mechanism including the acceleration sensor 136B, and outputs acceleration data obtained by measuring acceleration components in three axial directions using the acceleration sensor 136B during shooting. The acceleration data is time series data of the three-axial acceleration of the omnidirectional camera 110 (the acceleration sensor 136B therein) from the start to the end of shooting. The acceleration data does not need to be recorded in one-to-one correspondence with the frames, and can typically be recorded at a rate faster than the frame rate. In this case, the correspondence with each frame can be found later using the timestamp as a clue. Alternatively, the acceleration may be recorded in one-to-one correspondence with each frame of the omnidirectional video. In this embodiment, the acceleration detection unit 218 constitutes a recording means for recording sensor data in association with each frame of the video, and the acceleration data can be included in the sensor data in this embodiment.

The above-mentioned acceleration data and angular velocity data obtained directly from acceleration sensor 116B and angular velocity sensor 136A are data based on the three axial directions of the sensor coordinate system (or the coordinate system of omnidirectional camera 110). On the other hand, acceleration sensor 136B outputs the sum of the acceleration based on motion and the gravitational acceleration, and the above-mentioned acceleration data and angular velocity data can be converted into an absolute coordinate system based on the gravitational acceleration axis. Therefore, the above-mentioned sensor data indicates the amount of displacement in the absolute coordinate system.

The zenith correction amount calculation unit 222 calculates the tilt angle at the time of imaging with respect to the reference axis in each frame based on the acceleration data output from the acceleration detection unit 218, and creates zenith correction data. The zenith correction data is time-series data of the tilt angle of the omnidirectional camera 110 with respect to the reference axis from the start to the end of imaging, which is recorded in association with each frame of the omnidirectional video. The tilt angle with respect to the reference axis is typically configured as a vector consisting of values of the dimension of acceleration. The reference axis typically coincides with the direction of gravity in which gravitational acceleration acts, and the following description will be continued assuming that the reference axis is the direction of gravity. Since the acceleration sensor 136B does not distinguish between gravity and inertial force, the tilt angle obtained from the acceleration sensor 136B may preferably be corrected based on a signal measured by the angular velocity sensor 136A.

Based on the acceleration data output from the acceleration detection unit 218 and the angular velocity data output from the angular velocity detection unit 216, the first rotation amount calculation unit 224 calculates a vector of the shooting front direction of the omnidirectional camera 110 in a coordinate system of a plane (horizontal plane) perpendicular to the gravity direction from the change in the relative angle of the omnidirectional camera 110 for each frame, and records the first rotation correction data. The first rotation correction data is time series data of the angle change in the shooting front direction from a reference value from the start to the end of shooting, which is associated with each frame of the omnidirectional video and recorded. The angle change in the shooting front direction is represented by the difference between the shooting front direction in the horizontal plane based on the attitude of the omnidirectional camera 110 in the reference frame and the shooting front direction in the horizontal plane based on the attitude of the omnidirectional camera 110 in the target frame. Here, the reference frame can be, for example, a frame at the start of shooting or a frame at the start of application of this correction (in the case of an embodiment in which the application of correction can be instructed after the start of shooting). The first rotation correction data is represented by, for example, a numerical value of the rotation angle or a vector. In this embodiment, the first rotation correction data may also be included in the sensor data, and the first rotation amount calculation unit 224 may constitute a sensor data acquisition means and recording means that acquires and records the sensor data in association with each frame of the video.

Based on the acceleration data output from the acceleration detection unit 218 and the angular velocity data output from the angular velocity detection unit 216, the second rotation amount calculation unit 226 calculates the vector of the traveling direction of the omnidirectional camera 110 in a plane coordinate system of a plane (horizontal plane) perpendicular to the direction of gravity for each specified number of frames, and generates second rotation correction data. The traveling direction in the frame at the start of shooting is equal to the shooting front direction of the omnidirectional camera 110 in the frame at the start of shooting. The second rotation correction data is time-series data of the angle change amount of the traveling direction from a reference value from the start to the end of shooting, which is recorded in association with each frame of the omnidirectional video or each specified number of frames. The angle change amount of the traveling direction is represented by the difference between the traveling direction (shooting front direction) of the omnidirectional camera 110 in the horizontal plane in the reference frame and the traveling direction of the omnidirectional camera 110 in the horizontal plane in the target frame. The second rotation correction data is, for example, a numerical value of the rotation angle or a vector. In this embodiment, the second rotation correction data may also be included in the sensor data, and the second rotation amount calculation unit 226 may constitute a sensor data acquisition means and recording means that acquires and records the sensor data in association with each frame of the video.

When the acceleration of the omnidirectional camera 110 in a specific direction among the acceleration data satisfies a predetermined condition, the specific direction at that time is determined to be the traveling direction. The predetermined condition may be, for example, a case where the acceleration in a specific direction exceeds a predetermined threshold when the vectors of multiple frames are combined. If the predetermined condition is not satisfied, the traveling direction may be the data of the previous frame. The definition of the traveling direction is not particularly limited, and may be the traveling direction of the photographer or the traveling direction of the camera itself. The second rotation amount calculation unit 226 may also function as a determination means in this embodiment.

The image rotation unit 230 acquires the frame synthesis data, zenith correction data, first rotation correction data, and second rotation correction data output by the image synthesis unit 220, the zenith correction amount calculation unit 222, the first rotation amount calculation unit 224, and the second rotation amount calculation unit 226. The image rotation unit 230 performs rotation processing (zenith correction and rotation correction) on each frame of the acquired frame synthesis data based on the acquired zenith correction data, first rotation correction data, and second rotation correction data, and outputs corrected celestial sphere frame data.

Here, zenith correction and rotation correction will be described with reference to Figs. 6 and 7. Fig. 6 is a diagram for explaining zenith correction and rotation correction of a spherical image performed in this embodiment. Fig. 7 is a diagram for explaining a spherical image obtained by zenith correction and rotation correction of a spherical image performed in this embodiment. Fig. 7(A) shows a frame of a video before zenith correction, and Fig. 7(B) shows a frame of a video after zenith correction.

As described above, image data in the omnidirectional image format is expressed as an array of pixel values whose coordinates are the vertical angle φ with respect to a predetermined axis z0 and the horizontal angle θ corresponding to the rotation angle around the predetermined axis z0. Here, if no correction is made, the predetermined axis is defined based on the omnidirectional camera 110. For example, the imaging body 12 side of the omnidirectional camera 110 shown in FIG. 1 is the head and the opposite side is the bottom, and the omnidirectional image can be defined as the predetermined axis z0 that defines the horizontal angle θ and the vertical angle φ, with the central axis passing through the center of the housing from the bottom to the head. Also, for example, the horizontal angle θ of the omnidirectional image can be defined so that the optical axis direction of one optical element of the two imaging optical systems 20A and 20B is located at the center at the horizontal angle θ.

Zenith correction refers to a process (correction in the roll and pitch directions) of correcting a celestial sphere image (FIG. 7(A)) captured with the central axis z0 actually tilted with respect to the direction of gravity as shown in the left diagram of FIG. 6 to a celestial sphere image (FIG. 7(B)) as if it were captured with the central axis z0 aligned with the direction of gravity Z as shown in the right diagram of FIG. 6. The image rotation unit 230 functions as a zenith correction means that performs zenith correction on each frame so that a specific axis (central axis) approximately coincides with the direction of gravity, based on the tilt angle with respect to the direction of gravity in each frame included in the zenith correction data.

In contrast, rotation correction refers to a correction (Yaw direction correction) that further cancels the angle change around the gravity direction Z (change in the horizontal angle θ direction in FIG. 7 ) in the omnidirectional image corrected by zenith correction so that the central axis z0 coincides with the reference direction, and fixes the display reference (for example, the center at the horizontal angle θ in the omnidirectional format, which is the reference position displayed by default when playing video data) to a specific direction during shooting. The reference position displayed by default is an area in the omnidirectional format that is displayed when the user does not perform an operation to change the display direction for the video data. The specific direction is, for example, the front direction of the omnidirectional camera 110. The image rotation unit 230 functions as a rotation correction means that performs correction to cancel the rotation change around the reference axis (which is made to coincide with the gravity direction by zenith correction) based on the first rotation correction data and the second rotation correction data.

Here, the specific direction during shooting in which the angle change is cancelled and fixed as the reference for display may be the shooting front direction of the omnidirectional camera 110 in the above-mentioned horizontal plane in the above-mentioned reference frame. Also, after the traveling direction of the omnidirectional camera 110 is determined, the above-mentioned specific direction may be the traveling direction of the omnidirectional camera 110 in the above-mentioned horizontal plane.

More specifically, the image rotation unit 230 performs correction on each frame of the video based on the first rotation correction data and the second rotation correction data, to cancel out the rotational change around the reference axis so that the display reference approximately matches the specific direction during shooting, at least over a certain period of time over multiple frames. Here, the amount of rotational change that is canceled out is based on the difference between the specific direction in a plane perpendicular to the reference axis (e.g., the forward direction of shooting or the direction of travel at the start of shooting) and the forward direction of shooting in a plane perpendicular to the reference axis in the frame to be corrected. In addition, the period during which the specific direction is fixed is from the time when the specific direction is determined to the time when the next specific direction is determined.

In the embodiment to be described, at the start, the forward shooting direction of the omnidirectional camera 110 in the reference frame in the above-mentioned horizontal plane is set as a specific direction and the display reference (center of the image) is fixed, and after the traveling direction of the omnidirectional camera 110 is determined, the traveling direction in the above-mentioned horizontal plane is determined as a specific direction and the display reference (center of the image) is fixed to the traveling direction. In this case, regardless of the swing movement H shown in FIG. 2, at first, rotation correction is performed so that the forward shooting direction at the start point comes to the center of the image, and gradually as the traveling direction T is determined, rotation correction is performed so that the traveling direction T comes to the center of the image.

However, the method of determining the specific direction is not particularly limited. In another embodiment, the reference position displayed by default in the shooting front direction of the omnidirectional camera 110 in the above-mentioned horizontal plane in the reference frame may be fixed regardless of the traveling direction T. In such an embodiment, the rotation correction using the second rotation amount calculation unit 226 and the second rotation correction data can be omitted. In this case, the specific direction is not updated due to changes in the traveling direction during shooting, so the period fixed to the specific direction is the entire period from the start of the video or the time when the start of fixation is instructed to the end of the video or the time when the end of fixation is instructed.

The storage unit 228 is used to store the above-mentioned first frame data, second frame data, frame composite data, corrected frame composite data with zenith correction and rotation correction, acceleration data, angular velocity data, zenith correction data, first rotation correction data, and second rotation correction data. The storage unit 228 is provided as part of the storage area of the DRAM 132, external storage 134, or other storage shown in FIG. 3(A).

Note that the spherical video captured in one shot may be recorded in the storage unit 228 as, for example, a single file. In this case, various methods are envisioned for recording the spherical video.

For example, the corrected frame composite data can be recorded as one piece of spherical video data. In this case, the acceleration data, angular velocity data, zenith correction data, first rotation correction data, and second rotation correction data may be optionally recorded as metadata together with the corrected frame composite data. In this embodiment, when the corrected data becomes necessary, the corrected frame composite data can be read out.

In another embodiment, the pre-correction frame composite data can be recorded as one omnidirectional video data. In this case, the acceleration data, angular velocity data, zenith correction data, first rotation correction data, and second rotation correction data are recorded in a predetermined combination as metadata together with the pre-correction frame composite data. In this other embodiment, when post-correction data becomes necessary, the zenith correction data, first rotation correction data, and second rotation correction data are prepared from the metadata (missing data is calculated), and the image rotation unit 230 applies image rotation to the pre-correction frame composite data. For example, if acceleration data and angular velocity data are available, any missing data among the zenith correction data, first rotation correction data, and second rotation correction data can be calculated.

The frame composite data that has been zenith-corrected and rotation-corrected by the image rotation unit 230 is output to the output unit 240. The output unit 240 can transmit and output the corrected data as video data to an external information terminal 150 via the wireless NIC 140 or USB connector 138. Alternatively, the output unit 240 can output the corrected data as video data to a file in a specified storage device.

Note that although the term "video" is used, the video may be recorded in any format as long as the video can be played. For example, the video may be recorded as video data in which multiple frames are compressed using a specific codec such as H.264/MPEG-4 AVC (Advanced Video Coding) or H.265/HEVC (High Efficiency Video Coding). In addition, Motion JPEG (Joint Photographic Experts Group) is a format that represents video as a series of continuous still images, and video data may be recorded in this manner as a continuous series of still images of multiple frames, or the video may be recorded as a collection of files of still images of multiple frames. The output unit 240 is equipped with an appropriate codec.

Below, the functional blocks on the information terminal 150 side will be described. The information terminal 150 is a terminal device in which an application for communicating with the omnidirectional camera 110 and viewing and playing omnidirectional images is installed. The information terminal 150 may be a smartphone, a tablet computer, a personal computer, a head-mounted display (HMD), or the like. The information terminal 150 accepts various instructions from an operator via the application and issues various requests to the omnidirectional camera 110. For example, in response to accepting an instruction to play a specified omnidirectional video from the operator (e.g., an instruction to play a video with rotation correction applied), the information terminal 150 issues a request to the omnidirectional camera 110 for video data after correction of the specified omnidirectional video.

The receiving unit 252 of the information terminal 150 receives the video data output from the omnidirectional camera 110. The display control unit 254 of the information terminal 150 displays the omnidirectional video on a display device such as the display 162 of the information terminal 150 based on the received video data. In a specific embodiment, the display control unit 254 displays at least a part of the image of the video data that has been corrected on the display 162 based on the above-mentioned display criteria. The display control unit 254 also changes the direction displayed in the omnidirectional format in response to a user operation, and changes the area displayed on the display 162.

Note that the type of image displayed on the information terminal 150 based on the corrected image data is arbitrary. For example, the display control unit 254 may display the entire omnidirectional image on a display device, or may attach the omnidirectional image to a spherical object and display a video of an image of the spherical object observed from a predetermined position with a virtual camera at a predetermined viewing angle as a frame. In either case, in the omnidirectional format, a predetermined display standard is defined, and an image is displayed in a specific field of view based on the display standard. For example, when the entire omnidirectional image is displayed on the display 162, it can be displayed with a specific direction always fixed to the center of the display 162. When an image observed with a virtual camera is displayed on the display 162, the omnidirectional image is attached to the spherical object based on the display standard. In this case, the specific direction during shooting is not fixed to the center of the display 162, but unless the display range (the position and direction of the virtual camera) is changed, a specific direction (for example, the left hand of the traveling direction) is always fixed to the center of the display 162.

In this embodiment, the actual processing of zenith correction and rotation correction is performed using resources on the omnidirectional camera 110 side, not on the information terminal 150, and the correction results are output to and displayed on the information terminal 150. With this configuration, when sufficient resources are provided on the omnidirectional camera 110 side, it becomes possible to stably play back video while performing zenith correction and rotation correction, regardless of the processing performance of the information terminal 150.

In the embodiment described below, the output mode is to transmit image data of the omnidirectional video to the information terminal 150, but is not limited to this. If the omnidirectional camera 110 is equipped with a display device, the output mode can also be to display the image data on the display device.

In the embodiment shown in FIG. 4, the functional blocks of the omnidirectional camera 110 may further include an instruction receiving unit 242 in addition to the image processing unit 210 and the output unit 240. The instruction receiving unit 242 can receive an instruction to specify a specific direction during video capture. In the above-described embodiment, the specific direction was the front direction or the traveling direction of the video at a specific point in time, but by providing the instruction receiving unit 242, it becomes possible to specify a direction (such as the direction of a subject of interest) that is to be fixed as the reference for image display at any timing during capture. The instruction receiving unit 242 constitutes a receiving means in this embodiment.

Figures 8 and 9 are sequence diagrams that explain the process from shooting to viewing a video in a spherical video system in various specific embodiments.

Fig. 8(A) corresponds to the embodiment shown in Fig. 4. In the embodiment shown in Fig. 8(A), imaging and image processing (including image synthesis, zenith correction, and rotation correction) are performed on the omnidirectional camera 110 side, and video data after image processing is transmitted to the information terminal 150. Then, the information terminal 150 displays the video data after image processing. More specifically, in the embodiment of Fig. 8(A), the omnidirectional camera 110 acquires video data through an imaging process in S10, performs image processing on the video data in S11, and transmits the video data after image processing to the information processing terminal in S12. The information processing terminal displays the video data on a display in S13.

In the embodiment shown in FIG. 4 and FIG. 8(A), the omnidirectional camera 110 operates as an image processing device including video data acquisition means (first imaging unit 212, second imaging unit 214, and image synthesis unit 220 as necessary), sensor data acquisition means (angular velocity detection unit 216, acceleration detection unit 218, first rotation amount calculation unit 224, and first rotation amount calculation unit 224), and rotation correction means (image rotation unit 230).

4 and 8A, an example of performing image processing (including image synthesis, zenith correction, and rotation correction) using resources on the omnidirectional camera 110 side has been described, but the present invention is not limited to this. For example, some or all of the first frame data, second frame data, frame synthesis data, angular velocity data, acceleration data, zenith correction data, first rotation correction data, and second rotation correction data may be sent to the information terminal 150 side, and some or all of the image processing may be performed on the information terminal 150 side. In this case, some of the functions of the omnidirectional camera 110 side shown in the functional block diagram of FIG. 4 are performed on the information terminal 150 side by the CPU 152 on the information terminal 150 side by executing various programs. This makes it possible to perform high-load processing such as synthesis processing of the first frame data (fisheye image A) and the second frame data (fisheye image B) by the image synthesis unit 220 using a CPU with a higher processing speed, and even data with a large number of pixels can be processed in a short time or in real time.

For example, in the embodiment shown in FIG. 8(B), only imaging is performed on the omnidirectional camera 110 side (S30), and the frame data and rotation parameters before image processing are transmitted to the information terminal 150 (S31). Here, the rotation parameters collectively refer to parameters related to correction of the frame data, such as the acceleration data, angular velocity data, zenith correction data, first rotation correction data, and second rotation correction data described above. Then, the information terminal 150 performs image processing such as zenith correction and rotation correction on the received frame data based on the received rotation parameters (S32), and displays the video data after image processing (S33).

In the embodiment shown in FIG. 8(B), the information terminal 150 operates as an image processing device that includes a video data acquisition means (receiving unit), a sensor data acquisition means (receiving unit), and a rotation correction means (means corresponding to the image rotation unit 230).

In this case, image synthesis may be performed by either the omnidirectional camera 110 or the information terminal 150. When image synthesis is performed on the omnidirectional camera 110 side, the frame data transmitted from the omnidirectional camera 110 to the information terminal 150 is frame synthesis data (omnidirectional format). When image synthesis is performed on the information terminal 150 side, the frame data transmitted from the omnidirectional camera 110 to the information terminal 150 is the first frame data (fisheye image A) and the second frame data (fisheye image B). The first frame data and the second frame data may be transmitted as corresponding video data, or video data of one joined image in which two fisheye images are joined (video data in which fisheye image A and fisheye image B are joined side by side to form one image) may be transmitted. In this case, the information terminal 150 has a means corresponding to the image synthesis unit 220 as a video data acquisition means.

In addition, the data at any stage of the rotation parameters may be transmitted to the information terminal 150. For example, the acceleration data and angular velocity data described above may be transmitted to the information terminal 150 as rotation parameters. In this case, the zenith correction data, the first rotation correction data, and the second rotation correction data are calculated on the information terminal 150 side from the acceleration data and the angular velocity data. In this case, the information terminal 150 is provided with a means corresponding to the first rotation amount calculation unit 224 and the first rotation amount calculation unit 224 as a sensor data acquisition means. Alternatively, the zenith correction data, the first rotation correction data, and the second rotation correction data may be transmitted to the information terminal 150 as rotation parameters. In this case, the information terminal 150 may be provided with a receiving means for receiving the zenith correction data, the first rotation correction data, and the second rotation correction data.

In the embodiment described above with reference to FIG. 4, FIG. 8(A), and FIG. 8(B), the omnidirectional video system is composed of two devices, the omnidirectional camera 110 and the information terminal 150, but the present invention is not limited to this.

For example, as shown in FIG. 9(A), a second information terminal 170 connected to the first information terminal 150 may be included. Here, the second information terminal 170 is, for example, a head-mounted display. In the embodiment shown in FIG. 9(A), only imaging is performed on the omnidirectional camera 110 side (S50), and the frame data and rotation parameters before image processing are transmitted to the first information terminal 150 (S51). The first information terminal 150 performs image processing such as zenith correction and rotation correction on the received frame data (S52), and the video data after image processing is transmitted to the second information terminal 170 (S53). The second information terminal 170 displays the video data after image processing (S54).

In the embodiment shown in FIG. 9(A), the first information terminal 150 operates as an image processing device that includes a video data acquisition means (receiving unit), a sensor data acquisition means (receiving unit), and a rotation correction means (means corresponding to the image rotation unit 230). The image synthesis unit 220 and the first rotation amount calculation unit 224 and means corresponding to the first rotation amount calculation unit 224 may be provided in either the omnidirectional camera 110 or the first information terminal 150.

As shown in FIG. 9B, the omnidirectional video system may include a server device 190. In the embodiment shown in FIG. 9B, only image capture is performed on the omnidirectional camera 110 side (S70), and the frame data and rotation parameters before image processing are transmitted to the information terminal 150 (S71). The information terminal 150 transfers the received frame data and rotation parameters to the server device 190 (S72), and causes the server device 190 to perform image processing such as zenith correction and rotation correction on the frame data (S73). The information terminal 150 receives the video data after image processing from the server device 190 by download or streaming (S74), and displays the video data after image processing (S75). In addition, access to the server device 190 is not limited to the information terminal 150.

In the embodiment shown in FIG. 9(B), the server device 190 operates as an image processing device including a video data acquisition means (receiving unit), a sensor data acquisition means (receiving unit), and a rotation correction means (means corresponding to the image rotation unit 230). The image synthesis unit 220 and the first rotation amount calculation unit 224 and means corresponding to the first rotation amount calculation unit 224 may be provided in either the omnidirectional camera 110 or the server device 190.

In addition, if the omnidirectional camera 110 is equipped with a display device, the omnidirectional video system may be configured with the omnidirectional camera 110 alone. In that case, it is also possible for the omnidirectional camera 110 to display on its own display device.

The zenith correction and rotation correction in this embodiment will be described in more detail below with reference to Figs. 10 to 13. Fig. 10 is a flowchart showing the rotation process for frame synthesis data executed by the omnidirectional camera 110 (which may be the information terminal 150 or the server device 190 in certain embodiments) constituting the omnidirectional video system according to this embodiment.

The process shown in FIG. 10 begins at step S100 in response to the spherical camera 110 receiving a request for corrected video data.

In step S101, the omnidirectional camera 110 acquires frame synthesis data and rotation parameters. For ease of explanation, it is assumed that the recording of the first and second frame data, the generation of frame synthesis data by image synthesis, and the recording of angular velocity data and acceleration data are completed at the start of the rotation process. However, in other embodiments, such as when performing real-time distribution, the rotation process shown in FIG. 10 can be performed while sequentially recording the first and second frame data, recording the angular velocity data and acceleration data, and generating frame synthesis data by image synthesis. It is assumed here that the zenith correction data, the first rotation correction data, and the second rotation correction data have not yet been calculated.

In step S102, the omnidirectional camera 110 stores an initial value of the front direction of the omnidirectional camera 110. This front direction of the omnidirectional camera 110 at the start time is used when calculating the reference of the first rotation correction data and the second rotation correction data. Also, in step S102, the first frame is treated as the processing target, and the process proceeds to step S103.

In step S103, the zenith correction amount calculation unit 222 of the omnidirectional camera 110 calculates the tilt angle with respect to the direction of gravity in the frame being processed based on at least the acceleration data, and writes the calculated angle in the zenith correction data.

In step S104, the omnidirectional camera 110 detects the amount of change in the rotation angle in the horizontal plane compared to the previous frame by the first rotation amount calculation unit 224. In step S105, the omnidirectional camera 110 calculates the amount of angle change in the shooting front direction from the initial value by integrating the amount of angle change between frames calculated in step S104 by the first rotation amount calculation unit 224, and writes the first rotation correction data. As described above, the first rotation correction data is time series data of the amount of angle change in the shooting front direction in the target frame from the initial value of the shooting front direction of the omnidirectional camera 110 in the coordinate system of the horizontal plane.

The processes in steps S103 to S105 can be performed for each frame. Meanwhile, the processes in steps S106 to S108 can be performed for each specified number of frames. Here, the specified number of frames can be set to any value. By making the specified number of frames smaller, it is possible to increase tracking ability, but if the specified number of frames is too small, the effects of blurring components during progression and when still are likely to become apparent. Therefore, for a video of 60 frames per second (60 fps), it is recommended to set the specified number of frames to, for example, around 10 to 30 frames.

In step S106, the omnidirectional camera 110 calculates the movement distance and the travel direction vector of the omnidirectional camera 110 within a specified number of frames in the coordinate system of the horizontal plane by the second rotation amount calculation unit 226. Note that the specified frames to be calculated here may be a specified number of frames before and after the center frame in the case of playback of stored omnidirectional video data, or may be a specified number of frames from the first frame in the case of real-time distribution.

In step S107, the omnidirectional camera 110 uses the second rotation amount calculation unit 226 to determine whether the movement distance calculated in step S106 is equal to or greater than a predetermined threshold. If it is determined in step S107 that the movement distance is equal to or greater than the predetermined threshold (YES), the process proceeds to step S108. Here, the movement distance means the difference between the position of the first frame and the position of the last frame in the coordinate system of the horizontal plane for the specified number of frames.

In step S108, the omnidirectional camera 110 uses the second rotation amount calculation unit 226 to calculate the amount of change in angle of the traveling direction vector calculated in S105 from the initial value, and writes the second rotation correction data. Here, the second rotation correction data is time series data of the difference between the initial value of the traveling direction vector (equal to the initial value of the camera direction) and the traveling direction vector as the average for the specified number of frames as described above.

On the other hand, if it is determined in step S107 that the movement distance is equal to or less than the threshold (NO), processing proceeds directly to step S109.

In step S109, the omnidirectional camera 110 performs zenith correction on the target frame of the frame synthesis data using the image rotation unit 230 based on the zenith correction data (the tilt angle of the target frame therein).

In step S110, the omnidirectional camera 110 calculates the amount of rotation change (rotation correction amount) in the horizontal plane to be canceled in the target frame based on the first rotation correction data and the second rotation correction data by the image rotation unit 230. As described above, the first rotation correction data is the difference between the initial value of the shooting front direction of the reference frame in the horizontal plane and the shooting front direction at the time of the target frame, and the second rotation correction data is the difference between the initial value of the traveling direction (equal to the shooting front direction) of the reference frame in the horizontal plane and the traveling direction at the time of the target frame. Therefore, by adding up the first rotation correction data and the second rotation correction data, the amount of rotation change to be finally canceled can be calculated. Note that when the specific direction is the initial value of the shooting front direction, the second rotation correction data is the default value (zero) until the moving distance becomes equal to or greater than the threshold value, so that the amount of rotation change is based on the difference between the initial value of the shooting front direction and the shooting front direction in the horizontal plane in the correction target frame, as specified by the first rotation correction data. On the other hand, when a specific direction is determined as the travel direction, the amount of rotational change is based on the difference between the travel direction and the forward direction of the image captured in the frame to be corrected.

In step S111, the omnidirectional camera 110 applies rotation correction to the target frame based on the calculated amount of rotation correction.

In step S112, the spherical camera 110 determines whether or not there is a next frame. If it is determined in step S112 that there is a next frame (YES), the process loops to step S103. On the other hand, if it is determined in step S112 that there is no next frame (NO), the process ends in step S113.

The above-mentioned process applies corrections to the video so that the display reference roughly matches the specific orientation during shooting, at least over multiple frames over a certain period of time, to counteract rotational changes around the reference axis.

In the embodiment described above, the amount of rotation to be cancelled is calculated based on the first rotation correction data and the second rotation correction data in step S110, and rotation correction is performed collectively in step S111. However, this is not particularly limited, and in other embodiments, rotation correction based on the first rotation correction data and rotation correction based on the second rotation correction data may be applied sequentially. Furthermore, in other embodiments, zenith correction and rotation correction may be performed collectively based on the zenith correction data, the first rotation correction data, and the second rotation correction data.

The above rotation correction makes it possible to fix the reference for image display to a specific direction, such as the initial camera shooting direction or direction of travel, regardless of low-frequency rotational changes caused by tilting the camera, etc.

The relationship between the travel direction and the shooting front direction, and the rotation process by the image rotation unit at that time, will be described below with reference to Figs. 11 to 13. Fig. 11 is a diagram illustrating, with an example, the relationship between the travel direction T and the shooting front direction H when a photographer moves while wearing the omnidirectional camera 110 on his head 50. Figs. 12 and 13 are diagrams illustrating a omnidirectional image (left side) before rotation correction and a omnidirectional image (right side) after rotation correction captured when the photographer performs the action shown in Fig. 11.

Figure 11 is a diagram showing the time series with numbers 0, 1, 1', 1 to 10, and in Figure 11, the state of the omnidirectional camera 110 fixed to the head 50 of the photographer at each point in time indicated by the number is shown, along with the moving direction T of the photographer (omnidirectional camera 110) and the shooting front direction H of the omnidirectional camera 110 according to the head turning operation.

In FIG. 11, when we look at the direction of travel T, in this example, it is shown that the photographer is moving toward the "north" during the period from numbers 0 to 5. Also, when we look at the forward direction H in FIG. 11, it is shown that the photographer is facing the "north" in the same direction of travel at the time points of numbers 0 to 1, and is turning his head to look around at the time points of numbers 2 to 4, and is looking at the "west" to the left of the direction of travel at the time point of number 5. Note that numbers 1 and 1' represent the action of moving parallel to the right once and then returning to the original course, for example, when giving way to another person when passing by. Then, while looking to the "west," the photographer changes his direction of travel from the "north" to the "west" between numbers 6 and 7, and at the time points of numbers 7 and 8, it is shown that he is moving forward while looking at the "west" in the same direction of travel. Also, at the time point of number 9, he is moving "west" while looking at the "south" to the left of the direction of travel, and at the time point of number 10, he changes his direction of travel to the "south" that he was looking at. Please note that north, south, east and west are depicted for convenience, but they are not necessarily distinguishable.

12 and 13 show a omnidirectional image (left side) 300 before rotation correction and a omnidirectional image (right side) 310 after rotation correction, which correspond to the numbers 0, 1, 1', and 1 to 10 shown in FIG. 11. Also, in the omnidirectional image (right side) 310 after rotation correction, a predetermined range 316 set in the center of the omnidirectional image is shown to be displayed on the display screen of the information terminal 150.

During the periods of numbers 0, 1, 1', and 1 to 5, the photographer is moving "north," but as shown on the left side of FIG. 12 and FIG. 13, in the pre-correction omnidirectional image 300, the image rotates in response to the photographer's head-turning motion. More specifically, during the periods of numbers 0, 1, 1', and 1 to 5, the shooting front direction 302 changes from "north" to "northwest (60 degrees to the left of north)" to "northeast (60 degrees to the right of north)" to "northwest (45 degrees to the left of north)" to "west (90 degrees to the left of north)." Meanwhile, as shown on the right side of FIG. 12 and FIG. 13, in the corrected omnidirectional image 310, at each of the times of numbers 0, 1, 1', and 1 to 5, the display reference 314 is always fixed to "north," which is the shooting front direction and moving direction at the start time. Furthermore, the translation at time points 1 and 1' causes the movement direction of the omnidirectional camera 110 to change instantaneously, but because this change occurs over a short period of time, in step S107 shown in FIG. 10, it is determined that the movement distance over a specified number of frames is less than the threshold, and the travel direction is not re-determined, remaining as "north."

To explain the rotation correction by extracting representative points, at the time point of number 2 in FIG. 11, the photographer is moving north, while the omnidirectional camera 110 has rotated 60 degrees to the left and is facing "northwest". At this time, the photographer's shooting front direction H and traveling direction T of the omnidirectional camera 110 are different. Here, the rotation process of the image rotation unit 230 when the photographer changes to the state at the time point of number 2 in FIG. 11 will be explained. When the photographer changes the direction of his/her head from the time point of number 1 to the time point of number 2, that is, when the direction of the imaging device is rotated 60 degrees to the left, the image rotation unit 230 executes a rotation process to rotate 60 degrees to the right based on the first rotation correction data so that the rotation of the omnidirectional image is canceled. This makes it possible to fix the default direction displayed on the display to the traveling direction. In reality, the rotation process based on the first rotation correction data is executed sequentially for each frame that changes to the state from time point 1 to time point 2. At this time, since there is no change in the traveling direction, the rotation process based on the second rotation correction data is not executed. The same applies to the change from time point 2 to time point 3, the change from time point 3 to time point 4, and the change from time point 4 to time point 5.

In numbers 5 to 7, the photographer gradually changes his/her direction of travel to the "west" in the direction he/she is looking, but as shown on the left side of FIG. 13, in the omnidirectional image 300 before correction, the photographer is always facing the "west" and there is no change in the image. On the other hand, as shown on the right side of FIG. 13, in the omnidirectional image 310 after correction, the display reference 314 is fixed to the updated traveling directions "northwest (number 6)" and "west (number 7)", respectively. Here, the traveling direction at time point 6 is shown to have remained constant for a sufficiently long time, and then the direction is changed to be fixed to the re-determined "northwest".

Here, the rotation process of the image rotation unit when the photographer changes to the state at time points 5, 6, and 7 in FIG. 11 will be described. When the photographer changes the traveling direction T without changing the shooting front direction H from time point 5 to time point 6, based on the first rotation correction data, a rotation process is executed to rotate the omnidirectional image 90 degrees to the right so that the rotation of the omnidirectional image with respect to the shooting front direction is canceled from the initial value. On the other hand, based on the second rotation correction data, a rotation process is executed to rotate the omnidirectional image 45 degrees to the left based on the difference in the traveling direction from the initial traveling direction (initial shooting front direction) of the omnidirectional image. Then, by adding up the first rotation correction data and the second rotation correction data, a rotation process is executed to rotate the omnidirectional image 45 degrees to the right.

In addition, in the state at time number 7, based on the first rotation correction data, a rotation process is executed to rotate the celestial sphere image 90 degrees to the right so that the rotation of the celestial sphere image relative to the shooting front direction is canceled from the initial value. On the other hand, based on the second rotation correction data, a rotation process is executed to rotate the celestial sphere image 90 degrees to the left based on the difference in the traveling direction from the initial traveling direction (initial shooting front direction). Then, when the first rotation correction data and the second rotation correction data are added together, there is no rotation correction.

The change in direction of travel may be interpolated so that a smooth transition occurs from the previous direction of travel to the next direction of travel. In other words, the 45-degree angle change between time points 5 and 6, and between time points 6 and 7, can be applied by applying an interpolated value to each frame between these. Alternatively, the change in direction of travel may be instantaneous. For example, the screen display may be blacked out temporarily, and the state in which the display reference is fixed to the previous direction of travel may be switched to a state in which the display reference is fixed to the new direction of travel.

On the other hand, in the change from number 8 to number 9, the photographer changes the direction of view from "west" to "south" without changing the direction of travel. In this case, as shown on the left side of FIG. 13, at time points 8 and 9, the orientation of the photographer changes from "west" to "south" in the pre-correction omnidirectional image 300, but the direction of travel is fixed to "west", so the display reference 314 is fixed to "west" as shown on the right side of FIG. 13. On the other hand, in numbers 9 to 10, the photographer changes the direction of travel from "west" to "south" without changing the direction of view. In this case, as shown on the left side of FIG. 13, at time points 9 and 10, the orientation of the photographer is always looking to the "south" in the pre-correction omnidirectional image 300, but the direction of travel changes from "west" to "south" as shown on the right side of FIG. 13.

Here, the rotation process of the image rotation unit when the photographer changes to the state at the time points of numbers 8, 9, and 10 in FIG. 11 will be described. When the photographer changes the shooting front direction H without changing the traveling direction T from the time point of number 8 to the state at the time point of number 9, based on the first rotation correction data, a rotation process is performed to rotate 180 degrees to the right so that the rotation of the omnidirectional image with respect to the shooting front direction is canceled from the initial value. On the other hand, based on the second rotation correction data, a rotation process is performed to rotate 90 degrees to the left based on the difference in the traveling direction from the initial traveling direction (initial shooting front direction) of the omnidirectional image. Then, by adding up the first rotation correction data and the second rotation correction data, a rotation process is performed to rotate 90 degrees to the right. Also, in the state at the time point of number 10, based on the first rotation correction data, a rotation process is performed to rotate 180 degrees to the right so that the rotation of the omnidirectional image with respect to the shooting front direction is canceled from the initial value. On the other hand, based on the second rotation correction data, a rotation process is performed to rotate the image 180 degrees to the left based on the difference in the traveling direction from the initial traveling direction of the spherical image (the initial shooting front direction). Then, adding up the first rotation correction data and the second rotation correction data results in no rotation correction.

In this way, it is possible to fix the default direction shown on the display to either the forward direction or the direction of travel when shooting begins.

The zenith correction and rotation correction in another embodiment will be described in more detail below with reference to FIG. 14. FIG. 14 is a flowchart showing a rotation process for frame synthesis data executed by the omnidirectional camera 110 (which may be the information terminal 150 or the server device 190 in a specific embodiment) constituting the omnidirectional video system according to another embodiment. Note that FIG. 14 shows an embodiment in which the direction of travel is not determined, and the display reference is always fixed to the front direction at the start of shooting.

The process shown in FIG. 14 begins at step S200 in response to the spherical camera 110 receiving a request for corrected video data.

In step S201, the omnidirectional camera 110 acquires frame synthesis data and rotation parameters. As with the embodiment shown in FIG. 10, for the sake of convenience, it is assumed that the recording of the first frame data and the second frame data, the generation of the frame synthesis data by image synthesis, and the recording of the angular velocity data and acceleration data are completed at the start of the rotation process.

In step S202, the omnidirectional camera 110 stores the initial value of the shooting front direction of the omnidirectional camera 110. In step S203, the omnidirectional camera 110 calculates the tilt angle with respect to the direction of gravity in the frame to be processed based on at least the acceleration data using the zenith correction amount calculation unit 222, and writes the calculated tilt angle in the zenith correction data.

In step S204, the omnidirectional camera 110 causes the first rotation amount calculation unit 224 to integrate the amount of change in angle in the coordinate system of the horizontal plane for each clock, and detects the amount of change in the rotation angle in the horizontal plane compared to the previous frame. In step S205, the omnidirectional camera 110 causes the first rotation amount calculation unit 224 to integrate the amount of change in angle between frames calculated in step S204, calculate the amount of change in angle in the shooting front direction from the initial value, and writes the first rotation correction data.

In step S206, the omnidirectional camera 110 performs zenith correction on the target frame of the frame synthesis data using the image rotation unit 230 based on the zenith correction data (the tilt angle of the target frame therein).

In step S207, the omnidirectional camera 110 performs rotation correction on the target frame using the image rotation unit 230 based on the first rotation correction data. As described above, the first rotation correction data is the difference between the initial value of the front direction of shooting of the reference frame in the horizontal plane and the front direction of shooting at the time of the target frame, and directly represents the amount of rotation change in the horizontal plane that should be canceled out in the target frame.

In step S208, the spherical camera 110 determines whether or not there is a next frame. If it is determined in step S208 that there is a next frame (YES), the process loops to step S203. On the other hand, if it is determined in step S208 that there is no next frame (NO), the process ends in step S209.

The above process applies correction to the video to cancel out rotational changes around the reference axis so that the display reference roughly matches the front direction of the video at the start of shooting across multiple frames from the start to the end of shooting.

As described above, according to this embodiment, it is possible to provide an image processing device, an imaging device, a video playback system, a method, and a program that can suppress motion sickness in a viewer caused by rotation around a reference axis during imaging when watching a video.

In certain embodiments, the standard of the display during viewing is fixed to the direction of travel of the person filming, which helps to prevent viewers from feeling nauseous and allows them to experience a sense of realism close to that of the viewer seeing the scene.

In the above embodiment, the spherical camera 110 is fixed to the head 50 of the photographer. The rotation correction function according to this embodiment is preferably applicable to such a case, but the application scenes are not particularly limited, and it goes without saying that it can also be applied to cases where the user takes pictures by holding the camera in their hands.

In the above embodiment, the first rotation correction data and the second rotation correction data are calculated using the angular velocity sensor 136A and the acceleration sensor 136B. However, in other embodiments, when calculating the angle change amount, the first rotation correction data and the second rotation correction data may be calculated using a geomagnetic sensor. In that case, the initial value of the geomagnetic sensor is stored, and the rotation angle of the coordinate system of the horizontal plane is calculated based on the difference between the initial value of the geomagnetic sensor and the direction indicated by the geomagnetic sensor in the target frame, and the first rotation correction data and the second rotation correction data can be calculated. In the above description, the frame synthesis data, the acceleration data, and the angular velocity data are stored in advance, but this rotation correction function can be applied not only to stored video data, but also to a stream of video data distributed in real time.

Furthermore, in the above-mentioned embodiment, zenith correction is performed on video in a spherical coordinate system including two angular coordinates. However, depending on the embodiment, there may be cases where the tilt in the direction of gravity is not large enough to require zenith correction, and in such cases, zenith correction can be omitted. In this case, the reference axis may be the axis of the angular coordinate if the coordinate system is a cylindrical coordinate system including one angular coordinate, or may be the axis that gives the two angular coordinates if the coordinate system is a spherical coordinate system including two angular coordinates.

The above-mentioned functional units can be realized by a computer-executable program written in a legacy programming language such as assembler, C, C++, C#, Java (registered trademark), or an object-oriented programming language, and can be stored on a device-readable recording medium such as a ROM, EEPROM, EPROM, flash memory, flexible disk, CD-ROM, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, Blu-ray disk, SD card, or MO, or distributed via a telecommunications line. In addition, some or all of the above functional units can be implemented on a programmable device (PD) such as a field programmable gate array (FPGA), or can be implemented as an ASIC (application specific integrated circuit), and can be distributed on a recording medium as circuit configuration data (bitstream data) to be downloaded to the PD to realize the above functional units on the PD, or data written in HDL (Hardware Description Language), VHDL (Very High Speed Integrated Circuits Hardware Description Language), Verilog-HDL, or the like to generate the circuit configuration data.

Although the embodiments of the present invention have been described above, the embodiments of the present invention are not limited to the above-mentioned embodiments, and can be modified within the scope of what a person skilled in the art can imagine, including other embodiments, additions, modifications, deletions, etc., and any aspect is within the scope of the present invention as long as it provides the functions and effects of the present invention.

12...imaging body, 14...housing, 18...shutter button, 20...imaging optical system, 22, 130...imaging element, 110...spherical camera, 112, 152...CPU, 114...ROM, 116...image processing block, 118...video compression block, 120, 126...interface, 122...external storage interface, 124...external sensor interface, 126...USB interface, 128...serial block, 132...DRAM, 134, 160...external storage, 136A...angular velocity sensor, 136B...acceleration sensor 138, 166...USB connector, 150...information terminal, 154...RAM, 156...internal storage, 158...input device, 162...display, 164...wireless NIC, 200...functional block, 212...first imaging unit, 214...second imaging unit, 216...angular velocity detection unit, 218...acceleration detection unit, 220...image synthesis unit, 222...zenith correction amount calculation unit, 224...first rotation amount calculation unit, 226...second rotation amount calculation unit, 228...storage unit, 230...image rotation unit, 240...output unit, 242...instruction reception unit, 252...reception unit, 254...display control unit

JP 2017-147682 A

Claims (15)

An image processing device that processes a moving image including successive images represented in a coordinate system including at least an angular coordinate around a predetermined axis,
A video data acquisition means for acquiring video data;
a sensor data acquisition means for acquiring sensor data corresponding to the video data , the sensor data including first rotation correction data and second rotation correction data ;
a rotation correction means for correcting the image based on the first rotation correction data and the second rotation correction data so that a reference for displaying the image is fixed to a specific direction during shooting across a plurality of frames of the moving image, thereby canceling out a rotation change around a reference axis;
Including,
the sensor data corresponding to the moving image includes data output by an acceleration sensor and data output by an angular velocity sensor;
the first rotation correction data is time series data of a first angle change amount in a front direction of the image with respect to a reference value;
the second rotation correction data is time series data of a second angle change amount in a traveling direction relative to a reference value,
a direction in a plane perpendicular to the reference axis based on a posture of a camera capturing the video in a reference frame at a predetermined time point is determined as the specific direction during the shooting, and the amount of the rotational change is based on a difference between a direction based on the posture in the reference frame and a direction in a plane perpendicular to the reference axis based on the posture of the camera in a correction target frame;
Image processing device. The image processing device according to claim 1, wherein the predetermined time point is a start time point of the video recording or a time point when an instruction is received during the video recording. The image processing device according to claim 1 or 2, wherein the certain period corresponding to the plurality of frames is a period from a first time point when the start of the video or the start of the fixation is instructed to a second time point when the end of the video or the end of the fixation is instructed. The image processing device according to claim 1 or 2, further comprising a determination means for determining the direction of travel in which the camera is moving in a plane perpendicular to the reference axis as the specific direction during the image capture, and when the direction of travel is determined as the specific direction, the amount of the rotational change is based on the difference between the direction of travel and a direction in the plane perpendicular to the reference axis based on the attitude of the camera in the frame to be corrected. The image processing device according to claim 4, characterized in that the determining means determines the re-calculated traveling direction as the specific direction during shooting when the movement distance in a specified number of frames based on the sensor data satisfies a predetermined condition, and the fixed period corresponding to the plurality of frames corresponding to the determined specific direction during shooting is the period from the determination to the next determination. The coordinate system further includes an angle coordinate relative to the predetermined axis, and the image processing device
The image processing device according to any one of claims 1 to 5, further comprising: a zenith correction means for performing zenith correction on each frame of the video so that the predetermined axis approximately coincides with the reference axis, based on a tilt angle at the time of imaging in each frame of the video with respect to the reference axis. The image processing device according to any one of claims 1 to 6, wherein the video includes, as each frame, a celestial sphere image represented by a coordinate system including two angular coordinates, the video is captured by a plurality of image sensors, the celestial sphere image is formed by synthesizing a plurality of partial images captured by the plurality of image sensors, the video data includes, as each frame, the celestial sphere image or the plurality of partial images, the correction is applied to the celestial sphere image or the plurality of partial images, and the reference axis is the direction of gravity. The image processing device according to any one of claims 1 to 7, wherein the sensor data corresponding to the video includes at least one of data output by an acceleration sensor, data output by an angular velocity sensor, data output by a geomagnetic sensor, and rotation correction data calculated based on the data output by these sensors. An imaging device that captures a video including successive images expressed in a coordinate system including at least an angular coordinate around a predetermined axis,
An imaging means for capturing each frame of a moving image;
a recording means for recording sensor data indicating a displacement amount of the imaging device in an absolute coordinate system , the sensor data including first rotation correction data and second rotation correction data ;
a rotation correction means for correcting the image to cancel a rotation change around a reference axis so that a reference for displaying the image is fixed to a specific direction during shooting across a plurality of frames of the moving image based on the first rotation correction data and the second rotation correction data,
the sensor data corresponding to the moving image includes data output by an acceleration sensor and data output by an angular velocity sensor;
the first rotation correction data is time series data of a first angle change amount in a front direction of the image with respect to a reference value;
the second rotation correction data is time series data of a second angle change amount in a traveling direction relative to a reference value,
a direction in a plane perpendicular to the reference axis based on a posture of a camera capturing the video in a reference frame at a predetermined time point is determined as the specific direction during the shooting, and the amount of the rotational change is based on a difference between a direction based on the posture in the reference frame and a direction in a plane perpendicular to the reference axis based on the posture of the camera in a correction target frame;
Imaging device. The imaging device according to claim 9, further comprising a receiving means for receiving an instruction to specify the specific direction while the video is being captured. A video playback system for playing a video including continuous images represented in a coordinate system including at least an angular coordinate around a predetermined axis, comprising:
A video data acquisition means for acquiring video data;
a sensor data acquisition means for acquiring sensor data corresponding to the video data , the sensor data including first rotation correction data and second rotation correction data ;
a rotation correction means for correcting the image based on the first rotation correction data and the second rotation correction data so that a reference for displaying the image is fixed to a specific direction during shooting across a plurality of frames of the moving image, thereby canceling out a rotation change around a reference axis;
a display control means for causing a display means to display at least a part of the image of the moving image that has been corrected based on the display criteria;
the sensor data corresponding to the moving image includes data output by an acceleration sensor and data output by an angular velocity sensor;
the first rotation correction data is time series data of a first angle change amount in a front direction of the image with respect to a reference value;
the second rotation correction data is time series data of a second angle change amount in a traveling direction relative to a reference value,
a direction in a plane perpendicular to the reference axis based on a posture of a camera capturing the video in a reference frame at a predetermined time point is determined as the specific direction during the shooting, and the amount of the rotational change is based on a difference between a direction based on the posture in the reference frame and a direction in a plane perpendicular to the reference axis based on the posture of the camera in a correction target frame;
Video playback system. The video playback system of claim 11 further includes a rotation amount calculation means for calculating a specific direction during shooting in a plane coordinate system perpendicular to the reference axis. The video data acquisition means includes an imaging means for capturing images of each frame of the video, and the sensor data acquisition means includes a recording means for recording output data of a sensor provided together with the imaging means or correction data calculated from the output data in association with each frame of the video, or
the video data acquisition means and the sensor data acquisition means include a receiving means for receiving output data of a sensor provided together with the imaging means or correction data calculated from the output data, together with each frame of the video captured by the imaging means;
13. The video playback system according to claim 11 or 12. 1. A method for processing a video including a sequence of images represented in a coordinate system including at least angular coordinates about a given axis, the method comprising:
acquiring video data;
acquiring sensor data corresponding to the video data , the sensor data including first rotation correction data and second rotation correction data ;
and performing a correction on the image based on the first rotation correction data and the second rotation correction data to cancel out a rotation change around a reference axis so that a reference for displaying the image is fixed to a specific direction during shooting across a plurality of frames of the moving image,
the sensor data corresponding to the moving image includes data output by an acceleration sensor and data output by an angular velocity sensor;
the first rotation correction data is time series data of a first angle change amount in a front direction of the image with respect to a reference value;
the second rotation correction data is time series data of a second angle change amount in a traveling direction relative to a reference value,
a direction in a plane perpendicular to the reference axis based on a posture of a camera capturing the video in a reference frame at a predetermined time point is determined as the specific direction during the shooting, and the amount of the rotational change is based on a difference between a direction based on the posture in the reference frame and a direction in a plane perpendicular to the reference axis based on the posture of the camera in a correction target frame;
Method. A program for implementing an image processing device that processes a moving image including successive images expressed in a coordinate system including at least an angular coordinate around a predetermined axis, the program comprising:
A video data acquisition means for acquiring video data;
a sensor data acquisition means for acquiring sensor data corresponding to the video data , the sensor data including first rotation correction data and second rotation correction data ; and a rotation correction means for applying correction to the image to cancel out a rotation change around a reference axis, based on the first rotation correction data and the second rotation correction data, so that a reference for displaying the image is fixed to a specific direction during shooting across a plurality of frames of the video.
Function as a
the sensor data corresponding to the moving image includes data output by an acceleration sensor and data output by an angular velocity sensor;
the first rotation correction data is time series data of a first angle change amount in a front direction of the image with respect to a reference value;
the second rotation correction data is time series data of a second angle change amount in a traveling direction relative to a reference value,
a direction in a plane perpendicular to the reference axis based on a posture of a camera capturing the video in a reference frame at a predetermined time point is determined as the specific direction during the shooting, and the amount of the rotational change is based on a difference between a direction based on the posture in the reference frame and a direction in a plane perpendicular to the reference axis based on the posture of the camera in a correction target frame;
program.