JP7516755B2 - Imaging device, method and program - Google Patents

Description

This disclosure relates to an imaging device, a method, and a program.

Conventionally, imaging devices have been known that have multiple imaging units, each including a lens optical system and an image sensor, and generate a single image by synthesizing multiple images captured simultaneously using the multiple imaging units. One such imaging device is an omnidirectional imaging device that can capture 360° images in all directions with a single shooting command.

The above-mentioned imaging device uses multiple imaging units to capture a wide field of view at once, so there are cases where a specific subject, such as the photographer, is captured within the capture range. This is suitable when you want to capture the surrounding atmosphere, including the photographer himself. However, there are cases where you do not want the photographer to be the subject, and conventional technology has not been able to meet such demands.

In relation to the synthesis of images using multiple imaging units, Japanese Patent No. 6065474 (Patent Document 1) is known. The conventional technology of Patent Document 1 aims to reduce discontinuity that can occur at seams when multiple images captured by multiple imaging optical systems are synthesized. Patent Document 1 discloses an imaging control device, imaging control method, and program that can provide imaging conditions for each of multiple imaging optical systems. However, the technology of Patent Document 1 cannot deal with the problem of the photographer himself appearing as a subject in the captured image.

The present disclosure has been made in consideration of the above points, and aims to provide an imaging device that can prevent a specific subject from being captured when capturing an image using multiple imaging units together.

In order to solve the above-mentioned problems, the present disclosure provides an imaging device having the following characteristics: The imaging device includes a plurality of imaging sections each including an optical system and an imaging element, a control section that causes a first imaging section of the plurality of imaging sections to perform a first imaging process and then causes a second imaging section of the plurality of imaging sections to perform a second imaging process, and a generation section that generates a composite image based on a first image captured by the first imaging process and a second image captured by the second imaging process, and the control section executes the second imaging process in the second imaging section based on the elapsed time since imaging in the first imaging section.

The above configuration makes it possible to prevent a specific subject from being captured when shooting using multiple imaging units together.

FIG. 1 is a cross-sectional view of an omnidirectional imaging device according to an embodiment of the present invention. FIG. 2 is a hardware configuration diagram of the omnidirectional imaging device according to the present embodiment. FIG. 2 is a block diagram of main functions related to an omnidirectional imaging function realized on the omnidirectional imaging device according to the present embodiment. FIG. 1A is an image data flow diagram for generating a spherical image, and FIG. 1B is a diagram for explaining the data structure of the spherical image when it is represented on a plane and when it is represented on a spherical surface. 1A is a diagram for explaining a positional relationship between the omnidirectional imaging device according to the present embodiment and a subject including a photographer, (B) two captured fisheye images, and (C) a composite omnidirectional image. FIG. 4 is a flowchart showing a celestial sphere imaging process executed by the celestial sphere imaging device according to the present embodiment. 1A and 1B are diagrams illustrating fisheye images that can be captured in a first imaging process and a second imaging process in a time difference exposure mode according to the present embodiment, and a composite image that is generated by combining the fisheye images; 13 is a flowchart showing a celestial sphere imaging process executed by an octopus imaging device according to another embodiment. 13 is a flowchart showing a celestial sphere imaging process executed by an omnidirectional imaging device according to still another embodiment. 1A and 1B show fisheye images obtained at different times in a particular embodiment for detecting the disappearance of a predetermined object. FIG. 13 shows fisheye images obtained at different times in another particular embodiment for detecting the appearance of a predetermined subject.

The following describes this embodiment, but the embodiment is not limited to the embodiment described below. Note that in the following embodiment, the omnidirectional imaging device 110 is used as an example of an imaging device, but the embodiment is not limited to this.

The basic configuration of the omnidirectional imaging device 110 according to this embodiment will be described below with reference to Figures 1 and 2.

Figure 1 is a cross-sectional view of an omnidirectional imaging device 110 according to this embodiment. The omnidirectional imaging device 110 shown in Figure 1 includes an imaging body 12, a housing 14 that holds the imaging body 12 and components such as a controller and a battery, and a shutter button 18 provided on the housing 14.

The imaging body 12 shown in FIG. 1 is configured to include two lens barrel units each including an imaging optical system 20 and an imaging element 22. The imaging element 22 is a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor, and is controlled by a control command from a CPU described later. Each imaging optical system 20 is configured, for example, as a fisheye lens with 6 groups and 7 lenses. In the embodiment shown in FIG. 1, the fisheye lens has a total angle of view larger than 180 degrees (=360 degrees/n; number of optical systems n=2), preferably has an angle of view of 185 degrees or more, and more preferably has an angle of view of 190 degrees or more. A combination of one such wide-angle imaging optical system 20 and one imaging element 22 constitutes the imaging unit in this embodiment. In the embodiment described, a case in which the number of optical systems (imaging units) is two will be described as an example, but the number of optical systems (imaging units) may be three or more.

The optical elements (lenses, prisms, filters, and aperture stops) of the two imaging optical systems 20A and 20B are positioned relative to the image sensors 22A and 22B. More specifically, the optical elements of the imaging optical systems 20A and 20B are positioned so that their optical axes are perpendicular to the center of the light receiving area of the corresponding image sensor 22, and the light receiving area is positioned as the imaging plane of the corresponding fisheye lens.

In the embodiment shown in FIG. 1, the imaging optical systems 20A and 20B are of the same specification and are combined in the opposite directions so that their optical axes coincide. The image pickup elements 22A and 22B convert the received light distribution into an image signal and sequentially output image frames to the image processing means on the controller. As will be described in detail later, the images captured by the image pickup elements 22A and 22B are synthesized to generate an image with a solid angle of 4π steradians (hereinafter referred to as a "spherical image"). A spherical image is an image of all directions that can be seen from the shooting point. In the embodiment described here, a spherical image is generated, but it may be a spherical image or a so-called panoramic image in which only the horizontal plane is shot at 360 degrees.

Figure 2 shows the hardware configuration of the omnidirectional imaging device 110 according to this embodiment. The omnidirectional imaging device 110 corresponds to the imaging device in the embodiment to be described.

The omnidirectional imaging device 110 includes a CPU (Central Processing Unit) 112, a ROM (Read Only Memory) 114, an image processing block 116, a still image compression block 117, a video compression block 118, a resizing block 119, a dynamic random access memory (DRAM) 132 connected via a DRAM interface 120, and a motion sensor 136 connected via an external sensor interface 124.

The CPU 112 controls the operation of each part of the omnidirectional imaging device 110 and the overall operation. The ROM 114 stores a control program and various parameters written in a code that the CPU 112 can decode. When a rewritable flash ROM is used as the ROM 114, it is possible to change the control program and parameters for control, which in turn makes it easy to upgrade functions. In addition to or together with the ROM 114, a storage such as an SSD for storing the control program may be provided. The image processing block 116 is connected to the two image sensors 130A and 130B (the image sensors 22A and 22B in FIG. 1), and receives image signals of images captured by each of the image sensors. The image processing block 116 includes an ISP (Image Signal Processor) and performs shading correction, Bayer interpolation, white balance correction, gamma correction, and the like on the image signal input from the image sensor 130.

The still image compression block 117 is a codec block that performs still image compression and decompression using JPEG (Joint Photographic Experts Group) and other standards. The video compression block 118 is a codec block that performs video compression and decompression using MPEG (Moving Picture Experts Group)-4 AVC (Advanced Video Coding)/H.264 and other standards. Note that JPEG and H.264 are described here as examples of still image and video codecs, respectively, but the still image and video codecs are not particularly limited. There are many other variations in codecs, so other codecs may be used, or multiple codecs may be installed. The resizing block 119 is a block that enlarges or reduces the size of image data by interpolation processing.

DRAM 132 provides a storage area for temporarily storing data when various signal processing and image processing are performed. Examples of image data that can be stored include RAW-RGB image data after white balance and gamma settings have been performed in signal processing, YUV image data after luminance data and color difference data conversion has been performed in signal processing, and JPEG image data that has been JPEG compressed by still image compression block 117.

Motion sensor 136 is, for example, a sensor that detects three-axis acceleration components and three-axis angular velocity components. The detected acceleration components and angular velocity components are used to perform zenith correction of the omnidirectional image in the direction of gravity (reference direction) and rotation correction around the direction of gravity. Motion sensor 136 may further include other sensors such as a geomagnetic sensor for determining an azimuth angle, etc. Here, a three-axis sensor is used as motion sensor 136, but it may also be, for example, a six-axis acceleration sensor or angular velocity sensor.

The omnidirectional imaging device 110 further includes an external storage interface 122, a USB (Universal Serial Bus) interface 126, and a serial block 128. An external storage 134 is connected to the external storage interface 122. The external storage interface 122 controls reading and writing from and to the external storage 134, such as a memory card inserted into a memory card slot. A USB connector 138 is connected to the USB interface 126. The USB interface 126 controls USB communication with an external device, such as a smartphone, connected via the USB connector 138. The serial block 128 controls serial communication with an external device, such as a smartphone, and is connected to a wireless NIC (Network Interface Card) 140.

The omnidirectional imaging device 110 shown in FIG. 2 further includes a face detection block 144. The face detection block 144 performs face detection using, for example, a fisheye image or an image that has been zenith corrected, and identifies the position of a person's face. In a specific embodiment, the face detection block 144 can be used to detect the disappearance or appearance of a specific subject, such as a photographer, as will be described in detail below.

The omnidirectional imaging device 110 shown in FIG. 2 further includes an audio unit 146. A microphone 148 that inputs an audio signal and a speaker 150 that outputs an audio signal are connected to the audio unit 146. The audio unit 146 may generally include a microphone amplifier that amplifies the audio signal input via the microphone 148, an audio recording circuit that records the amplified audio signal, an audio playback circuit that converts the recorded audio signal into a signal that can be output from the speaker 150, and an audio amplifier that amplifies the converted audio signal and drives the speaker 150. The audio unit 146 operates under the control of the CPU 112.

The omnidirectional imaging device 110 shown in FIG. 2 further includes an LCD (Liquid Crystal Display) driver 152, to which an LCD monitor 154 is connected. The LCD driver 152 is a drive circuit that drives the LCD monitor 154, and has a function of converting various states into signals for displaying on the LCD monitor 154. An OLED (Organic Light Emitting Diode) display may be mounted instead of the LCD monitor 154.

The above-mentioned components 112 to 128, 144, 146, and 152 are connected to each other via a bus 160. When the power switch is operated to turn on the power, a control program is loaded into the main memory (DRAM 132 or other RAM). The CPU 112 controls the operation of each part of the device according to the program loaded into the main memory, and temporarily stores data required for control in the memory. This realizes each of the functional parts and processes of the omnidirectional imaging device 110, which will be described later. The control program may include firmware, an operating system (OS), plug-in applications, and the like.

The omnidirectional imaging function of the omnidirectional imaging device 110 according to this embodiment will be described below with reference to Figures 3 to 7.

FIG. 3 shows a main functional block 200 related to the omnidirectional imaging function realized on the omnidirectional imaging device 110 according to this embodiment. As shown in FIG. 3, the functional block 200 of the omnidirectional imaging device 110 includes a control unit 210, multiple imaging units, multiple image signal processing units, and a distortion correction/image synthesis unit 240. In the embodiment shown in FIG. 3, as described with reference to FIG. 1, the number of optical systems (imaging units) is two, and a first imaging unit 220, a second imaging unit 230, a first image signal processing unit 222, and a second image signal processing unit 232 are provided. Even if the number of optical systems (imaging units) is three or more, it is sufficient to provide imaging units and image signal processing units according to the number of optical systems.

The control unit 210 includes the CPU 112 shown in FIG. 2. The control unit 210 controls the imaging units 220 and 230, and performs overall control of the imaging operation of the omnidirectional image.

The first imaging unit 220 and the second imaging unit 230 each include the imaging optical system 20 and the imaging element 22 (130) shown in FIG. 1 and FIG. 2. The imaging elements 22 (130) of the imaging units 220 and 230 are controlled by control commands from the control unit 210, which includes the CPU 112. In normal spherical photography processing (simultaneous exposure mode), the imaging elements 22A and 22B (130A and 130B) of the two imaging units 220 and 230 simultaneously start exposure and capture data under the control of the control unit 210.

The first image signal processing unit 222 and the second image signal processing unit 232 perform white balance setting and gamma setting for the image data input from the first imaging unit 220 and the second imaging unit 230, respectively. The first image signal processing unit 222 and the second image signal processing unit 232 further convert the image data into luminance data and color difference data by filtering processing, and output fisheye images 224, 234 in YUV format. The data of the fisheye images 224, 234 in YUV format may be written to the DRAM 132, or may be output to the external storage 134 via the external storage interface 122.

The distortion correction/image synthesis unit 240 is used to synthesize the fisheye images 224, 234 obtained from the multiple imaging units 220, 230, and more specifically, performs distortion correction processing and image synthesis processing.

Below, the generation of a celestial sphere image and the generated celestial sphere image will be described with reference to FIG. 4. FIG. 4(A) describes the data structure of each image and the data flow of the image in the generation of the celestial sphere image. First, the image directly captured by each image sensor 130 is an image that covers roughly a hemisphere of the celestial sphere in its field of view. Light incident on the imaging optical system is imaged on the light receiving area of the image sensor 130 according to a predetermined projection method. The image captured here is captured by a two-dimensional image sensor whose light receiving area forms a planar area, and becomes image data expressed in a planar coordinate system. Typically, the image obtained is configured as a fisheye image including the entire image circle on which each shooting range is projected, as shown by "fisheye image A" and "fisheye image B" in FIG. 4(A). Note that in the embodiment described, a fisheye image captured using a fisheye lens will be described, but this fisheye image also includes a wide-angle image captured using a wide-angle lens other than a fisheye lens.

The multiple fisheye images captured by the multiple image sensors 130 are subjected to distortion correction processing and image synthesis processing to compose one omnidirectional image. In the image synthesis processing, first, each omnidirectional image including each complementary hemispherical portion is generated from each fisheye image configured as a planar image. Then, the two omnidirectional images including each hemispherical portion are aligned based on matching of overlapping regions, and are synthesized to generate a omnidirectional image including the entire omnidirectional sphere.

Fig. 4(B) is a diagram illustrating the data structure of image data of a celestial sphere image used in this embodiment, expressed on a plane. Fig. 4(C) is a diagram illustrating the data structure of image data of a celestial sphere image, expressed on a spherical surface. As shown in Fig. 4(B), the image data of the celestial sphere image is expressed as an array of pixel values whose coordinates are a vertical angle φ made with respect to a predetermined axis and a horizontal angle θ corresponding to a rotation angle around the predetermined axis. The vertical angle φ is in the range of 0 degrees to 180 degrees (or -90 degrees to +90 degrees), and the horizontal angle θ is in the range of 0 degrees to 360 degrees (or -180 degrees to +180 degrees).

As shown in FIG. 4C, each coordinate value (θ, φ) in the omnidirectional format is associated with each point on a sphere representing all directions centered on the shooting point, and the all directions are associated on the omnidirectional image. The planar coordinates of the fisheye image captured by the fisheye lens and the coordinates on the spherical surface of the omnidirectional image are associated with each other in a predetermined conversion table. The conversion table is data created in advance by a manufacturer or the like according to a predetermined projection model based on the design data of each lens optical system, and is data for converting the coordinates of the fisheye image into a omnidirectional image while taking distortion into consideration. When performing distortion correction and image synthesis processing, the data may be temporarily stored in DRAM 132.

The distortion correction and image synthesis unit 240 performs distortion correction and image synthesis processing using information from the motion sensor 136, and can perform top-bottom correction at the same time to generate a top-bottom corrected synthetic image. Generally, top-bottom correction and distortion correction are each irreversible conversion processes, so although they are described here as being processed simultaneously, they may be processed separately. Top-bottom correction refers to a process of correcting an image captured in a state where the central axis (shown in FIG. 1) of the omnidirectional imaging device 110 is actually tilted with respect to a reference direction (e.g., the direction of gravity) to an image as if it were captured in a state where the central axis coincides with the direction of gravity Z. In a specific embodiment, in addition to the zenith correction, a rotation correction may be performed to cancel the angle change around the reference direction.

The distortion correction and image synthesis unit 240 performs distortion correction processing and image synthesis processing to output a YUV format omnidirectional image (equirectangular format) 242. As described above, the output destination of this image may be, for example, the DRAM 132 or the external storage 134. The distortion correction and image synthesis unit 240 constitutes a generation unit that generates a synthetic image based on multiple fisheye images captured by the multiple imaging units 220 and 230 in this embodiment.

Here, the distortion correction and image synthesis unit 240 will be described as outputting a YUV-format omnidirectional image 242 obtained by coordinate conversion as a synthetic image. However, the synthetic image does not necessarily need to be output as a synthetic omnidirectional image as shown in FIG. 4. As long as a omnidirectional image can be constructed at the time of viewing, it may be output in any form. For example, on the premise that a omnidirectional image is generated by performing distortion correction and synthesis from fisheye images at the time of viewing, a still image (YUV-format joint image (dual fisheye format) 244) in which still image data of two fisheye images directly captured by the image pickup elements 130A and 130B (still images corresponding to fisheye image A and fisheye image B shown in FIG. 4A) are arranged and joined to form one image may be output. In addition, the synthetic image does not need to be combined into one file, and the two fisheye images directly captured by the image pickup elements 130A and 130B may be associated and stored as separate files. Furthermore, the composite image is not limited to a spherical image, but may be a 360-degree panoramic image in the horizontal direction, or an image in which a portion of a spherical image or a panoramic image is missing.

The functional block 200 of the omnidirectional imaging device 110 may further include a codec processing unit 250 as shown in FIG. 3. The codec processing unit 250 includes the still image compression block 117 shown in FIG. 2. The YUV-format omnidirectional image 242 or the joined image 244 output from the distortion correction/image synthesis unit 240 is converted into a predetermined still image format by the codec processing unit 250 and output as a omnidirectional still image 254 or a joined still image 252 in a predetermined still image format (e.g., JPEG). The output destination may be the external storage 134, or may be in the storage of an external information terminal connected via the USB connector 138 or the wireless NIC 140. The still image may be recorded in any format as long as the still image can be played back. Examples of the still image format include PNG (Portable Network Graphics) and BMP (Bitmap) formats in addition to JPEG.

According to the above-mentioned omnidirectional imaging device 110, the imaging ranges of the two imaging units 220 and 230 are combined, so that imaging in all directions from the imaging point at the center can be performed at once.

Fig. 5(A) illustrates an example of the positional relationship between the omnidirectional imaging device 110 according to this embodiment and a subject including the photographer P. Fig. 5(B) illustrates two fisheye images (fisheye image A, fisheye image B) captured in the positional relationship shown in Fig. 5(A). Fig. 5(C) illustrates an omnidirectional image obtained by combining two fisheye images such as those shown in Fig. 5(B) captured in the positional relationship shown in Fig. 5(A).

In FIG. 5A, photographer P, subject O1, and subject O2 are arranged around omnidirectional imaging device 110. As shown in FIG. 5A, omnidirectional imaging device 110 is provided with optical system 20A (front) and optical system 20B (rear) at the front and back positions. As shown in FIG. 5B, fisheye images captured through the respective optical systems 20A and 20B include subjects on the respective sides (photographer P and subject O1 in fisheye image A, and subject O2 in fisheye image B). The entire subjects, including photographer P and subjects O1 and O2, are also included in the synthesized omnidirectional image.

Such a spherical image can be said to meet the desire to capture the surrounding atmosphere including the photographer P himself. On the other hand, since a wide field of view in all directions is captured at once, the photographer P will be captured within the combined captured range, but there are cases where it is not desirable for the photographer P himself to be captured as a subject.

Therefore, in the omnidirectional imaging device 110 according to this embodiment, when capturing images using multiple imaging units 220, 230 together, in addition to the simultaneous exposure mode, a capturing mode that can prevent a specific subject from being captured in the image (hereinafter referred to as a time-difference exposure mode) is provided. In the following, an example will be described in which the subject to be prevented from being captured in the image is the photographer, but it goes without saying that the subject can also be a subject other than the photographer.

In this embodiment, the control unit 210 controls the multiple image capture units 220, 230 in the time-difference exposure mode, so that each of the multiple image capture units 220, 230 captures an image at a different time. The control unit 210 also causes the distortion correction/image synthesis unit 240 to synthesize the multiple images captured at different times.

Even when capturing one fisheye image, there is a certain time span in the image capture process due to the exposure time, the exposure timing for each line in the case of a rolling shutter, and the lag in readout time. Here, "different times" in relation to image capture using multiple image capture units 220, 230 means that the timing of the image capture process (from the start of exposure to the completion of data capture) in each image capture unit 220, 230 is intentionally spaced apart to the extent that they do not overlap. If there is a time difference that may occur due to the accuracy of synchronization, it is not considered to be "different times". Preferably, the "different times" are sufficiently spaced apart to allow the subject to be moved from one image capture range to outside that image capture range.

In addition, "simultaneous" in the simultaneous exposure mode described above does not mean simultaneous to the extent that images are acquired at the same time for all pixels as in the case of a global shutter, but is considered to be "simultaneous" even if there is a time difference that can occur due to differences in exposure timing and readout time for each line in the case of a rolling shutter, or due to synchronization accuracy.

By performing the imaging operations of the multiple imaging units 220, 230 at different times, the photographer P can move outside the imaging range of the imaging unit that will not next capture images or the imaging range of the imaging unit that will next capture images after finishing imaging with one unit.

In a specific embodiment, the control unit 210 can time the elapsed time from the shooting with the first imaging unit 220, and execute the shooting with the second imaging unit 230 in response to the elapsed time reaching a predetermined specified time. Here, the description is given assuming that the shooting with the second imaging unit 230 is performed after the shooting with the first imaging unit 220, but it may be the other way around, and which shooting is performed first may be fixed or may be set by the user. Furthermore, the elapsed time from one of the above-mentioned shootings may be the elapsed time from any reference related to the shooting process. For example, the elapsed time may be the elapsed time based on the end of the shooting process, may be the elapsed time based on the start of the shooting process, or may be the elapsed time based on the completion of the image output from the image signal processing unit 222, 232 after the shooting process, and is not particularly limited.

As described above, in the time-difference exposure mode, imaging is performed by one imaging unit (e.g., 230) based on the time that has elapsed since imaging by the other imaging unit (e.g., 220). The photographer P will move outside the imaging range of the subsequent imaging unit (e.g., 230) within this predetermined time. For this reason, it is preferable for the photographer P to be aware of the elapsed time and remaining time, and to be able to anticipate the timing of the subsequent imaging.

FIG. 3 also illustrates a specific embodiment that is effective in such a case, and in such a specific embodiment, the functional block 200 of the omnidirectional imaging device 110 further includes a guidance unit 212 indicated by a dotted line. Under the control of the control unit 210, the guidance unit 212 can provide guidance regarding subsequent imaging based on the above-mentioned elapsed time by audio or light notification, or by display on a display device. Here, examples of guidance provided regarding imaging include guidance indicating that timing has started, guidance regarding the elapsed time, and guidance regarding the remaining time.

Examples of audio notifications include a beep from the speaker 150 indicating that timing has started, reading out the elapsed time or remaining time, and outputting a beep with a period or tone corresponding to the elapsed time or remaining time. Examples of optical notifications include lighting an LED indicating that timing has started, an electric light display of the elapsed time or remaining time, lighting or blinking an LED with a period, color, or brightness corresponding to the elapsed time or remaining time, and the like. Examples of displays on a display device include a display of the elapsed time or remaining time on the LCD monitor 154 provided in the omnidirectional imaging device 110, or on an application screen of an external information terminal connected via USB, Wifi (registered trademark), Bluetooth (registered trademark), or other wired or wireless protocol. However, the manner of guidance by the guidance unit 212 is not particularly limited.

The omnidirectional imaging process according to this embodiment will be described in more detail below with reference to FIG. 6. FIG. 6 is a flowchart showing the omnidirectional imaging process executed by the omnidirectional imaging device 110 according to this embodiment. The process shown in FIG. 6 is mainly executed by the control unit 210.

The control shown in FIG. 6 starts from step S100 in response to a shooting request from the photographer. In step S101, the control unit 210 acquires the currently selected shooting mode. The shooting mode is not particularly limited, but here it is assumed that the shooting mode is selected from two options: "simultaneous exposure mode" and "time-difference exposure mode" which specifies that shooting is performed at different times by multiple imaging units. However, this does not prevent the existence of modes other than these.

In step S102, the control unit 210 determines whether the currently selected shooting mode is the simultaneous exposure mode. If it is determined in step S102 that the shooting mode is the simultaneous exposure mode (YES), the process branches to step S103.

In step S103, the control unit 210 causes the imaging units 220, 230 and the image signal processing units 222, 232 to simultaneously execute the first imaging process and the second imaging process, and proceeds to step S107. Here, the first imaging process corresponds to the period from the start of exposure of the imaging element 130 of the first imaging unit 220 in FIG. 3 to the output of a fisheye image in YUV format by the first image signal processing unit 222. In contrast, the second imaging process corresponds to the period from the start of exposure of the imaging element 130 of the second imaging unit 230 in FIG. 3 to the output of a fisheye image in YUV format by the second image signal processing unit 232.

On the other hand, if it is determined in step S102 that the simultaneous exposure mode is not selected (determined to be the "time-difference exposure mode") (NO), the process branches to step S104.

In step S104, the control unit 210 first causes the first imaging unit 220 and the first image signal processing unit 222 to perform the first imaging process to capture fisheye image A, and starts measuring the elapsed time thereafter. In step S105, the control unit 210 refers to the elapsed time and determines whether the next imaging start time has arrived. In step S105, step S105 is looped until the next imaging start time arrives (while NO is returned). If it is determined in step S105 that the next imaging start time has arrived (YES), the process proceeds to step S106. In step S106, the control unit 210 causes the second imaging unit 230 and the second image signal processing unit 232 to perform the second imaging process to capture fisheye image B at a time different from that in step S104.

For example, if the timing to start measuring the elapsed time is set to the end of the first imaging process and the specified time is set to 10 seconds, the second imaging process will start 10 seconds after the end of the first imaging process. As described above, it is also possible to set the timing to start measuring the specified time to the start of the first imaging process or to the completion of outputting the YUV image related to the first imaging process. The specified time may also be set to any time by user selection or input.

The photographer P (predetermined subject) who does not want to be captured in the spherical image is moved outside the capture range in the subsequent second capture process within the specified time period described above. Alternatively, the subject who does not want to be captured in the spherical image is moved outside the capture range in the subsequent second capture process within the specified time period described above.

Figure 7 (A) shows two fisheye images that can be captured in the first imaging process in the time-difference exposure mode according to this embodiment. Figure 7 (B) shows two fisheye images that can be captured in the second imaging process in the time-difference exposure mode according to this embodiment.

If the first imaging process is the process on the front side (20A) shown in FIG. 5(A), the photographer P who does not want to be captured in the omnidirectional image needs to be on the second imaging process side (rear side, fisheye image B side) as shown in FIG. 7(A) at the first time point of the first imaging process. Then, within a specified time until the second time point at which the subsequent second imaging process is performed, the photographer P needs to move to the first imaging process side (front side, fisheye image A side) as shown in FIG. 7(B). Note that although FIGS. 7(A) and (B) show a case in which a fisheye image on the opposite side to the side where the main imaging process is performed is also captured at the same time, it is not necessary that a fisheye image is captured on the opposite side to the side where the main imaging process is performed.

In the embodiment described above, the first imaging process is performed on the front side and the second imaging process is performed on the rear side, but the relationship between the first imaging process and the second imaging process may be reversed. In other words, when the photographer P is on the front side, the first imaging process may be performed on the rear side, and after the photographer P moves to the rear side, the second imaging process may be performed on the front side.

In addition, in the loop during which the determination in step S105 shown in FIG. 6 is NO, guidance can be provided by emitting a measurement start sound from the speaker when time measurement starts, outputting audio from the speaker according to the elapsed time or remaining time, turning on or blinking the LEDs provided in the omnidirectional imaging device 110, and displaying the elapsed time or remaining time on the LCD monitor 154. This allows the photographer to easily understand the amount of time available for movement.

In step S107, the control unit 210 causes the distortion correction/image synthesis unit 240 to execute synthesis processing based on the fisheye image A (first time point) and the fisheye image B (second time point) obtained in the first and second imaging processes to generate a synthetic image, and ends this processing in step S108.

By performing the first and second imaging processes in steps S104 to S106, a plurality of YUV-format fisheye images 224, 234 are obtained. Then, in step S107, distortion correction and image synthesis are performed on the two YUV-format fisheye images 224, 234 to generate a YUV-format omnidirectional image 242 or a combined image 244. Note that it is possible to arbitrarily determine whether to use the YUV-format omnidirectional image 242 or the YUV-format combined image 244. FIG. 7C shows a combined image 244, which is a composite image of the two fisheye images 224, 234 captured at different times in FIG. 7A and FIG. 7B. As shown in FIG. 7C, it can be understood that the photographer P is not included as a subject in the combined image 244.

In the above-described embodiment, imaging is performed by one imaging unit based on the time that has elapsed since imaging by the other imaging unit. On the other hand, in another embodiment, after imaging by one imaging unit, imaging can also be performed by the subsequent imaging unit in response to receiving an explicit instruction that triggers imaging by the subsequent imaging unit. FIG. 3 also shows such a specific embodiment. In this other embodiment, the functional block 200 of the omnidirectional imaging device 110 further includes an instruction receiving unit 214, as shown in FIG. 3, which is indicated by a dashed line.

After capturing an image with one of the imaging units (e.g., 220), the instruction receiving unit 214 waits for an explicit instruction from the photographer P to trigger capturing an image with the other imaging unit (e.g., 230). This explicit instruction is given using a remote controller (which may be a dedicated remote controller or a general-purpose terminal such as a smartphone with a control application installed) connected to the omnidirectional imaging device 110 by wire or wirelessly (Wifi (registered trademark) or Bluetooth (registered trademark)). When connected to a general-purpose terminal, it is also possible to actually show the photographer preview images of the first and second imaging processes. For this reason, for example, the photographer can explicitly instruct the start of the next capture when he or she has moved to a position where he or she is not captured as a subject while looking at the preview image.

In this other embodiment, instead of the control according to the elapsed time described above, the control unit 210 can perform control to execute imaging in the subsequent imaging unit (e.g., 230) in response to receiving an explicit instruction. The user may be able to select whether to perform control according to the elapsed time or control triggered by an explicit instruction from the photographer. The instruction may be an instruction that means that imaging can be performed in the other imaging unit (e.g., 230) (such as pressing a button when preparation is complete), or an instruction that means that imaging cannot be performed in the other imaging unit (e.g., 230) (such as holding down a button during preparation and releasing the button when preparation is complete).

The omnidirectional imaging process according to another embodiment will be described in more detail below with reference to FIG. 8. FIG. 8 is a flowchart showing the omnidirectional imaging process executed by the omnidirectional imaging device 110 according to another embodiment. The process shown in FIG. 8 is a process that is mainly executed by the control unit 210 including the CPU 112, similar to the embodiment shown in FIG. 6.

The control shown in FIG. 8 starts from step S200 in response to a shooting request from the photographer. The processing of steps S201 to S203 is similar to the processing of steps S101 to S103 shown in FIG. 6, so a description thereof will be omitted.

If it is determined in step S202 that the simultaneous exposure mode is not selected (NO), the process branches to step S204. In step S204, the control unit 210 first causes the first imaging unit 220 and the first image signal processing unit 222 to perform the first imaging process, and then waits for the next shooting start instruction in step S205. The photographer P (predetermined subject) who does not want to be captured in the spherical image will move out of the shooting range in the subsequent second shooting process before the next shooting start instruction is issued. If it is determined in step S205 that the control unit 210 has received the next shooting start instruction (YES), the process proceeds to step S206. In step S206, the control unit 210 causes the second imaging unit 230 and the second image signal processing unit 232 to perform the second imaging process.

In step S207, the control unit 210 causes the distortion correction/image synthesis unit 240 to execute synthesis processing based on the multiple fisheye images obtained in the first imaging process and the second imaging process to generate a synthetic image, and ends this process in step S208.

In the above embodiment, the two imaging units 220 and 230 are exposed to light and data is captured at different times, and two fisheye images corresponding to the two imaging units 220 and 230 are captured. However, the two imaging units 220 and 230 may be exposed to light and captured sequentially in synchronization with each other until processing by the distortion correction and image synthesis unit 240, and two fisheye images captured at different times may be acquired for synthesis by the distortion correction and image synthesis unit 240. In this case, images other than the selected one are discarded. In other words, capturing images at different times includes not only a case where the two imaging units 220 and 230 are not synchronized but are exposed to light and captured at different times, but also a case where the two imaging units 220 and 230 are synchronized to constantly expose and capture images, and then two fisheye images exposed to light and captured at different times are selected.

Hereinafter, with reference to Fig. 3 and Figs. 9 to 11, a further embodiment in which the omnidirectional imaging device 110 itself determines the timing to start the next image capture will be described. Fig. 3 also illustrates an embodiment in which the omnidirectional imaging device 110 itself determines the timing to start the next image capture. In this embodiment, as shown in Fig. 3, the control unit 210 of the omnidirectional imaging device 110 further includes a subject detection unit 216 indicated by the dashed line.

In a specific embodiment, the subject detection unit 216 captures (exposes and captures) images in sequence in one imaging unit (e.g., 220) after capturing an image in the other imaging unit (e.g., 230). The subject detection unit 216 is configured to detect the disappearance of a specific subject in the captured image based on the image. The subject detection unit 216 constitutes a disappearance detection unit in a specific embodiment. In response to detecting the disappearance of a specific subject in an image captured by the other imaging unit (e.g., 230), the control unit 210 completes capturing images in the other imaging unit (e.g., 230). Here, after detecting the disappearance of the specific subject, the other imaging unit (e.g., 230) may capture an image again, or the image used to detect the disappearance of the specific subject may be selected as the image to be used for compositing.

The disappearance of a specified subject can be determined, for example, by using the detection results from the fisheye image in face detection block 144. For example, a simple method is to consider a decrease in the number of faces in the fisheye image on the detection target side as the disappearance of the specified subject. Alternatively, a face to be detected can be registered in advance, and the disappearance of this face to be detected can be considered as the disappearance of the specified subject. Alternatively, without using such face recognition technology, since brightness tends to increase when a subject disappears, it can also be considered as the disappearance of a specified subject when the overall average brightness within the target imaging processing range changes by a certain value or more.

In another specific embodiment, the subject detection unit 216 continues to sequentially acquire (expose and capture) images with one of the imaging units (e.g., 220) after capturing an image with this imaging unit (e.g., 220). The subject detection unit 216 is configured to detect the appearance of a specific subject in an image captured by this one of the imaging units (e.g., 220) based on the image. The subject detection unit 216 constitutes an appearance detection unit in another specific embodiment. For example, the photographer's face is registered as a specific subject, and the control unit 210 executes shooting with the other imaging unit (e.g., 230) in response to detecting its appearance in an image captured by one of the imaging units (e.g., 220).

Fig. 9 is a flowchart showing omnidirectional imaging processing executed by the omnidirectional imaging device 110 according to yet another embodiment. The processing shown in Fig. 9 is processing that is mainly executed by the control unit 210 including the CPU 112. Note that the processing shown in Fig. 9 is a flowchart common to both the case where the disappearance of a specific subject is detected and the case where the appearance of a specific subject is detected.

The control shown in FIG. 9 starts from step S300 in response to a shooting request from the photographer. The processing of steps S301 to S303 is similar to the processing of steps S101 to S103 shown in FIG. 6, so a description thereof will be omitted.

If it is determined in step S302 that the simultaneous exposure mode is not set (NO), the process branches to step S304. In step S304, the control unit 210 first causes the first imaging unit 220 and the first image signal processing unit 222 to perform the first imaging process. In step S305, the control unit 210 executes a process for detecting the disappearance or appearance of a subject, and in step S306, it determines whether the disappearance or appearance of a specific subject has been detected. If it is determined in step S306 that the control unit 210 has not detected the disappearance or appearance of a specific subject (NO), the process loops to step S305 and continues the process for detecting the disappearance or appearance of a subject. If it is determined in step S306 that the control unit 210 has detected the disappearance or appearance of a specific subject (YES), the process proceeds to step S307. In step S307, the control unit 210 causes the second imaging unit 230 and the second image signal processing unit 232 to perform the second imaging process.

Figure 10 shows fisheye images obtained at each time point in a specific embodiment for detecting the disappearance of a specific subject. As shown in Figure 10 (A), at the first time point when the first imaging process on the front side is performed, the photographer P is on the second imaging process side (rear side, fisheye image B side) where imaging is not performed. The face F of the photographer P is detected in the fisheye image B by the subject detection unit 216. At a predetermined time point after the first imaging process on the front side is performed, as shown in Figure 10 (B), the disappearance L of the face of the photographer P detected in the fisheye image B is detected by the subject detection unit 216. Then, in response to this, at the second time point, as shown in Figure 10 (C), the second imaging process is performed to capture the rear side. In this case, the photographer P is, for example, on the first imaging process side (front side, fisheye image A side) where imaging is not performed.

In contrast, FIG. 11 is a diagram showing fisheye images obtained at each time point in another specific embodiment for detecting the appearance of a predetermined subject. As shown in FIG. 11(A), at the first time point when the first imaging process on the front side is performed, the photographer P is on the second imaging process side (rear side, fisheye image B side) where no imaging is performed. The face of the photographer P is not detected in the fisheye image A. At a predetermined time point after the first imaging process on the front side is performed, the appearance of the photographer P's face is detected in the fisheye image A by the subject detection unit 216, as shown in FIG. 11(B). In response to this, at the second time point, the second imaging process on the rear side is performed, as shown in FIG. 11(C). In this case, the photographer P is on the first imaging process side (front side, fisheye image A side) where no imaging is performed, but the face of the photographer P is detected in the fisheye image A.

Referring again to FIG. 9, after steps S307 and S303, the process proceeds to step S308. In step S308, the control unit 210 causes the distortion correction/image synthesis unit 240 to execute synthesis processing based on the multiple fisheye images obtained in the first imaging process and the second imaging process to generate a synthetic image, and ends this process in step S309.

As described above, according to the above-mentioned embodiment, it is possible to provide an imaging device, method, and program that can prevent a specific subject from being captured when capturing an image using multiple imaging units together.

In a specific embodiment in which subsequent imaging is performed based on the time elapsed since one imaging, the photographer can easily move outside the range of the subsequent imaging while referring to the easy-to-understand indicator of the elapsed time. In addition, if guidance is given by audio or light notification or a display on a display device at that time, the photographer can easily grasp the time available for movement.

Furthermore, in certain embodiments where an instruction is received that triggers the taking of a subsequent image, the photographer can ensure that there is sufficient time to move out of the range of the subsequent image capture and does not need to worry about time.

Furthermore, in a specific embodiment in which the disappearance or appearance of a specific subject within an acquired image is detected based on the image, the omnidirectional imaging device performs imaging with the subsequent imaging unit at its own discretion, so the photographer does not need to be concerned about the elapsed time or give explicit instructions.

The above-mentioned functional units can be realized by a computer-executable program written in a legacy programming language such as assembler, C, C++, C#, Java (registered trademark), or an object-oriented programming language, and can be stored on a device-readable recording medium such as a ROM, EEPROM, EPROM, flash memory, flexible disk, CD-ROM, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, Blu-ray disk, SD card, or MO, or distributed via a telecommunications line. In addition, some or all of the above functional units can be implemented on a programmable device (PD) such as a field programmable gate array (FPGA), or can be implemented as an ASIC (application specific integrated circuit), and can be distributed on a recording medium as circuit configuration data (bitstream data) to be downloaded to the PD to realize the above functional units on the PD, or data written in HDL (Hardware Description Language), VHDL (Very High Speed Integrated Circuits Hardware Description Language), Verilog-HDL, or the like to generate the circuit configuration data.

Although the embodiments of the present invention have been described above, the embodiments of the present invention are not limited to the above-mentioned embodiments, and can be modified within the scope of what a person skilled in the art can imagine, including other embodiments, additions, modifications, deletions, etc., and any aspect is within the scope of the present invention as long as it provides the functions and effects of the present invention.

12...imaging body, 14...housing, 18...shutter button, 20...imaging optical system, 22, 130...imaging element, 110...spherical camera, 112...CPU, 114...ROM, 116...image processing block, 117...still image compression block, 118...moving image compression block, 119...resize block, 120, 126...interface, 122...external storage interface, 124...external sensor interface, 126...USB interface, 128...serial block, 132...DRAM, 134...external storage, 136...motion sensor, 138...USB connector, 144...face detection block, 146...audio unit, 148...microphone, 150...speaker, 152...LCD driver, 154...LCD monitor, 200...function block, 210...control unit, 220...first imaging unit, 222...first image signal processing unit, 230...second imaging unit, 232...second image signal processing unit, 240...distortion correction/image synthesis unit, 250...codec processing unit

Patent No. 6065474 specification

Claims (10)

A plurality of imaging units each including an optical system and an imaging element;
a control unit that causes a first imaging unit of the plurality of imaging units to perform a first imaging process, and then causes a second imaging unit of the plurality of imaging units to perform a second imaging process;
a generation unit that generates a composite image based on a first image captured by the first imaging process and a second image captured by the second imaging process,
The control unit executes the second imaging process in the second imaging unit based on an elapsed time since imaging in the first imaging unit. A plurality of imaging units each including an optical system and an imaging element;
A control unit that controls each of the plurality of imaging units to capture an image;
a generation unit that generates a composite image based on a plurality of images captured at different times by the plurality of imaging units; and a disappearance detection unit that detects disappearance of a predetermined subject in an image acquired by a second imaging unit of the plurality of imaging units after a first imaging process by a first imaging unit of the plurality of imaging units,
The imaging device, wherein the control unit completes a second imaging process in the second imaging unit in response to the disappearance detection unit detecting the disappearance. A plurality of imaging units each including an optical system and an imaging element;
A control unit that controls each of the plurality of imaging units to capture an image;
a generation unit that generates a composite image based on a plurality of images captured at different times by the plurality of image capturing units; and an appearance detection unit that detects the appearance of a predetermined subject in an image acquired by a first image capturing unit after a first image capturing process in the first image capturing unit of the plurality of image capturing units,
An imaging device characterized in that the control unit executes a second imaging process in a second imaging unit among the multiple imaging units in response to the appearance detection unit detecting the appearance. The imaging device according to any one of claims 1 to 3, characterized in that the control unit causes the first imaging unit and the second imaging unit to simultaneously perform imaging. 4. The imaging device according to claim 2 , wherein the control unit executes the second imaging process in the second imaging unit based on an elapsed time since imaging in the first imaging unit. The imaging device according to claim 5, further comprising a guidance unit that provides guidance by audio or light notification, or by display on a display device, regarding the second imaging process in the second imaging unit based on the elapsed time. The imaging device according to any one of claims 1 to 3, further comprising a reception unit that receives an instruction to perform the second imaging process in the second imaging unit after the first imaging process in the first imaging unit, and the control unit executes the second imaging process in response to receiving the instruction. The imaging device according to any one of claims 1 to 7, wherein the images captured by each of the multiple imaging units are fisheye images, the composite image is a celestial sphere image obtained by combining the multiple fisheye images through coordinate conversion, or an image obtained by joining the multiple fisheye images, and the imaging device is a celestial sphere imaging device. A method implemented by an imaging device, comprising:
a first imaging step of capturing an image by a first imaging unit among a plurality of imaging units each including an optical system and an imaging element;
a second imaging step of capturing an image by a second imaging unit among the plurality of imaging units after the first imaging step is executed;
generating a composite image based on a plurality of images captured in the first imaging step and the second imaging step;
The second imaging step includes:
(1) based on the elapsed time from the first imaging step;
(2) After the first imaging step, in response to receiving an instruction to perform imaging by the second imaging unit,
(3) After the first imaging step, in response to detecting disappearance of a predetermined subject in an image acquired by the second imaging unit, the process is completed; or
(4) after the first imaging step, in response to detection of the appearance of a predetermined subject in the image acquired by the first imaging unit,
A method comprising : A program for implementing an imaging device, the program comprising: a control unit that causes a computer to perform a first imaging process on a first imaging unit among a plurality of imaging units each including an optical system and an imaging element, and then causes a computer to perform a second imaging process on a second imaging unit among the plurality of imaging units; and
a generating unit that generates a composite image based on a first image captured by the first imaging process and a second image captured by the second imaging process ;
The control unit executes the second imaging process in the second imaging unit based on an elapsed time since imaging in the first imaging unit.