JP7043707B2 - Scene recognition device, image pickup device, scene recognition method, and program - Google Patents

Description

The present invention relates to a scene recognition device, an image pickup device, a scene recognition method, and a program.

Patent Document 1 describes that "before determining the position of the object 50, the background and the object 50 are separated from the measurement data and noise processing is performed".
[Prior Art Document]
[Patent Document]
[Patent Document 1] Japanese Unexamined Patent Publication No. 11-63889

Depending on the content of the image, the accuracy of scene recognition of the image may decrease.

The scene recognition device according to one aspect of the present invention includes a circuit configured to acquire an image captured by an image pickup device including a range-finding sensor and a depth map of an image based on the range-finding result of the range-finding sensor. .. The circuit may be configured to crop a plurality of partial images according to the distance from the image to the subject based on the depth map. The circuit may be configured to perform scene recognition for each of the plurality of partial images according to their respective algorithms. The circuit may be configured to determine the scene of the image based on the result of each scene recognition of the plurality of partial images.

Based on the depth map, the circuit captures the first partial image including the subject whose distance from the image pickup device is less than or equal to the predetermined distance from the image, and the subject whose distance from the image pickup device is longer than the predetermined distance. It is configured to crop the included second partial image. The circuit is configured to perform first scene recognition for the first partial image according to the first algorithm. The circuit is configured to perform a second scene recognition for the second partial image according to the second algorithm. The circuit is configured to determine the scene of the image based on the result of the first scene recognition and the result of the second scene recognition.

Each algorithm may be a different algorithm.

The circuit may be configured to determine the scene of the image based further on the proportion of each of the plurality of partial images in the image.

The distance measuring sensor may be a TOF sensor.

The image pickup apparatus according to one aspect of the present invention may include the scene recognition device, a distance measuring sensor, and an image sensor. The circuit may be configured to determine at least one of the imaging conditions and the image processing conditions based on the determined image scene.

The scene recognition method according to one aspect of the present invention includes a step of acquiring an image captured by an image pickup apparatus equipped with a distance measuring sensor and a depth map of an image based on the distance measuring result of the distance measuring sensor. The scene recognition method may include a step of cropping a plurality of partial images according to the distance from the image to the subject based on the depth map. The scene recognition method may include a step of executing scene recognition according to each algorithm for each of a plurality of partial images. The scene recognition method may include a step of determining the scene of the image based on the result of each scene recognition of the plurality of partial images.

The program according to one aspect of the present invention may be a program for operating a computer as the scene recognition device.

According to one aspect of the present invention, it is possible to suppress a decrease in the accuracy of image scene recognition depending on the content of the image.

The outline of the above invention does not list all the necessary features of the present invention. A subcombination of these feature groups can also be an invention.

It is a figure which shows the functional block of an image pickup apparatus. It is a figure for demonstrating image cropping and scene recognition. It is a flowchart which shows an example of the procedure of scene recognition. It is a figure which shows an example of the appearance of an unmanned aerial vehicle and a remote control device. It is a figure which shows an example of a hardware configuration.

Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. Also, not all combinations of features described in the embodiments are essential to the means of solving the invention. It will be apparent to those skilled in the art that various changes or improvements can be made to the following embodiments. It is clear from the description of the claims that the form with such changes or improvements may be included in the technical scope of the present invention.

The claims, description, drawings, and abstracts include matters subject to copyright protection. The copyright holder will not object to any person's reproduction of these documents as long as they appear in the Patent Office files or records. However, in other cases, all copyrights are reserved.

Various embodiments of the present invention may be described with reference to flowcharts and block diagrams, wherein the block is (1) a stage of the process in which the operation is performed or (2) a device having a role of performing the operation. May represent a "part" of. Specific steps and "parts" may be implemented by programmable circuits and / or processors. Dedicated circuits may include digital and / or analog hardware circuits. It may include integrated circuits (ICs) and / or discrete circuits. The programmable circuit may include a reconfigurable hardware circuit. Reconfigurable hardware circuits include logical AND, logical OR, logical XOR, logical NAND, logical NOR, and other logical operations, flip-flops, registers, field programmable gate arrays (FPGA), programmable logic arrays (PLA), etc. It may include a memory element such as.

The computer readable medium may include any tangible device capable of storing instructions executed by the appropriate device. As a result, the computer-readable medium having the instructions stored therein will include the product, including instructions that can be executed to create means for performing the operation specified in the flowchart or block diagram. Examples of computer-readable media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable media include floppy (registered trademark) disks, diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), Electrically erasable programmable read-only memory (EPROM®), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), Blu-ray (registered trademark) disc , Memory sticks, integrated circuit cards, etc. may be included.

Computer-readable instructions may include either source code or object code written in any combination of one or more programming languages. Source code or object code includes traditional procedural programming languages. Traditional procedural programming languages are assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcodes, firmware instructions, state setting data, or Smalltalk®, JAVA®, C ++. It may be an object-oriented programming language such as, and a "C" programming language or a similar programming language. Computer-readable instructions are used locally or on a local area network (LAN), wide area network (WAN), etc., to the processor or programmable circuit of a general purpose computer, special purpose computer, or other programmable data processing device. ) May be provided. The processor or programmable circuit may execute computer-readable instructions to create means for performing the operations specified in the flowchart or block diagram. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, and the like.

FIG. 1 shows an example of a functional block of the image pickup apparatus 100 according to the present embodiment. The image pickup apparatus 100 includes an image pickup unit 102, a TOF sensor 160, and a lens unit 200. The image pickup unit 102 includes an image sensor 120, an image pickup control unit 110, a memory 170, a display unit 180, and an operation unit 182.

The image sensor 120 may be configured by CCD or CMOS. The image sensor 120 outputs the image data of the optical image formed through the plurality of lenses 154 to the image pickup control unit 110. The image pickup control unit 110 may be configured by a microprocessor such as a CPU or MPU, a microcontroller such as an MCU, or the like.

The image pickup control unit 110 generates image data by performing demosaic processing on the image signal output from the image sensor 120 in response to an operation command of the image pickup device 100 from the operation unit 182. The image pickup control unit 110 stores the image data in the memory 170. The image pickup control unit 110 controls the TOF sensor 160. The image pickup control unit 110 is an example of a circuit. The TOF sensor 160 is a time-of-flight sensor that measures the distance to an object. The image pickup apparatus 100 executes focusing control by adjusting the position of the focus lens based on the distance measured by the TOF sensor 160.

The memory 170 may be a computer-readable recording medium and may include at least one of flash memories such as SRAM, DRAM, EEPROM, EEPROM, and USB memory. The memory 170 stores a program or the like necessary for the image pickup control unit 110 to control the image sensor 120 or the like. The memory 170 may be provided inside the housing of the image pickup apparatus 100.

The plurality of lenses 154 may function as a zoom lens, a varifocal lens, and a focus lens. At least some or all of the plurality of lenses 154 are movably arranged along the optical axis. The lens control unit 150 drives the lens drive unit 152 in accordance with a lens control command from the image pickup control unit 110 to move one or more lenses 154 along the optical axis direction. The lens control command is, for example, a zoom control command and a focus control command. The lens driving unit 152 may include a voice coil motor (VCM) that moves at least a part or all of the plurality of lenses 154 in the optical axis direction. The lens drive unit 152 may include an electric motor such as a DC motor, a coreless motor, or an ultrasonic motor. The lens driving unit 152 transmits power from the electric motor to at least a part or all of the plurality of lenses 154 via a mechanical member such as a cam ring and a guide shaft, and emits light at least a part or all of the plurality of lenses 154. It may be moved along the axis. In this embodiment, an example in which the plurality of lenses 154 are integrated with the image pickup apparatus 100 will be described. However, the plurality of lenses 154 may be interchangeable lenses and may be configured separately from the image pickup apparatus 100.

The display unit 180 may display the image output from the image sensor 120. The display unit 180 may display various setting information of the image pickup apparatus 100. The display unit 180 may be a liquid crystal display, a touch panel display, or the like. The display unit 180 may include a plurality of liquid crystal displays or touch panel displays.

The TOF sensor 160 includes a light emitting unit 162, a light receiving unit 164, a light emitting control unit 166, a light receiving control unit 167, and a memory 168. The TOF sensor 160 is an example of a distance measuring sensor. The image pickup apparatus 100 may include another distance measuring sensor such as a stereo camera that measures the distance based on the parallax instead of the TOF sensor 160. The light emitting unit 162 includes at least one light emitting element 163. The light emitting element 163 is a device that repeatedly emits high-speed modulated pulsed light such as an LED or a laser. The light emitting element 163 may emit pulsed light which is infrared light. The light emission control unit 166 controls the light emission of the light emitting element 163. The light emission control unit 166 may control the pulse width of the pulsed light emitted from the light emitting element 163.

The light receiving unit 164 includes a plurality of light receiving elements 165 that measure the distance to the subject associated with each of the plurality of distance measuring regions. The plurality of light receiving elements 165 correspond to each of the plurality of ranging regions. The light receiving element 165 repeatedly receives the reflected light of the pulsed light from the object. The light receiving control unit 167 controls the light receiving of the light receiving element 165. The light receiving control unit 167 measures the distance to the subject associated with each of the plurality of distance measuring regions based on the amount of reflected light repeatedly received by the light receiving element 165 during a predetermined light receiving period. The light receiving control unit 167 determines the distance to the subject by specifying the phase difference between the pulsed light and the reflected light based on the amount of reflected light repeatedly received by the light receiving element 165 during a predetermined light receiving period. You may measure the distance. The light receiving unit 164 may measure the distance to the subject by reading the frequency change of the reflected wave. This is called an FMCW (Frequency Modulated Continuous Wave) method.

The memory 168 may be a computer-readable recording medium and may include at least one of SRAM, DRAM, EPROM, and EEPROM. The memory 168 stores a program required for the light emitting control unit 166 to control the light emitting unit 162, a program required for the light receiving control unit 167 to control the light receiving unit 164, and the like.

The TOF sensor 160 can measure the distance to the subject associated with each of the plurality of distance measuring regions corresponding to the number of pixels of the light receiving unit 164.

The image pickup apparatus 100 configured in this way executes scene recognition of the image captured by the image pickup apparatus 100. The image pickup apparatus 100 determines at least one of an image pickup condition and an image processing condition based on the result of scene recognition of an image. Imaging conditions include exposure conditions including shutter speed, aperture (F value), and ISO sensitivity. Image processing conditions include digital signal processing conditions such as white balance adjustment, gradation (tone curve) control, and color adjustment.

The image pickup apparatus 100 includes at least one of image brightness information, image color information, and a histogram of pixel values constituting the image, and a reference brightness information, color information, and pixel values predetermined for each scene type. Scene recognition of an image may be performed based on its similarity to at least one of the histograms. The image pickup apparatus 100 may discriminate a landscape, a person, a plant, a dish, a night view, a firework, or the like as an image scene based on at least one of an image luminance information, an image color information, and a histogram of pixel values. .. The image pickup apparatus 100 may execute scene recognition of an image based on the degree of similarity between the teacher data for each type of scene and the image to be discriminated.

Here, since the image is two-dimensional information and does not have depth information, the image pickup apparatus 100 may not be able to properly recognize the scene depending on the content of the image. For example, the image pickup apparatus 100 may not be able to distinguish between a landscape image and an actual landscape image for scene recognition.

Therefore, the image pickup apparatus 100 according to the present embodiment performs scene recognition by using the depth map based on the distance measurement result from the TOF sensor 160 in addition to the two-dimensional information that can be acquired from the image.

The image pickup control unit 110 acquires an image captured by the image pickup unit 102. The image pickup control unit 110 acquires a depth map of the image based on the distance measurement result of the TOF sensor 160. The depth map may be information showing the distribution of the distance to the subject.

Based on the depth map, the image pickup control unit 110 has a foreground partial image including a subject whose distance from the image pickup unit 102 is less than or equal to a predetermined distance from the image, and a predetermined distance from the image pickup unit 102. Crop a distant view partial image including a distant subject. The image pickup control unit 110 may specify the distance to the subject in the image based on a predetermined correspondence relationship between each pixel of the image and each pixel of the depth map. The image pickup control unit 110 generates a near-view partial image and a distant partial image by dividing the image into a region including a subject within a predetermined distance and a region including a subject farther than the predetermined distance. do.

The image pickup control unit 110 executes near-view scene recognition for a near-view partial image according to a predetermined near-view algorithm. The foreground algorithm may be an algorithm specialized for recognizing a foreground scene. The near view algorithm may be an algorithm that outputs only the near view scene and does not output the distant view scene as the recognition result.

The image pickup control unit 110 executes distant view scene recognition for the distant view partial image according to a predetermined distant view algorithm. The distant view algorithm may be an algorithm specialized for recognizing a distant view scene. The distant view algorithm may be an algorithm that outputs only the distant view scene and does not output the near view scene as the recognition result. A distant view scene is a scene including a subject at a distance farther than a predetermined distance, and is a scene such as a mountain, a sky, a night view, and fireworks. The foreground scene is a scene including a subject at a distance equal to or less than a predetermined distance, and is a scene of a person, food, a flower, or the like.

An algorithm different from the near view algorithm and the distant view algorithm may be used. However, the image pickup control unit 110 may execute scene recognition for each of the near view partial image and the distant view partial image according to the same algorithm.

The image pickup control unit 110 determines the scene of the image based on the recognition result of the scene of the near view partial image and the recognition result of the scene of the distant view partial image. The image pickup control unit 110 may determine the result of combining the recognition result of the scene of the near view partial image and the recognition result of the scene of the distant view partial image as the scene of the image. For example, if the recognition result of the scene of the near view partial image is "mountain" and the recognition result of the scene of the distant view partial image is "person", the image pickup control unit 110 sets the image scene as "mountain" + "person". You may decide.

The image pickup control unit 110 may determine the scene of the image based on the ratio of the near view partial image and the distant view partial image to the image. The image pickup control unit 110 may determine, for example, the recognition result of one of the near view partial image and the distant view partial image, which occupy a large proportion of the image, as the image scene. The image pickup control unit 110 may determine the recognition result of each scene of the near view partial image and the distant view partial image and the ratio of each of the near view partial image and the distant view partial image to the image as the scene of the image. For example, the image pickup control unit 110 may determine an image scene as a scene in which "mountains" occupy 70% and "people" occupy 30%.

As shown in FIG. 2, the image pickup control unit 110 is farther than a predetermined distance by comparing the image 300 captured by the image pickup unit 102 with the depth map 310 based on the distance measurement result by the TOF sensor 160. A distant view partial image 320 including a subject and a near view partial image 322 including a subject within a predetermined distance are cropped from the image 300.

The image pickup control unit 110 may crop the image 300 so that the boundary portions of the distant view partial image 320 and the near view partial image 322 overlap. The image pickup control unit 110 may specify the correspondence between each pixel of the image 300 and each pixel of the depth map 310 by image matching the image 300 and the depth map 310. The predetermined distance may be determined based on the distance measuring range of the TOF sensor 160. The predetermined distance may be the farthest distance in the range measuring range that the TOF sensor 160 can measure with a predetermined accuracy or higher.

The image pickup control unit 110 executes scene recognition of the distant view partial image 320 according to the distant view algorithm, and determines the scene of the distant view partial image 320 as "landscape". Further, the image pickup control unit 110 executes scene recognition of the near view partial image 322 according to the near view algorithm, and determines the scene of the near view partial image 322 as a “person”. Next, the image pickup control unit 110 determines the scene of the image 300 as "landscape" + "person" based on the result of the scene recognition of the distant view partial image 320 and the result of the scene recognition of the near view partial image 322.

FIG. 3 is a flowchart showing an example of the scene recognition procedure.

The image pickup control unit 110 acquires an image captured by the image pickup unit 102 and a depth map corresponding to the image based on the distance measurement result measured by the TOF sensor 160 (S100). The image pickup control unit 110 associates the positional relationship between the image and the depth map (S102). The image pickup control unit 110 may associate the positional relationship with the image by matching the image with the depth map.

By referring to the depth map, the image pickup control unit 110 refers to a near-view partial image in which the distance from the image to the subject is equal to or less than a predetermined distance, and a region in which the distance to the subject is farther than the predetermined distance. The distant view partial image is cropped (S104). The image pickup control unit 110 executes scene recognition of the near view partial image according to the near view algorithm, and executes scene recognition of the distant view partial image according to the distant view algorithm (S106).

The image pickup control unit 110 executes scene recognition of the image based on the result of the scene recognition of the near view partial image and the result of the scene recognition of the distant view partial image, and determines the scene of the image (S108).

According to the present embodiment, the image pickup control unit 110 executes scene recognition by using the depth map based on the distance measurement result from the TOF sensor 160 in addition to the information that can be acquired from the image. The image pickup control unit 110 divides the image into partial images according to the distance to the subject, and executes scene recognition for each divided image. This makes it possible to perform scene recognition of the image with higher accuracy. For example, scene recognition can be performed by distinguishing between a landscape painting image and an actual landscape image.

In this embodiment, an example in which the image pickup control unit 110 divides an image into two, a near view partial image and a distant view partial image, has been described. However, the image pickup control unit 110 divides into three or more partial images according to the distance to the subject included in the image, and executes scene recognition of each partial image according to the algorithm of each partial image. May be good.

From the image, the image pickup control unit 110 includes a first partial image including a subject in the first distance range, a second partial image including a subject in the second distance range farther than the first distance range, and a third partial image far from the second distance range. A third partial image containing a subject in a distance range may be cropped. The image pickup control unit 110 determines the scenes of the first partial image, the second partial image, and the third partial image according to the algorithm of the first range, the algorithm of the second range, and the algorithm of the third range, respectively. The recognition may be performed and the scene of the image may be determined based on the results of the scene recognition of the first partial image, the second partial image, and the third partial image.

The image pickup apparatus 100 as described above may be mounted on a moving body. The image pickup apparatus 100 may be mounted on an unmanned aerial vehicle (UAV) as shown in FIG. The UAV 1000 may include a UAV main body 20, a gimbal 50, a plurality of image pickup devices 60, and an image pickup device 100. The gimbal 50 and the image pickup apparatus 100 are examples of an image pickup system. The UAV1000 is an example of a moving body propelled by a propulsion unit. The moving body is a concept including a UAV, a flying object such as another aircraft moving in the air, a vehicle moving on the ground, a ship moving on the water, and the like.

The UAV main body 20 includes a plurality of rotary wings. The plurality of rotor blades are an example of a propulsion unit. The UAV main body 20 flies the UAV 1000 by controlling the rotation of a plurality of rotary blades. The UAV body 20 flies the UAV 1000 using, for example, four rotor blades. The number of rotor blades is not limited to four. Further, the UAV1000 may be a fixed-wing aircraft having no rotor blades.

The image pickup apparatus 100 is a camera for taking an image of a subject included in a desired imaging range. The gimbal 50 rotatably supports the image pickup device 100. The gimbal 50 is an example of a support mechanism. For example, the gimbal 50 rotatably supports the image pickup device 100 on a pitch axis using an actuator. The gimbal 50 further rotatably supports the image pickup device 100 around each of the roll axis and the yaw axis by using an actuator. The gimbal 50 may change the posture of the image pickup device 100 by rotating the image pickup device 100 around at least one of the yaw axis, the pitch axis, and the roll axis.

The plurality of image pickup devices 60 are sensing cameras that image the surroundings of the UAV 1000 in order to control the flight of the UAV 1000. Two image pickup devices 60 may be provided in front of the nose of the UAV 1000. Yet two other imaging devices 60 may be provided on the bottom surface of the UAV 1000. The two image pickup devices 60 on the front side may be paired and function as a so-called stereo camera. The two image pickup devices 60 on the bottom side may also be paired and function as a stereo camera. Three-dimensional spatial data around the UAV 1000 may be generated based on the images captured by the plurality of image pickup devices 60. The number of image pickup devices 60 included in the UAV 1000 is not limited to four. The UAV 1000 may include at least one image pickup device 60. The UAV1000 may be equipped with at least one image pickup device 60 on each of the nose, tail, side surface, bottom surface, and ceiling surface of the UAV1000. The angle of view that can be set by the image pickup device 60 may be wider than the angle of view that can be set by the image pickup device 100. The image pickup apparatus 60 may have a single focus lens or a fisheye lens.

The remote control device 600 communicates with the UAV 1000 to remotely control the UAV 1000. The remote control device 600 may communicate wirelessly with the UAV 1000. The remote control device 600 transmits to the UAV 1000 instruction information indicating various commands related to the movement of the UAV 1000 such as ascending, descending, accelerating, decelerating, advancing, reversing, and rotating. The instruction information includes, for example, instruction information for raising the altitude of the UAV 1000. The instruction information may indicate the altitude at which the UAV 1000 should be located. The UAV 1000 moves so as to be located at an altitude indicated by the instruction information received from the remote control device 600. The instruction information may include an ascending instruction to ascend the UAV 1000. The UAV1000 rises while accepting the rise order. Even if the UAV1000 accepts an ascending order, the ascending may be restricted if the altitude of the UAV1000 has reached the upper limit altitude.

FIG. 5 shows an example of a computer 1200 in which a plurality of aspects of the present invention may be embodied in whole or in part. The program installed on the computer 1200 can cause the computer 1200 to function as an operation associated with the device according to an embodiment of the present invention or as one or more "parts" of the device. Alternatively, the program may cause the computer 1200 to perform the operation or the one or more "parts". The program may cause the computer 1200 to perform a process according to an embodiment of the present invention or a step of the process. Such a program may be run by the CPU 1212 to cause the computer 1200 to perform certain operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

The computer 1200 according to this embodiment includes a CPU 1212 and a RAM 1214, which are connected to each other by a host controller 1210. The computer 1200 also includes a communication interface 1222, an input / output unit, which are connected to the host controller 1210 via an input / output controller 1220. The computer 1200 also includes a ROM 1230. The CPU 1212 operates according to a program stored in the ROM 1230 and the RAM 1214, thereby controlling each unit.

The communication interface 1222 communicates with other electronic devices via the network. A hard disk drive may store programs and data used by the CPU 1212 in the computer 1200. The ROM 1230 stores in it a boot program or the like executed by the computer 1200 at the time of activation and / or a program depending on the hardware of the computer 1200. The program is provided via a computer-readable recording medium such as a CR-ROM, USB memory or IC card or a network. The program is installed in RAM 1214, which is also an example of a computer-readable recording medium, or ROM 1230, and is executed by the CPU 1212. The information processing described in these programs is read by the computer 1200 and provides a link between the program and the various types of hardware resources described above. The device or method may be configured to implement the operation or processing of information in accordance with the use of the computer 1200.

For example, when communication is executed between the computer 1200 and an external device, the CPU 1212 executes a communication program loaded in the RAM 1214, and performs communication processing with respect to the communication interface 1222 based on the processing described in the communication program. You may order. Under the control of the CPU 1212, the communication interface 1222 reads the transmission data stored in the transmission buffer area provided in the RAM 1214 or a recording medium such as a USB memory, and transmits the read transmission data to the network, or The received data received from the network is written to the reception buffer area or the like provided on the recording medium.

Further, the CPU 1212 makes the RAM 1214 read all or necessary parts of a file or a database stored in an external recording medium such as a USB memory, and executes various types of processing on the data on the RAM 1214. good. The CPU 1212 may then write back the processed data to an external recording medium.

Various types of information such as various types of programs, data, tables, and databases may be stored in recording media and processed. The CPU 1212 describes various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, and information retrieval described in various parts of the present disclosure with respect to the data read from the RAM 1214. Various types of processing may be performed, including / replacement, etc., and the results are written back to the RAM 1214. Further, the CPU 1212 may search for information in a file, database, or the like in the recording medium. For example, when a plurality of entries each having an attribute value of the first attribute associated with the attribute value of the second attribute are stored in the recording medium, the CPU 1212 specifies the attribute value of the first attribute. Search for an entry that matches the condition from the plurality of entries, read the attribute value of the second attribute stored in the entry, and associate it with the first attribute that satisfies the predetermined condition. The attribute value of the second attribute obtained may be acquired.

The program or software module described above may be stored on a computer 1200 or in a computer readable storage medium near the computer 1200. Also, a recording medium such as a hard disk or RAM provided in a dedicated communication network or a server system connected to the Internet can be used as a computer readable storage medium, thereby allowing the program to be transferred to the computer 1200 over the network. offer.

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various changes or improvements can be made to the above embodiments. It is clear from the description of the claims that the form with such changes or improvements may be included in the technical scope of the present invention.

The order of execution of each process such as operation, procedure, step, and step in the apparatus, system, program, and method shown in the claims, specification, and drawings is particularly "before" and "prior to". It should be noted that it can be realized in any order unless the output of the previous process is used in the subsequent process. Even if the scope of claims, the specification, and the operation flow in the drawings are explained using "first", "next", etc. for convenience, it means that it is essential to carry out in this order. It's not a thing.

100 Imaging device 102 Imaging unit 110 Imaging control unit 120 Image sensor 150 Lens control unit 152 Lens drive unit 154 Lens 160 TOF sensor 162 Light emitting unit 163 Light emitting element 164 Light receiving unit 165 Light receiving element 166 Light receiving control unit 167 Light receiving control unit 168 Memory 170 Memory 180 Display unit 182 Operation unit 200 Lens unit 1000 UAV
20 UAV main unit 50 gimbal 60 image pickup device 600 remote control device 1200 computer 1210 host controller 1212 CPU
1214 RAM
1220 Input / Output Controller 1222 Communication Interface 1230 ROM

Claims (6)

An image captured by the image sensor of an image pickup device including a distance measurement sensor which is a TOF sensor and an image sensor, and a depth map of the image based on the distance measurement result of the distance measurement sensor are acquired.
Based on the depth map, a plurality of partial images according to the distance from the image to the subject are cropped.
Scene recognition is executed according to each algorithm for each of the plurality of partial images, and the scene recognition is performed.
A circuit configured to determine the scene of the image based on the result of each scene recognition of the plurality of partial images is provided .
The circuit is
Based on the depth map, the first partial image including a subject whose distance from the image pickup device is equal to or less than a predetermined distance from the image, and the distance from the image pickup device are farther than the predetermined distance. Crop the second part image including the subject,
The first scene recognition for the first partial image is executed according to the first algorithm,
The second scene recognition for the second partial image is executed according to the second algorithm.
It is configured to determine the scene of the image based on the result of the first scene recognition and the result of the second scene recognition.
The predetermined distance is the farthest distance in the distance measuring range in which the TOF sensor can measure a distance with a predetermined accuracy or higher . The scene recognition device according to claim 1, wherein each of the algorithms is a different algorithm. The scene recognition device according to claim 1, wherein the circuit is configured to determine a scene of the image based on the ratio of each of the plurality of partial images to the image. The scene recognition device according to any one of claims 1 to 3 .
With the distance measuring sensor
Equipped with the image sensor
The circuit is an imaging device configured to determine at least one of an imaging condition and an image processing condition based on the determined scene of the image. A step of acquiring an image captured by the image sensor of an image pickup device including a distance measurement sensor which is a TOF sensor and an image sensor, and a step of acquiring a depth map of the image based on the distance measurement result of the distance measurement sensor.
Based on the depth map, a step of cropping a plurality of partial images according to the distance from the image to the subject, and
The stage of executing scene recognition according to each algorithm for each of the plurality of partial images, and
A step of determining the scene of the image based on the result of each scene recognition of the plurality of partial images is provided .
The cropping stage is
Based on the depth map, the first partial image including a subject whose distance from the image pickup device is equal to or less than a predetermined distance from the image, and the distance from the image pickup device are farther than the predetermined distance. Including the stage of cropping the second part image including the subject,
The stage of determining the scene of the image is
The stage of executing the first scene recognition for the first partial image according to the first algorithm, and
The stage of executing the second scene recognition for the second partial image according to the second algorithm, and
The stage of determining the scene of the image based on the result of the first scene recognition and the result of the second scene recognition.
Including
The predetermined distance is a scene recognition method in which the TOF sensor is the farthest distance in a distance measuring range that can be measured with a predetermined accuracy or higher . A program for operating a computer as the scene recognition device according to any one of claims 1 to 3 .

Images