JP6155032B2 - Shooting system - Google Patents

Description

  The present invention relates to a photographing system for photographing a bandit about to escape from a building from a mobile robot, and more particularly to a photographing system for photographing a bandage photographed inside a building from a mobile robot even outside the building.

2. Description of the Related Art Conventionally, there has been proposed a monitoring system in which various sensors are installed in a building and its surrounding monitoring area, and when the sensor detects an abnormality, the mobile robot moves to the place where the abnormality is detected and images the monitoring area.
For example, Patent Document 1 discloses a monitoring system in which a ground-traveling mobile robot moves to a place where an abnormality occurs and images an abnormal situation based on signals from a sensor that detects a fire or a sensor that detects an intruder. It is disclosed. Such a monitoring system may monitor a wide monitoring area such as a factory or a shopping mall where a parking lot is provided. In the system of Patent Document 1, the mobile robot travels to a place where the sensor detects an abnormality, and performs imaging of the place where the abnormality has occurred, notification to the monitoring center, and the like.

JP 2009-181270 A

By the way, even for a mobile robot having a wide monitoring area, it is often difficult to enable both indoor movement and outdoor movement. For example, the mobile robot described in Patent Document 1 targets a site such as an outdoor parking lot. For this reason, when the mobile robot tries to enter the inside of the building, the door of the building must be opened, and if there is a staircase in the building, it is necessary to have a function of moving up and down the staircase. For this reason, the location in charge of the mobile robot is determined in advance, such as indoor use and outdoor use.

In such a situation, an outdoor mobile robot that cannot enter the indoor area receives a signal that a bandit has entered the building, completes preparations for shooting, and tries to shoot a person outside the building. There was a problem that I didn't know which person was the bandit and I couldn't know if the bandit flew away.

  Therefore, the present invention can extract a person feature for identifying a bandit from an image of a bandit photographed indoors, and perform collation with the person feature extracted from the image of a person filmed outdoors. Aiming to realize a shooting system that can

In order to achieve such an object, the present invention provides an in-building sensor for detecting an intruder into a building, an in-building camera for photographing inside the building, a flying robot for photographing while flying outside the building, The image processing unit includes an image processing unit that processes an image captured by a camera or a flying robot, and a control device that has a flying robot control unit that controls the flight of the flying robot. After the building sensor detects an intruder, the image processing unit The person area extracted from the image inside the building photographed by the internal camera and the person area extracted from the outdoor image photographed by the flying robot are collated, and the collation result is output to the flying robot controller. The robot control unit provides an imaging system that controls the flying robot to track a person when the image processing unit determines that the person is the same.

The imaging system according to the present invention determines whether an intruder who has entered a building matches a person who is outside the building. It is possible to identify the invading bandit and track the person.

The imaging system further includes an object sensor that outputs an object position that exists in an external area of the building, and the control device further includes an object position calculation unit that calculates a person position based on the output of the object sensor, and the flying robot It is preferable that the control unit controls the flying robot so as to photograph the person calculated by the object position calculation unit.

Thereby, a person can be detected in a wider area than when a person outside the building is detected from an image taken by the flying robot.

The image processing unit preferably extracts and collates at least one of a face image, clothes information, and height information as a person feature from the person area.

Thereby, it is possible to collate with high accuracy by using in addition to a face image useful as information characterizing a person, clothes information and the like.

According to the present invention, it is possible to determine whether any of the persons existing outside the building is a bandit, which can be utilized for the subsequent tracking process.

Illustration explaining the flying image of a flying robot Functional block diagram of a flying robot Overall configuration diagram of the monitoring system Functional block diagram of security equipment Functional block diagram of the robot control module Schematic diagram showing how facial and torso areas are extracted from the in-building image Schematic diagram of reference face image used for face image matching processing Diagram showing the detection area of the laser sensor Processing flow for building images Control flow of a flying robot when an intruder is detected Processing flow when outdoor images are acquired

  Hereinafter, an embodiment in which a photographing system according to the present invention is applied to a monitoring system will be described.

FIG. 3 is a diagram schematically illustrating the overall configuration of the monitoring system 1. The monitoring system 1 includes a security device 2, a flying robot 3, a laser sensor 4, a building sensor 5 installed in a monitoring area E, a center device 6 installed in a monitoring center connected via a network, and a building It is composed of a color in-building camera 7 whose detection range of the sensor 5 is approximately the visual field range.
The center device 6 is connected to the security device 2 via an IP network, receives an image taken by the flying robot 3, an image taken by the in-building camera 7, a detection signal from the in-building sensor 5 and the like from the security device 2, and monitors them. To display. The monitor looks at this monitor to grasp the status of the monitoring area E and performs an appropriate response. Further, although the network is described as an IP network, the network is not limited to this as long as there is no problem in image transmission / reception such as a general public network or a mobile phone network.
Although not shown, it is also preferable to provide a lighting device in order to cope with a case where the inside of the building is dark and there is a problem in shooting with the in-building camera 7.

The flying robot 3 receives a wireless flight control signal from the security device 2, flies while photographing to a predetermined target position, and transmits the photographed image to the security device 2. FIG. 2 is a diagram showing functional blocks of the flying robot 3.
The flying robot 3 includes an antenna 31 for performing wireless communication with the security device 2, four rotors 32 for flying such as ascending / descending / turning direction / advancing, a motor for providing driving force to the rotor 32, and the like. A rotor drive unit 33, an altitude sensor 34 for projecting and receiving a laser beam vertically below to measure the current altitude of the flying robot 3, and a distance measuring sensor for projecting and receiving a laser beam in the horizontal direction and surroundings to measure the surrounding situation of the flying robot 3 35, a camera 36 that captures a color image of the front of the flying robot 3, an illumination 37 that is an LED illumination that lights up when the surroundings are dark and assists the camera 36, and a robot controller that controls the entire flying robot 3. 38, a power source 39 which is a lithium polymer battery for supplying electricity to each part of the flying robot 3.

The robot control unit 38 also acquires communication control means 381 for controlling wireless communication with the security device 2 via the antenna 31, acquisition start / end of the camera 36 and images taken by the camera 36, and communication control means 381. Is transmitted from the communication control means 381 as scan data using the altitude information measured by the camera control means 382, the distance measuring sensor 35 and the altitude sensor 34, and the distance data between the surrounding object and the own device. A scanning unit 383 that performs processing such as transmission to the device 2 and a flight control unit 384 that controls the rotor driving unit 33 based on the flight control signal from the security device 2 to control the flying robot 3 to fly to the target position. It is composed of

Next, the security device 2 will be described in detail with reference to FIGS. 1, 4, and 5. The security device 2 is installed inside the building B in the monitoring area E shown in FIG. The security device 2 includes a sensor 5 in the building, a sensor 5 in the building, and a sensor 5 in the building installed at an appropriate place for detecting a bandit that illegally attempts to intrude into the building B The camera 7 in the building whose detection range is approximately the visual field range, and the laser sensor 4 that outputs the object position within the monitoring area E and outside the building B such as a parking lot as a detection area are connected.

FIG. 4 is a diagram illustrating functional blocks of the security device 2. The security device 2 includes a security mode switching unit 21 that performs a switching operation between a security set state monitored by the monitoring center and a security release state that is not monitored by the monitoring center, and various types of sensors such as the laser sensor 4 and the in-building sensor 5. A sensor interface 22 that receives input of signals from the sensor and an image from the in-building camera 7, a flying robot communication unit 25 that communicates with the flying robot 3, images taken by the flying robot 3, abnormal signals detected by various sensors, and the like The monitoring center communication unit 26 communicates with the center device 6 via the network, and includes peripheral parts such as a ROM / RAM that stores programs necessary for processing of the security device 2, various data, parameters, and the like. From the security control unit 23 including a storage unit 24 and a CPU, MPU, etc. that control the entire security device 2 It has been made.

  Here, information stored in the storage unit 24 will be described. The monitoring space map 241 is information that represents the monitoring area E in three dimensions, and is map information that represents the monitoring space from the ground to a height required for the flight of the flying robot 3. In the present embodiment, information on objects existing in the monitoring space in advance, such as the presence of a fence that separates the monitoring area E from the outside, the building B, and the installation position of the laser sensor 4 is stored. Note that the monitoring space map 241 also includes three-dimensional information inside the building B, and places where people can enter and exit are registered, such as doors and windows.

The in-building sensor camera arrangement information 242 is position information in the monitoring space map 241 of the monitoring location of each in-building sensor 5 and each in-building camera 7. This is determined in advance by a security plan, and a position on the monitoring space map 241 is associated with each in-building sensor 5.

The laser sensor parameter 243 is information including the correspondence between the position of the laser sensor 4 in the monitoring space map 241 and the position in the detection area of the laser sensor 4 and the position on the monitoring space map 241. It is a parameter for converting the determined position into a position on the monitoring space map 241.

The various parameters 244 are various other parameters such as the IP address of the center device 6 necessary for the security device 2 to monitor the monitoring area E and data for communication with the flying robot 3. In addition to these, the storage unit 24 stores various programs for realizing the functions of the security device 2.

The three-dimensional face shape data 245 is data for expressing a standard three-dimensional shape of a person's face with a frame model, and is a characteristic part of the person's face, for example, the eyes, the corners of the eyes, and the nose. Feature points are set at the corner points of the nose tip and mouth. The number of feature points is set in advance as many as necessary for the synthesis process with a two-dimensional face image described later.
The expression method of the three-dimensional face shape data 245 is not limited to the wire frame model, and other methods may be used as long as the method can model the three-dimensional structure of a person, such as a surface model.

Next, the security control unit 23 will be described in detail. The security control unit 23 reads a software module (not shown) in the storage unit 24 and performs each process by the CPU or the like.

The laser sensor analysis module 231 is software that operates as an object position calculation unit and analyzes a signal of the laser sensor 4 input from the sensor interface 22. Specifically, an exploration signal, which is a result of scanning a detection area with a laser beam, which is an object sensor, is analyzed in time series. If there is no new approaching object or the like in the detection area, the search signal of the laser sensor 4 input in time series does not change so much, and the analysis result indicates that there is no moving object. On the other hand, if there is a new approaching object or the like in the detection area, the search signal of the laser sensor 4 changes, so the position in the detection area where the change has occurred is obtained by analysis. Further, using the laser sensor parameter 243 of the storage unit 24, the position is converted into a position on the monitoring space map 241 to calculate the position / size / movement direction of the approaching object, and the approaching object is tracked on the monitoring space map 241. . Further, when the approaching object stops, there is no change in the signal thereafter, so it can be determined that the object such as the person being tracked has stopped.
The analysis result of the laser sensor analysis module 231 is output to an abnormality determination module 232 and a flying robot control module 233 described later.

The abnormality determination module 232 receives a security set / release signal from the security mode switching unit 21 and signals from the in-building sensor 5 and the laser sensor 4 and determines whether an abnormality has occurred in the monitoring area E. When receiving the security set signal from the security mode switching unit 21, the abnormality determination module 232 sets the monitoring area E to the security set mode that warns, and when receiving the security release signal, sets the monitoring area E to the security release mode that is not warning.
In the security release mode, no special processing is performed even if a detection signal from the in-building sensor 5 or the laser sensor 4 is received. On the other hand, in the security set mode, when a detection signal from the in-building sensor 5 or the laser sensor 4 is received, it is determined that an abnormality has occurred, and the monitoring center communication unit 26 notifies the center device 6 of the abnormality. When the detection signal from the in-building sensor 5 is received together with the abnormality report, the flying robot control module 233 is controlled to start the flying robot 3. And the process which transmits the image which the flying robot 3 received from the flying robot communication part 25 image | photographed from the monitoring center communication part 26 to the center apparatus 6 is continued until an abnormal state is cancelled | released. Although there are various methods for canceling the abnormal state, the description thereof is omitted because it is not relevant to the present invention.

The flying robot control module 233 operates as a flying robot control unit, and performs flight control of the flying robot 3 from the flying robot communication unit 25 when receiving an activation signal of the flying robot 3 by the abnormality determination module 232. In addition, the flying robot control module 233 receives the collation result output from the image processing module 234 described later, and controls the flight of the flying robot 3 so as to track the person who matches the intruder photographed by the in-building camera 7.

Here, the flying robot control module 233 will be described in detail with reference to FIG. FIG. 5 is a functional block diagram of the flying robot control module 233. The flying robot control module 233 includes a target position setting means A that determines a target position that the flying robot 3 should reach, and a flight path calculation means B that calculates a flight path for reaching the target position set by the target position setting means A. The robot control means C for generating and transmitting a flight control signal to the flying robot 3 so as to fly along the flight path calculated by the flight path calculation means B, and the current flight on the monitoring space map 241 of the flying robot 3 It comprises flight position calculation means D for calculating the position.

The target position setting means a sets the altitude of about 3 m above the ground and 3 m or more from the position of the person at the position on the person's monitoring space map 241 calculated by the laser sensor analysis module 231.
Here, the target position with an altitude of 3 m or more from the position of the person and about 3 m above the ground is a bandit with the depression angle when the flying robot 3 captures the face of the person being 30 degrees or less, as will be described later. This is a position to prevent the flying robot 3 from reaching from. Depending on the focal length of the camera mounted on the flying robot 3, the human face can be photographed from a further distance, but the focal length is taken into account so that the entire body can be photographed in addition to the face to acquire clothes information and height information. Set the target position.
The target position setting means a preferably sets the target position after it is determined by the laser sensor 4 that a person has come out of the building B. This is considered to be a stage where the intruder is still inside the building B immediately after the in-building sensor 5 detects the intruder. At that time, a person who is not in the monitoring space E by chance is not mistaken for an intruder. It is to make it.

Returning to FIG. 4, the image processing module 234 processes the image captured by the flying robot 3 and the image captured by the in-building camera 7 received from the flying robot communication unit 25.

The image processing module 234 operates as an image processing unit, and extracts and compares human features of a person shown in an in-building image taken by the in-building camera 7 and an outdoor image taken by the camera 36 included in the flying robot 3. Do. That is, images from both cameras are captured, and the following processing is performed on the captured images.
First, processing for an in-building image will be described with reference to a schematic diagram shown in FIG.
When the in-building sensor 5 detects the presence of an intruder, the in-building camera 7 captures the detection range of the in-building sensor 5 and acquires an in-building image. FIG. 6A shows an example of the in-building image 600 acquired. The in-building image 600 schematically shows a state where an intruder 602 who has broken the window 601 and entered the building B is captured.

When the image processing module 234 obtains the in-building image 600 from the in-building camera 7, the image processing module 234 extracts a person region using a learning discriminator that has learned from images of various persons. For example, a person area is extracted by a learning classifier constructed in advance using learning data about the presence or absence of a person using a feature quantity called a HOG feature quantity or a Haar-Like feature quantity. The extraction of the person area by the learning classifier may be performed by using a known one such as an AdaBoost classifier as appropriate, and the details are omitted. FIG. 6A shows a state in which a person area 603 for the intruder 602 has been extracted.

The image processing module 234 extracts the face area and the body area from the extracted person area 603 in order to obtain information used to identify the intruder 602 outside the building B.
As for the face area, as shown in FIG. 6B, the face area 604 can be extracted from an image included in the person area 603 by using an elliptical detection algorithm that seems to be a human head. Alternatively, a learning discriminator for face area extraction may be constructed and applied in the same manner as the person area 603 is extracted.
With respect to the body region, the body region 605 can be extracted by removing the face region 604 from the person region 603.
FIG. 6C shows a face image 606 obtained by cutting out an image included in the face area 604, and FIG. 6D shows a body image 607 obtained by cutting out an image included in the body area 605.

In addition, the image processing module 234 performs a feature point extraction process that represents a characteristic part of the facial part of the face image 606. In order to extract the feature points, a publicly known method may be used as appropriate, but edge information can be extracted from the face image 606 and template matching with edge information of each part of the face can be performed. FIG. 6C schematically shows a state in which eight feature points 608 of the eye corners, the corners of the eyes, the tip of the nose, the left and right mouth corner points, and the mouth points are set. The feature points set in the face image 606 are set at positions corresponding to the feature points set in the 3D face shape data 245, and the number thereof is the number of feature points set in the 3D face shape data 245. The same or more than In the present embodiment, the number of feature points set in the three-dimensional face shape data 245 is extracted as many as the number of feature points.

The image processing module 234 extracts clothes information worn by the intruder 602 from the extracted trunk image 607. The image processing module 234 divides the body image 607 into two in the vertical direction as shown in FIG. And the average of a luminance value is calculated | required about the pixel contained in each area | region, and the color of clothes is determined using the RGB value. For example, when the RGB values are (245, 15, 20), the intensity of the R component is stronger than the other components, so that the intruder 602 is determined to be wearing red clothes. Similarly, when the RGB value is (10, 5, 15), since each color component is a low value, it is determined that the intruder 602 is wearing black clothes.
Further, the image processing module 234 performs frequency analysis on the two-dimensional image with reference to pixel values included in regions corresponding to the upper body and the lower body, and extracts texture information regarding clothes. For example, if it is determined that a lot of straight line components are included in the height direction of the person, “vertical striped clothes” is determined, and if it is determined that no straight line component is included in any direction, “solid clothes” is determined.
As the frequency analysis method, a known method such as Fourier transform or Gabor wavelet transform can be appropriately employed.

The image processing module 234 calculates the height of the intruder 602. For this purpose, the image processing module 234 refers to shooting parameters related to the in-building camera 7 such as the focal length and the depression angle, and based on the position and size of the person region 603 in the in-building image 600, based on a known method, the intruder 602 Calculate your height. Since a specific method may be based on a geometric calculation known in the field of imaging technology, it will be omitted.
The clothing information and height information of the upper body and lower body of the intruder 602 are temporarily stored in the storage unit 24.

The image processing module 234 combines the face image 606 and the 3D face shape data 245 stored in the storage unit 24 to create a 3D face model, and generates a reference face image used for face image matching. . The process will be described with reference to FIG.
FIG. 7A schematically shows three-dimensional face shape data 245. Similarly to the face image 606 shown in FIG. 6C, the three-dimensional face shape data 245 includes 8 parts of the corners of the eyes, the corners of the eyes, the tip of the nose, the left and right corners of the mouth, and the mouth points. A feature point 2451 is set at each location.
Then, the image processing module 234 performs alignment processing by rotating and expanding / contracting the three-dimensional face shape data 245 based on the feature points of the face image 606 and the three-dimensional face shape data 245, and determines that the best alignment has been performed. Then, a synthesis process for pasting the face image 606 to the 3D face shape data 245 is performed to obtain a 3D face model. As a method for aligning the two-dimensional face image and the three-dimensional face shape data with the feature point as a reference, a known method may be appropriately employed.

Then, the image processing module 234 generates a reference face image obtained by adjusting the obtained three-dimensional face model to face orientations at a plurality of angles and projecting it on a two-dimensional plane, and temporarily stores it in the storage unit 24. Note that the feature point 608 extracted from the face image 606 before synthesis is also projected onto the reference face image and set.
FIG. 7B schematically shows a reference face image 700. As shown in the figure, the reference face image 700 is a three-dimensional face model obtained by synthesizing the face image 606 and the three-dimensional face shape data 245 in various directions. Rotated 10 degrees and 20 degrees, and further rotated 15 degrees and 30 degrees downward so as to make the face look darker.
For example, the reference face image indicated as “A7” in the figure is a two-dimensional image obtained by rotating a three-dimensional face model 10 degrees to the left and 15 degrees downward.

These rotation angle ranges and step angles are only examples, and may be set to other values, but if the face is rotated too much, facial features will not be included in the face image. Can not. Further, the range of the rotation angle affects the process of setting the target position at the height and how far the flying robot 3 is set with respect to the person in the target position setting means a. It is necessary to determine the range that can withstand authentication as a design parameter. In the present embodiment, the angle is set to 20 degrees in the left-right direction and 30 degrees in the downward direction, and the target position is determined by the target position setting means A accordingly.

The reference face image 700 is created and used for the face image collation process because the flying robot 3 captures the face image acquired by the in-building camera 7 in order to capture the person while flying around the person. This is because it is not always possible to acquire a face image having the same face direction as the face direction. Therefore, if the reference face image 700 of various orientations is generated using the three-dimensional face shape data 245, any one of them is considered to be roughly aligned with the face orientation of the face image acquired by the flying robot 3, so that the accuracy is high. Verification is possible.

The reference face image 700 may be generated with the face facing upward, but as can be seen from FIG. 1, the flying robot 3 shoots as if looking down from above the person. It can be considered that the image captured by is taken with the face facing forward or downward with respect to the flying robot 3. Therefore, from the viewpoint of reducing the processing amount and the storage capacity, it is possible to cope with the problem by generating a reference face image with the face facing forward and downward as shown in FIG.

The image processing module 234 performs face image extraction, clothing information extraction, and height information calculation on the outdoor image captured by the flying robot 3 by the same processing as the processing performed on the in-building image captured by the in-building camera 7. . The same applies to extracting feature points from a face image.
In calculating the height information, information on the focal length and the depression angle from the horizontal direction as the shooting parameters of the camera 36 is necessary. However, if the flying robot 3 always takes images in a horizontal state, the design parameters are used. It can be known. Moreover, what is necessary is just to refer the output of the altitude sensor 34 with which the flight robot 3 is provided for the height from the ground. What is necessary is just to grasp | ascertain a positional relationship with reference to the output of the ranging sensor 35, or the output of the laser sensor 4 for the distance with the person used as imaging | photography object.

The image processing module 234 includes the face image, clothing information, and height information extracted from the in-building image captured by the in-building camera 7, and the face image, clothing information, and height information extracted from the outdoor image captured by the flying robot 3. Perform verification.
The image processing module 234 uses each of the reference face images 700 generated by using the face image extracted from the in-building image and temporarily stored in the storage unit 24 as the collation of the face image, and the outdoor photographed by the flying robot 3. The face image extracted from the image is collated.
That is, feature amounts such as edge information and color information are extracted from pixels included in a small area centered on each feature point on the reference face image 700, and corresponding feature points on the face image captured by the flying robot 3 are extracted. Similarly, the feature amount is calculated. When the sum of the differences between the feature amounts for each feature point is equal to or smaller than the threshold value, it is assumed that in the collation by the face image, the persons shown in the two face images are the same person.
Although there are a plurality of reference face images 700 as shown in FIG. 7B, if the sum is equal to or less than a threshold value in any of the reference face images, it is determined that the person is the same in the collation using the face images. Since various methods for collating face images are well known, any method may be used.

The image processing module 234 compares the clothing information extracted from the in-building image with the clothing information extracted from the outdoor image, and performs matching based on the clothing information. The collation based on the clothes information is based on whether or not the colors and textures of the upper body clothes and the colors and textures of the lower body clothes match. For example, the color of the upper body clothes, the texture of the upper body clothes, the color of the lower body clothes, and the texture of the lower body clothes are arranged in order. When the image is (black, vertical stripes, black, plain), it is determined that they are the same person by collation based on clothes information. On the other hand, for example, in the case of an image in a building (black, vertical stripes, black, plain), and in the case of an outdoor image (gray, plain, dark blue, plain), it is not the same person in matching by clothes information judge.

The image processing module 234 compares the height information calculated from the in-building image with the height information calculated from the outdoor image, and performs collation based on the height information. The collation based on the height information is determined based on whether or not the difference between the height values calculated from both images is within a predetermined threshold. For example, when the difference is within 5 centimeters, it is determined that they are the same person in the collation based on the height information, and when the difference exceeds 5 centimeters, it is determined that they are not the same person.

The image processing module 234 collates each of the face image, clothes information, and height information, and if all match, the person shown in the building image and the person shown in the outdoor image are the same person. If any of them is different, it is determined that they are not the same person. Then, the determination result is output to the flying robot control module 233.

Returning to FIG. 3, the laser sensor 4 is installed outdoors, and monitors the approach of the parking area in the monitoring area E and the surroundings of the building B as an object sensor. FIG. 8 is a diagram showing a detection area of the laser sensor 4. As shown in the figure, the laser sensor 4-1 is installed from the upper left of the monitoring area E as the detection area in the direction of the building B, and the laser sensor 4-2 is detected from the lower right of the monitoring area E in the direction of the building B. It is installed so that.

The laser sensor 4 transmits a search signal that is a laser beam in a radial manner so as to scan a preset detection area, and receives the search signal that is reflected back to the object in the detection area. Then, the distance to the object is calculated from the time difference between transmission and reception, and the direction in which the search signal is transmitted and the calculated distance are obtained.

And the laser sensor 4 transmits the result of the scanning unit which scanned the detection area with the predetermined period to the security apparatus 2. As a result, the laser sensor analysis module 231 of the security device 2 can perform the outdoor object placement situation in the monitoring area E, the presence or absence of a person, tracking, and the like. In this embodiment, since the purpose is to monitor the approach of a person existing on the ground, scanning is performed in one step in the horizontal direction. However, depending on the purpose of monitoring, scanning in a plurality of steps may be performed in the vertical direction. Also good.

Next, an operation image of the monitoring system 1 configured as described above will be described with reference to FIG. FIG. 1A shows a state where an intruder has entered the building B during the security set mode.
When an intruder breaks the window glass and enters the building B, the sensor detects the breakage of the window glass or a human intrusion and notifies the security device 2. In FIG. 1A, the in-building sensor 5f detects that the intruder has broken the window glass, and the in-building sensor 5d configured by a passive infrared human body detection sensor or the like detects the intruder. A state is shown in which an in-building camera 7a that is set so as to be able to image a range that is approximately the same as or including the detection range of the in-building sensor 5d captures an intruder.
In response to the reaction of the in-building sensor 5f and the in-building sensor 5d, the flying robot 3 starts flying in preparation for photographing an intruder.

FIG. 1B shows a situation when a person who appears to be an intruder appears from the building B. In this embodiment, such a person who may be an intruder is called a candidate intruder. First, when a candidate intruder comes out of the building B, based on a signal from the laser sensor 4, the laser sensor analysis module 231 of the security control device 23 obtains the position of the candidate intruder in the monitoring space. Then, the security device 2 controls the flying robot 3 to fly the candidate intruder's face to a position where it can be photographed with a depression angle as small as possible, and shoots the whole body including the candidate intruder's face. .
The monitoring system 1 to which the imaging system according to the present invention is applied determines whether or not the candidate intruder in FIG. 1B matches the intruder in FIG.

Next, after the intruder has entered the building B, the procedure for calculating the features used for verification in the image processing module 234 will be described with reference to the flowchart of FIG.
First, when the surveillance area E and the building B are in the security set state, when the intruder breaks the window glass and enters the building B, the in-building sensor 5 installed on the window glass reacts and is considered to be in an abnormal state. A signal indicating that is transmitted. Accordingly, the in-building camera 7 installed in the room in which the in-building sensor 5 is installed acquires an in-building image including an intruder and transmits it to the security device 2 (S91).

The image processing module 234 extracts a person area from the in-building image transmitted from the in-building camera 7 (S92, see FIG. 6A).
The image processing module 234 calculates the height of the intruder using the person region extracted in step S92 and the shooting parameters of the in-building camera 7 (S93).
The image processing module 234 extracts a face area and a torso area from the person area extracted in step S93 (S94, see FIG. 6B).
The image processing module 234 combines the face image included in the face area extracted in step S93 with the 3D face shape data 245 stored in the storage unit 24 to generate a 3D face model (S95).
The image processing module 234 generates a reference face image 700 obtained by adjusting the three-dimensional face model generated in step S95 in various directions and projecting it on a two-dimensional plane (S96, see FIG. 7B), and a storage unit 24 is temporarily stored (S98).

Further, the image processing module 234 extracts the clothing information of the upper body and the clothing information of the lower body from the body image included in the body region extracted in step S94 (S97), and temporarily stores them in the storage unit 24 (S98).

Next, processing for controlling the flying robot 3 by the security device 2 will be described with reference to FIG. FIG. 10 is a processing flow of the security device 2 after the in-building sensor 5 detects an intruder while the monitoring area E is in the security set state.
First, when an intruder is detected by the in-building sensor 5, the target position setting means a is a candidate intruder who may be an intruder as a person existing outside the building B analyzed by the laser sensor analysis module 231. And the current position of the candidate intruder is specified (S101).
Then, the altitude of about 3 m above the ground and 3 m or more from the current position of the candidate intruder is set as the target position (S102).

Then, the flight path calculation means b calculates the flight path by a predetermined path search algorithm using the target position set in step S101, the current position of the flying robot 3, and the monitoring space map 241. In the predetermined route search algorithm, when the current position and the target position are set, the route that can be reached safely and in the shortest is calculated in consideration of the arrangement state of the monitoring space map 241 and the size of the flying robot 3 (S103). ).
Since the flying robot 3 is located at a predetermined standby position until the activation signal is received, that position is the current position. At other times, the flight position calculation means D receives the scan data acquired by the scanning means 383 of the flying robot 3 and calculates a place where the scan data matches the monitoring space map 241. The current position is calculated. In the present embodiment, the current position is calculated based on the scan data. However, the present invention is not limited to this, and the flying robot 3 is provided with a GPS signal receiving function, and the current position is calculated based on the GPS signal. Also good.

Next, the robot control means C calculates the flight control signal of the flying robot 3 so that the flying robot 3 can fly along the route calculated by the flight path calculation means B. A specific flight control signal is the number of rotations of each of the four rotors 32 in the flying robot 3. Then, a flight control signal is transmitted by radio signal from the flying robot communication unit 25 (S104).

When the flying robot 3 receives the flight control signal from the antenna 31, the flying robot 3 flies based on the received flight control signal. Specifically, the flight control signal received from the antenna 31 is input to the flight control means 384, and the number of rotations of each rotor 32 is individually controlled from the rotor drive unit 33 to fly.

The flying robot control module 233 compares the target position set by the target position setting means A in step S102 with the current position of the flying robot 3 calculated by the flying position calculation means D, and whether the flying robot has arrived at the target position. It is determined whether or not (S105). If it is determined that the vehicle has not arrived at the target position, the process returns to step S94 to continue the flight control. If it is determined that the vehicle has arrived at the target position (Yes in S105), the flying robot 3 captures a candidate intruder present in the monitoring area E, temporarily stores the acquired outdoor image in the storage unit 2, and performs verification. Processing is performed (S106). Details of the processing in step S106 will be described later with reference to FIG.
When there are a plurality of candidate intruders in the monitoring area E, the process is repeated from step S101 for each candidate intruder.

Here, although not described in the flow shown in FIG. 10, when the flying robot 3 first receives the flight control signal, the scanning unit 383 activates the distance measuring sensor 35 and the altitude sensor 34, and scan data is transmitted to the security device 2. Start sending.

Next, the processing of S106 in FIG. 10 will be described using the flowchart of FIG.
First, the camera 36 mounted on the flying robot 3 acquires an outdoor image including a candidate intruder and transmits it to the security device 2 (S111).

The image processing module 234 extracts a person area from the outdoor image transmitted from the flying robot 3 (S112). This is the same as step S92 in FIG.
The image processing module 234 calculates the height of the candidate intruder using the person region extracted in step S111, the imaging parameters of the camera 36 mounted on the flying robot 3, the current position of the flying robot 3, and the like. The height of the intruder shown in the image in the building temporarily stored in the unit 24 is checked (S113). When the difference in height is within a predetermined threshold (for example, 5 cm), it is determined that the person is the same person in the collation based on the height information.
The image processing module 234 extracts a face area and a torso area from the person area extracted in step S112 (S114). This is the same as step S94 in FIG.

The image processing module 234 compares the face image included in the face area extracted in step S114 with each of the reference face images 700 temporarily stored in the storage unit 24, and matches any of the reference face images. Furthermore, it is determined that the same person is shown in the collation using the face image (S115).
The image processing module 234 extracts clothing information from the body image included in the body region extracted in step S114, and collates it with the clothing information temporarily stored in the storage unit 24 (S116). If the clothes information matches, it is determined that the person is the same person in the matching by the clothes information.

If the image processing module 234 determines that the person is the same person in any of the collation by the face image, the collation by the height information, and the collation by the clothes information, the person in the building image and the person in the outdoor image are the same. It is determined that the person is a person. That is, it is determined that the intruder who has entered the building B has been photographed outside the building B (Yes in S117), and the security control unit 23 displays the fact and the face image, the height information, and the clothing information to the monitoring center communication unit 26. Is sent to the monitoring center 6 via (step S118). Matching by face image is essential, even if it is not determined that all of the collation by face image, the collation by height information, and the collation by clothing information are determined to be the same person. It may be conditional on being determined.

On the other hand, if it is determined in step S117 that they are not the same person (No branch), the process returns to step S111, and the image capturing and collation processes are repeated for the same candidate intruder. If a result with the same person is not obtained even after the predetermined number of collations, it is determined that the candidate intruder photographed by the flying robot 3 is not an intruder who has entered the building B. Then, in order to ensure trail performance, the photographed outdoor image is transmitted to the monitoring center. Furthermore, when there is another person in the monitoring area E, the position of the person is reset to the target position, and then the flying robot 3 flies to the vicinity of the person and performs the processing from step S111. When it is determined that all candidate intruders existing in the monitoring area E are not intruders, the processes after step S111 are performed again on the already determined candidate intruders. These processes are repeated until one of the candidate intruders is determined to be an intruder or a signal indicating security release is input to the security control unit 23 by the security mode switching unit 21.

Since the candidate intruder outside the building B is not always stationary, it is necessary to track the flying robot 3 as the candidate intruder moves. Therefore, when the laser sensor analysis module 231 detects the movement of the candidate intruder, the processing after step S101 in FIG. 10 is repeated.
That is, the target position setting means a refers to the position after the movement of the candidate intruder by referring to the position after the movement of the candidate intruder outputted by the laser sensor analysis module 231 and the monitoring space map 241. To reset. Then, the flying robot control module 233 tracks the candidate intruder by moving the flying robot 3 to the reset target position by the same processing as the processing in steps S103 and S104. If it is confirmed that the flying robot 3 has moved to the re-set target position, the candidate intruder is photographed and collated by the same process as shown in step S106 and FIG. When the candidate intruder continues to move, the candidate intruder may be photographed and collated at appropriate time intervals or distance intervals.
Further, when it is confirmed by the collation processing in the image processing module 234 that any of the candidate intruders is an intruder who has entered the building B, the above-described tracking process is performed only on the candidate intruder. As a result, the escape route can be determined from the appearance when the candidate intruder goes out of the monitoring area E, and can be used for subsequent investigation information.

In the present embodiment, it has been described that the flying robot 3 starts to operate after the in-building sensor 5 detects an intruder and a signal indicating an abnormal state is input to the security control device 23. However, when the laser sensor 4 detects an intruder and is in an abnormal state, the flying robot 3 may start operation after a signal is input to the security control device 23.
In this case, before the in-building camera 7 acquires the in-building image, the flying robot 3 acquires the outdoor image, acquires the personal features of the candidate intruder, and sets the position in the monitoring area E to the laser sensor 4. Continue to grasp at. Then, after the intruder comes out of the building B, among the candidate intruders existing in the monitoring area E, the person who has invaded the building B can be immediately determined, so that quick tracking is possible.

In this embodiment, an example of using one face image for generating a three-dimensional face model in the image processing module has been described. However, when only one face image is used, a characteristic part of the face depends on the orientation of the face. There is a possibility that information is lost in the three-dimensional face model.
Therefore, it is preferable that the image processing module extracts face images from a plurality of in-building images taken by the in-building camera and sequentially synthesizes them into three-dimensional face shape data. At that time, when an upper limit is set on the number of face images used for composition, a face image with a large size is preferentially selected, a face image facing the front is preferentially selected, and conversely, It is preferable to impose conditions such as selecting the directions to be equally different.
Furthermore, when there is a large difference in brightness between facial images during synthesis, features that are not originally found in the face appear at the boundary of synthesis, and it may be possible to adversely affect matching, so there is little difference in brightness. It is preferable to perform correction processing so that the combination of face images is considered or the luminance changes gently near the boundary of synthesis.

Furthermore, when the in-building camera is equipped with a PIR (passive infrared human body detector) type sensor as the in-building sensor for acquiring a plurality of in-building images, the sensor continues to detect intruders. While the image is being acquired, it is possible to keep acquiring the in-building image and select an image suitable for the above processing.

In addition to keeping the distance to the person constant, the flying robot shoots the person multiple times and selects a face image suitable for collation from the photographer. It is also preferable to preferentially use the face image acquired when the warning voice is notified. This is because, as a human psychology, it is considered that when a warning voice is heard, a face is often turned unconsciously in that direction.

Moreover, in this Embodiment, the camera in a building shall use the type which can acquire a color image, and when the inside of the building B was dark, it was suitable to use an illuminating device together. Since this illumination light acquires color information about clothes with a camera in a building, visible light illumination is assumed. Here, when using near-infrared illumination as a lighting device so that an intruder is not aware of it or considering the influence on neighboring residents, a general monochrome type of surveillance camera is used. . In that case, instead of using color information as clothes information, brightness information is used.

1 ... Monitoring system 2 ... Security device (control unit)
3 ... Flying robot 4 ... Laser sensor (object sensor)
5 ... In-building sensor

Claims (3)

A sensor in the building that detects intruders in the building,
An in-building camera for photographing inside the building;
A flying robot for shooting while flying outside the building;
An image processing unit that processes an image captured by the in-building camera or the flying robot, and a control device that includes a flying robot control unit that controls the flight of the flying robot;
Wherein the image processing unit, said after building the sensor detects an intruder, human region where the building camera the flying robots and extracted human region from building images captured is extracted from the captured outdoors image same person Whether or not and output the verification result to the flying robot controller,
The said flying robot control part controls the said flying robot so that the person corresponding to the person area extracted from the said outdoor image may be tracked, if the said image process part determines with the same person.
Install an object sensor that outputs the position of an object that exists in the external area of the building,
The control device further includes an object position calculation unit that calculates a person position based on the output of the object sensor,
The imaging system according to claim 1, wherein the flying robot control unit controls the flying robot so as to photograph the person calculated by the object position calculation unit.
The imaging system according to claim 1, wherein the image processing unit extracts and collates at least one of a face image, clothes information, and height information as a person feature from the person region.

Images