JP6038643B2 - Background information generation apparatus and object detection apparatus - Google Patents

Description

  The present invention relates to a background information generation apparatus that generates background information from an image of a predetermined space and an object detection apparatus that detects an object using the background information generation apparatus.

  In recent years, in order to monitor improper traffic such as co-entrance, research has been conducted to track passersby by processing images taken near the door. At this time, if the both sides of the door are photographed by different monitoring cameras and the tracking of the passers-by passing through the doors is taken over between the monitoring cameras, the behavior of the passers-by can be strictly monitored, so that security is improved.

  For example, Patent Document 1 discloses a traffic control in which a photographing unit is installed in each of an open area outside a door and a standby area inside a door, and a person area is tracked based on a change area extracted by background difference processing. An apparatus is disclosed. In this traffic control device, a door area is set between two areas, and a passerby who disappears from one of the open area and the standby area to the door area and a passerby who appears from the door area in the other of the open area and the standby area. Are taken over by the same passerby.

JP 2010-157149 A

  However, in the prior art, since tracking blank period and blank area occur in the door area, there is a case where tracking cannot be correctly taken over. For example, when two people enter the door area and one of them returns after making a U-turn, or when the person who opened the door passes the subsequent person first, there are multiple people in the door area. Sometimes a problem of erroneous takeover is likely to occur. Therefore, it is desired to continue tracking while a person is present in the door area, and it is necessary to continue extracting the person area of the person in order to continue tracking.

  However, when the door is opened, the other side of the door that was previously shielded by the doorboard is reflected in the opening and the background changes. For this reason, if background difference processing is performed on the image at this time, the entire opening becomes a change region, and it becomes difficult to extract a person region of a person whose part or whole is reflected in the opening. Even if the background image when the door was opened last time is retained, it cannot follow changes in the environment such as lighting fluctuations beyond the door, and many of the openings become changed areas. Is difficult to extract.

  Furthermore, since the door plate is displaced when the door is opened and closed, the door plate itself is included in the change region in addition to the opening. This also makes it difficult to extract a person area near the door.

  The present invention has been made to solve the above-described problem, and a background information generation device capable of continuously generating background information of a monitoring space even when a shielding object such as a door board is displaced in the monitoring space. An object of the present invention is to provide an object detection device capable of continuously detecting a detection object such as a person.

  One aspect of the present invention provides background information for generating background information of the monitoring space that does not include the object in order to detect a predetermined object in the monitoring space in which a shield that can be displaced from a steady position is installed. A first imaging unit configured to capture the monitoring space from a first viewpoint; and when the shielding object is in the steady position in the monitoring space, the shielding object is not captured from the first viewpoint. A second imaging unit that captures an image of a shielding space photographed from the first viewpoint when the object is displaced from the stationary position from a second viewpoint that can be photographed at least when the shielding object is at the stationary position; A displacement detection unit that detects the displacement of the object, a second viewpoint background generation unit that generates background information at the second viewpoint from the second image captured by the second imaging unit, and displacement in the displacement detection unit Not detected In this case, the background information at the first viewpoint is generated from the first image captured by the first imaging unit, and when the displacement is detected by the displacement detection unit, the background generated by the second viewpoint background generation unit. A background information generation apparatus comprising: a first viewpoint background generation unit that uses information as background information at the first viewpoint.

  Further, the displacement detection unit detects a shielding object region in which the shielding object being displaced appears in the second image from a comparison between the second image and a shape feature of the shielding object region registered in advance. In addition, it is preferable that the second viewpoint background generation unit generates the background information at the second viewpoint by excluding the shielding object region from the second image.

  In addition, it is preferable that the second viewpoint background generation unit generates a histogram of an image obtained by imaging the shielding space as background information.

  In addition, the displacement detection unit may be configured such that a shielding cancellation area where the shielding space appears due to the displacement of the shielding object in the first image is a shape characteristic of the shielding cancellation area registered in advance with the first image. It is preferable that the first viewpoint background generation unit detects the background information of the occlusion cancellation region from the background information generated by the second viewpoint background generation unit.

  In addition, it is preferable that the first viewpoint background generation unit generates background information outside the shielding release area from the first image when the displacement of the shielding object is detected.

  In addition, the first image and the background information generated by the first viewpoint background generation unit are compared, and an object that is different from the background information by a predetermined amount or more in the first image is detected as an area of the object. It is preferable to provide a detection unit.

  According to the present invention, it is possible to continue to generate background information even when a shield such as a door plate is displaced. Further, by using the generated background information, it is possible to continue to detect an object such as a person regardless of whether or not the shielding object is displaced.

It is a figure which shows the structure and arrangement | positioning of the target object detection apparatus in embodiment of this invention. It is a figure explaining each area | region in the image in embodiment of this invention. It is a functional block diagram of the target object detection apparatus in embodiment of this invention. It is a flowchart of the target object detection process in embodiment of this invention. It is a figure which shows the example of the door pattern in embodiment of this invention. It is a flowchart of the person area | region detection process in embodiment of this invention. It is a flowchart of the outside door / whole background information generation processing in the embodiment of the present invention. It is a flowchart of the opening part background information generation process in embodiment of this invention. It is a flowchart of the opening part background information generation process in embodiment of this invention. It is a flowchart of the doorboard background information generation process in the embodiment of the present invention. It is a flowchart of another example of the opening part background information generation process in embodiment of this invention.

<Configuration of object detection device>
As illustrated in FIG. 1, the object detection device 100 includes an imaging unit 2 (first imaging unit 2a, second imaging unit 2b), a control unit 3, and a display unit 4.

  The object detection device 100 is installed in a room 6 (first room 6a and second room 6b), generates background information from images obtained by photographing the first room 6a and the second room 6b, and uses the background information. A process for detecting an object is performed.

  The first room 6a and the second room 6b are partitioned by a door plate 15 of the door 5 and a wall. The door plate 15 is installed so as to be displaceable by a hinge or the like. When the door 5 is in the closed state, the door plate 15 remains in the steady position, and the door 5 is opened by the displacement of the door plate 15 from the steady position.

  The first imaging unit 2a is installed in the first room 6a. The first photographing unit 2a photographs the space on the first room 6a side from the first viewpoint every predetermined time, and outputs the photographed image (first image) to the control unit 3. The second imaging unit 2b is installed in the second room 6b. The second photographing unit 2b photographs the space on the second room 6b side from the second viewpoint every predetermined time, and outputs the photographed image (second image) to the control unit 3. In the present embodiment, the first photographing unit 2a and the second photographing unit 2b are disposed downward on the ceilings of the first room 6a and the second room 6b, respectively. As will be described later, from the viewpoint of accuracy and stability of acquisition of background information, the background of the second room 6b included in the field of view (angle of view) of the first photographing unit 2a is the floor 9 when the door 5 is in the open state. It is preferable to install the first photographing unit 2a. For example, the first photographing unit 2a may be installed on the ceiling near the door 5 so that the depression angle is large.

  Here, it is preferable that the shooting timings of the first shooting unit 2a and the second shooting unit 2b are synchronized.

  When the door 5 is closed, the space of the second room 6b including the shielded space 7 is not photographed from the viewpoint (first viewpoint) of the first photographing unit 2a. That is, when the door plate 15 that is a displaceable shield is not displaced, the shielding space 7 is shielded from the first viewpoint by the door plate 15 and is not photographed. However, even when the door 5 is in the closed position, the shielding space 7 is imaged from the second viewpoint by the second imaging unit 2b different from the first imaging unit 2a. On the other hand, when the opening 8 is generated by opening the door 5, a part or all of the shielding space 7 is imaged from the first imaging unit 2 a through the opening 8. As a result, on the first image photographed by the first photographing unit 2a, when the door 5 is in the closed state, the floor 9 of the second room 6b appears as a background in the region where the door plate 15 is the background. . As described above, when the door plate 15 is displaced, the shielding space 7 suddenly appears as a background on the first image, so that it is difficult to detect an object such as a person. Incidentally, when the door 5 is opened, the shielding space 7 becomes a common visual field common to the visual field of the first photographing unit 2a and the visual field of the second photographing unit 2b.

  FIG. 2 shows an example of a first image taken when the door plate 15 is displaced. The opening area 11 is an area where a part of the second room 6b is photographed when the door plate 15 is displaced. The door board area | region 12 is an area | region where the door board 15 is image | photographed. The outside door area 13 is an area excluding the opening area 11 and the door board area 12 in the first image. The door 5, the floor 9, the wall, and the like are backgrounds that are not detection targets, whereas the person 10 (moving object) that is a detection target is a foreground. The person area 14 in which the person 10 appears may overlap over any one or a plurality of areas of the opening area 11, the doorboard area 12, and the outside door area 13.

  In the opening area 11, the background reflected by the displacement of the door plate 15 changes greatly. In the door panel region 12, the background changes due to a change in shadow caused by the displacement of the door panel 15. In addition, when the door board 15 is not displaced, the opening area | region 11 does not appear.

  The second viewpoint background acquisition region 311 is a region in the second image corresponding to the opening region 11 in the first image. That is, the second viewpoint background acquisition region 311 in the second image is a region for acquiring background information of the opening region 11 in the first image. Back projection of the opening area 11 of the first image onto the floor, which is a reference plane of a common coordinate system (world coordinate system) common to the first photographing unit 2a and the second photographing unit 2b, by pre-calculation using the camera parameter 310 Further, the second viewpoint background acquisition region 311 is obtained by projecting the back projection of the opening region 11 onto the second image and stored in the storage unit 3. The area of the floor 9 in FIG. 1 is an example of back projection of the opening area 11, and the second viewpoint background acquisition area 311 in FIG. 2 is an example of projecting back projection of the opening area 11 onto the second image.

  The camera parameters 310 are internal parameters and external parameters of the first photographing unit 2a and the second photographing unit 2b. These values measured by the prior calibration (calibration) work are stored in advance in the storage unit 31 in association with the imaging unit IDs of the first imaging unit 2a and the second imaging unit 2b. The internal parameters are information such as the focal length and the optical center position of the lens. External parameters are information such as position and orientation in the world coordinate system. By applying a pinhole camera model or the like to the camera parameter 310, coordinate conversion between corresponding coordinates in the common field of view of the first imaging unit 2a and the second imaging unit 2b becomes possible.

  When the door plate 15 is displaced, an opening area appears on the second image. The first viewpoint background acquisition area in the second image corresponding to the opening area in the second image is also obtained in advance and stored in the storage unit 31.

  The object detection apparatus 100 is realized by a computer that is the control unit 3 executing an object detection program. The control unit 3 is connected to the first imaging unit 2a, the second imaging unit 2b, and the display unit 4, processes images sequentially input from the first imaging unit 2a and the second imaging unit 2b, and processes the first imaging unit 2a. The person region 14 appearing in the first image and the second image of the second photographing unit 2b is extracted, and the position of the person 10 is tracked from the extraction result. The extraction result and the tracking result are displayed on the display unit 4. The display unit 4 includes a monitor such as a liquid crystal display or a CRT, and a communication interface that receives these data.

  FIG. 3 is a functional block diagram of each function realized by the control unit 3. The control unit 3 functions as an image acquisition unit 30, a storage unit 31, an image processing unit 32, and an output unit 33.

  The image acquisition unit 30 includes a communication interface that sequentially receives images captured by the first imaging unit 2 a and the second imaging unit 2 b and outputs them to the image processing unit 32. The storage unit 31 includes a memory device such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The storage unit 31 includes various data and images including a camera parameter 310, a second viewpoint background acquisition region 311, a door pattern 312, outside / overall background information 313, opening background information 314, doorboard background information 315, and a color conversion function. An object detection program that defines the processing of the processing unit 32 is stored. The image processing unit 32 includes an arithmetic device such as a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or an MCU (Micro Control Unit). The image processing unit 32 reads out the object detection program from the storage unit 31 and executes it to execute processing in each of the door state detection unit 320, the background information generation unit 321, the person area extraction unit 322, and the person tracking unit 323. To do. The output unit 33 includes a communication interface that outputs information processed by the image processing unit 32 to the display unit 4.

  Hereinafter, the object detection process realized by the object detection program will be described with reference to the flowchart of FIG. The object detection process is started after confirming that no person exists in the first room 6a and the second room 6b, which are monitoring spaces. In the object detection process, the following steps S1 to S9 are repeated each time an image is acquired.

  In step S1, the first image and the second image are acquired from the first photographing unit 2a and the second photographing unit 2b. The first imaging unit 2a and the second imaging unit 2b capture the first image and the second image at a predetermined cycle and output them to the image acquisition unit 30. The image acquisition unit 30 receives the first image and the second image and stores them in the storage unit 31.

  In step S2, the state of the door 5 is detected. The door state detection unit 320 detects the opening degree of the door plate 15 by performing pattern matching between the first image and the door pattern 312.

  The door pattern 312 is an image pattern that represents the shape characteristics that the opening area 11 and the door board area 12 can take when the door board 15 is in a steady position and is being displaced. As the door pattern 312, a shape characteristic for each opening degree of the door plate 15 is registered in advance (stored in the storage unit 31). Specifically, the door pattern 312 includes a pattern image showing the position of the contour pixel in the opening area 11 at each opening angle of the door plate 15, the position of the contour pixel in the door board area 12, the pixels belonging to the opening area 11, and the door plate area 12. The pattern image which showed the pixel which belongs to is included. For example, as shown in FIG. 5, 13 levels of image patterns are registered every 10 degrees from 0 degrees (steady position) to 120 degrees (fully open). The door pattern 312 is created in advance by a simulation using a three-dimensional model imitating the door 5 and registered in advance. Further, the shape feature of the opening region 11 and the shape feature of the door plate region 12 may be separately registered. The door state detection unit 320 compares the shape feature of each open angle with the first image of the current frame, and detects the open angle that most closely matches the shape feature as the door state. Specifically, the door state detection unit 320 generates an edge image from the first image, performs pattern matching between the door pattern 312 at each opening angle and the edge image, and detects the opening angle having the maximum degree of coincidence. . Thereby, the displacement state of the door plate 15 which corresponds to the state of the door 5 appearing in the first image of the current frame can be detected. The door state detection unit 320 outputs the detected opening angle to the background information generation unit 321 and the person area extraction unit 322, and stores the detected opening angle in the storage unit 31 for processing in the next frame.

  The door state may be detected by pattern matching between the second image and the door pattern 312. In this case, the door pattern 312 may be registered in advance at each opening angle of the door plate 15 with respect to the second image. Since the opening angle in the first image and the second image is common, the process in step S2 may be performed on either one.

  In step S3, a region attribute is set for each pixel of the first image and the second image. The door state detection unit 320 sets the opening region 11, the door plate region 12, and the other outside door region 13 on the first image and the second image according to the region attribute set in the door pattern 312 with the detected opening angle. To do.

  For example, if the opening angle is other than 0 degree, the region attribute of any one of the opening region 11, the doorboard region 12, and the outside door region 13 is set for each pixel of the first image and the second image, Each of the second images is divided into three regions. If the opening angle is 0 degree, the region attributes of the doorboard region 12 and the outside door region 13 are set in the first image and the second image, and the first image and the second image are each divided into two regions.

  As described above, the door state detection unit 320 (displacement detection unit) registers in advance the opening region 11 (shielding release region) where the shielding space 7 appears in the first image due to the displacement of the door plate 15 (shielding object). It detects from the comparison with the shape characteristic (door pattern 312) of the opening area | region 11. FIG. Thereby, the shape change of the opening area | region 11 which arises with the displacement of the doorplate 15 in a 1st image can be tracked. Therefore, since the opening area 11 to which the opening background information 314 is to be applied is accurately obtained in the first image, it is possible to generate highly accurate background information and detect a highly accurate person area.

  Moreover, the door state detection part 320 (displacement detection part) is the shape characteristic (door pattern) of the doorplate area | region 12 which has registered beforehand the doorboard area | region 12 (shielding object area | region) where the doorboard 15 currently displaced in the 2nd image has appeared. 312). Thereby, the shape change of the doorboard area | region 12 which arises in a 2nd image can be tracked. Therefore, since the doorboard region 12 to be excluded from the second viewpoint background acquisition region 311 is accurately obtained in the second image, it is possible to generate highly accurate background information and detect a highly accurate person region.

  In step S4, a person region is extracted from the first image and the second image. First, the person region extraction unit 322 processes the first image according to the flowchart shown in FIG. In steps S30 to S40, each pixel is sequentially set as a target pixel, and a loop process is performed on all pixels of the first image.

  In step S30, the target pixel is set in the first image. That is, the person region extraction unit 322 sequentially selects one pixel included in the first image and sets it as a target pixel each time the processing returns to step S30.

  In step S31, it is confirmed whether the state of the door 5 is an open state. If the opening angle detected by the door state detection unit 320 is not 0 degree, the person region extraction unit 322 determines that the state of the door 5 is an open state (YES in step S31), and shifts the process to step S32. Otherwise, the process proceeds to step S38 (NO in step S31).

  In step S <b> 32, it is determined whether or not the target pixel is the opening area 11. The person area extraction unit 322 refers to the area attribute of each pixel determined by the door state detection unit 320 in step S4, and shifts the process to step S33 when the area attribute of the target pixel is the opening area 11. (YES in step S32), otherwise, the process proceeds to step S36 (NO in step S32).

  In steps S33 to S35, the person area extraction unit 322 compares each pixel of the second image in the opening area 11 with the opening background information 314 and is an index representing the degree of difference of the pixel with respect to the opening background information 314. A person likelihood (a value increases as the difference is calculated) is calculated.

  The opening background information 314 is background information of the opening area 11 in the first image and the second image. The opening background information 314 of each pixel is information including two types of color histogram (opening color histogram) of the opening area 11 and a background pixel value (opening background pixel value) corresponding to each pixel of the opening area 11. And stored in the storage unit 31. The opening color histogram is, for example, a three-dimensional array of 8 bins each of YUV. The opening background information 314 is generated by the background information generation unit 321 and updated as appropriate. The generation process of the opening background information 314 will be described later.

  In step S33, it is determined whether or not the background pixel value of the target pixel has already been generated. If the background pixel value is already stored in the opening background information 314, the person area extraction unit 322 moves the process to step S35 (YES in step S33), and otherwise moves the process to step S34. Shift (NO in step S33).

In step S34, the person likelihood of the pixel of interest for the aperture color histogram is calculated. The person area extraction unit 322 reads the opening color histogram H _{0 of} the first image from the opening background information 314 and the color histogram H _P of the person near the door 5 from the person information 316, respectively, and formulas (1) and (2) The person likelihood L _O (x) of the pixel of interest x is calculated as follows.

Here, x is the pixel of interest, P _O (x) is the person probability of the pixel of interest x, c _x is the color of the pixel of interest x, H _O (c _x ) is the frequency of bins of the color c _x in the aperture color histogram, H _P (c _x ) is the bin frequency of the color c _x in the color histogram read from the person information 316, L _O (x) is the person likelihood of the pixel of interest x, and β _O1 is appropriately adjusted through previous experiments. It is a constant. The person likelihood L _O (x) increases monotonously with respect to the person probability P _O (x). Therefore, the lower the frequency of the pixel value of the target pixel in the aperture color histogram, the higher the person likelihood.

  As described above, the person area extraction unit 322 calculates the person likelihood of the pixel so high that each pixel of the opening area 11 in the first image is different from the opening color histogram. In addition, the person area extraction unit 322 calculates the person likelihood of the pixel higher as the pixels of the opening area 11 in the first image are more similar to the person information 316.

  Here, the person information 316 is information indicating a history of image feature amounts and positions of the person area 14 of each person (that is, a person being tracked) existing in the monitoring space. Each person is assigned a person ID that uniquely identifies the person, and the person ID, the image feature amount, and the position history are associated with the person ID and stored in the storage unit 31. Here, the image feature amount is a color histogram, and is represented by, for example, a three-dimensional array of 8 bins of YUV. The person area 14 is represented by a likelihood representing the likelihood that a person exists in each pixel constituting the first image. The information on the person area is stored in the form of a three-dimensional shape model in the common coordinate system. The three-dimensional shape model can be data obtained by approximating the head, torso and feet with a spheroid. The three-dimensional shape model becomes a human mask image by projecting it onto the image coordinate system using the camera parameters 310.

In step S35, the person likelihood of the target pixel with respect to the aperture background pixel value is calculated. The person likelihood is calculated using Equation (3). Specifically, the person region extraction unit 322 calculates a background difference between the pixel value I (x) of the target pixel and the background pixel value B _O (x), and calculates the difference value (I (x) −B _O (x )) Is converted into person likelihood L _O (x).

Here, β _O2 is a constant adjusted appropriately through previous experiments.

The person likelihood L _O (x) monotonously increases with respect to the difference value (I (x) −B ₀ (x)). Therefore, the person likelihood increases as the difference between the pixel value of the target pixel and the opening background pixel value increases.

  As described above, the person region extraction unit 322 calculates the person likelihood of the pixel so high that each pixel of the opening region 11 in the first image is different from the opening background information 314.

  In steps S36 and S37, the person area extraction unit 322 compares each pixel of the first image in the doorboard area with the doorboard background information 315, and the person likelihood that is an index representing the degree to which the pixel differs from the background information. Is calculated.

  Here, the doorboard background information 315 is background information of the doorboard region 12 in the first image and the second image, and is represented by a color histogram (doorboard color histogram). The door panel color histogram of each image is expressed in, for example, a three-dimensional array of 8 bins of YUV and stored in the storage unit 31. The door board background information 315 is generated by the background information generator 321 while the displacement of the door board 15 is detected, and is updated as appropriate.

  In step S <b> 36, it is determined whether or not the target pixel is the door panel region 12. If the region attribute of the pixel of interest is the doorboard region 12 (YES in step S36), the person region extraction unit 322 shifts the process to step S37, and otherwise (NO in step S36), step S38. Shift processing to.

In step S37, the person likelihood of the pixel of interest for the door panel color histogram is calculated. The person area extraction unit 322 compares the color of the pixel of interest x, that is, the pixel value, with the door panel color histogram of the first image and the color histogram of the person near the door 5, and the pixel of interest according to equations (4) and (5). The person likelihood L _D (x) of x is calculated.

Here, P _D (x) is the human probability of the pixel of interest x, H _D (cx) is the frequency of the bin of the color c _x in the door panel color histogram, L _D (x) is the human likelihood of the pixel of interest x, β _D is a constant adjusted appropriately through previous experiments. The person likelihood L _D (x) monotonically increases with respect to the person probability P _D (x).

  In other words, the person region extraction unit 322 calculates the person likelihood of the pixel so high that each pixel of the doorboard region 12 in the first image is different from the doorboard background information 315. In addition, the person area extraction unit 322 calculates the person likelihood of the pixel higher as the pixel constituting the door board area 12 of the first image is more similar to the person information 316.

In step S38, the person likelihood is calculated for the pixel of interest that is neither the opening area 11 nor the door panel area 12 being displaced with respect to the outside / overall background pixel value. The person region extraction unit 322 reads the background luminance value B _G (x) corresponding to the target pixel x from the outside / overall background information 313, and the pixel value I (x) of the target pixel x and the background pixel value B _G (x ) And the difference value (I (x) −B _G (x)) is calculated and converted into a person likelihood L _G (x). Specifically, the person likelihood L _G (x) is calculated according to Equation (6).

Here, L _G (x) is a person likelihood of the pixel of interest x, and β _G is a constant adjusted appropriately through a prior experiment. The person likelihood L _G (x) monotonically increases with respect to the difference value (I (x) −B _G (x)). Therefore, the person likelihood increases as the difference between the pixel value of the target pixel and the overall background pixel value or the outside background pixel value increases.

  As described above, when the door panel 15 is not displaced, the person region extraction unit 322 calculates the person likelihood of the pixel as high as it differs from the overall background pixel value with respect to all the pixels of the first image. When 15 is displaced, the person likelihood of each pixel of the first image in the outside door region 13 is calculated so high that the pixel differs from the outside background pixel value.

  Here, the outside / overall background information 313 is background information in the entire first image and background information in the entire second image. As the outside / overall background information 313, a background pixel value corresponding to each pixel of the first image and a background pixel value corresponding to each pixel of the second image are stored in the storage unit 31. The outside / overall background information 313 is generated by the background information generation unit 321 and updated as appropriate. When the door plate 15 is not displaced, the background pixel values of all the pixels are used as the entire background pixel value. When the door plate 15 is displaced, the background pixel value corresponding to the pixel in the outside door region 13 is the background pixel value outside the door. Used as

  In step S39, it is determined whether or not the process of calculating the person likelihood has been completed for all the pixels of the first image. The person area extraction unit 322 shifts the process to step S40 if the process has been completed for all the pixels of the first image, and returns the process to step S30 if not, and returns the person likelihood of the next target pixel. Perform the calculation. Through the processing up to this point, a person likelihood image having the person likelihood calculated for all the images of the first image as the value of each pixel is obtained.

  The person likelihood image obtained in this way becomes information representing a person region. That is, an area where pixels having high person likelihood are densely arranged in a human shape is obtained as the person area 14. In step S40, a process of determining the person region 14 from the person likelihood image is performed.

  Specifically, the person area extraction unit 322 determines an area where pixels having high person likelihood are densely arranged in a human shape as a passing area. The person area extraction unit 322 generates a humanoid mask image by projecting a three-dimensional human shape model at each position on the person likelihood image, and the total value of the person likelihoods set for the pixels in the mask image is An area indicated by the mask image at the position when the detection threshold value is equal to or greater than the detection threshold T1 is extracted as the person area 14. Alternatively, each pixel of the person likelihood image is compared with a predetermined detection threshold T2, and an area in which pixels having a person likelihood equal to or greater than the detection threshold T2 are adjacent to each other within a person size range is extracted as the person area 14. Also good.

  In the above subroutine, only the extraction process of the person area 14 included in the first image has been described, but the person area 14 included in the second image is obtained in the same manner. That is, the person region extraction unit 322 calculates a person likelihood for each pixel of the second image to generate a person likelihood image corresponding to the second image, and pixels having a high person likelihood are densely arranged in a human shape. The area is determined as a traveling area. At this time, if the door plate 15 is being displaced, the person region extraction unit 322 calculates the person likelihood by comparing the pixels of the second image in the opening region 11 with the opening background information 314, and calculates the person likelihood in the door plate region 12. The person likelihood is calculated by comparing the pixels of the two images with the door panel background information 315, and the person likelihood is calculated by comparing the pixels of the second image in the outside door region 13 with the outside / overall background information 313. In addition, if the door panel 15 is not displaced, the person region extraction unit 322 compares all the pixels of the second image with the outside / overall background information 313 to calculate a person likelihood.

  When the process in the person area extraction unit 322 is completed, the process returns to step S5 of the main routine.

  In step S5, a person tracking process is performed. The person tracking unit 323 extracts the position of each person area 14 and the color histogram of the first image in the person area 14 as the feature quantity of the person, and the extracted feature quantity and the feature quantity of each person stored in the person information 316. Is checked. The person tracking unit 323 determines that the person area obtained by the collation is the same as the person in the person information 316 and associates it. The person area extracted from the second image is similarly identified with the person information. Further, the person tracking unit 323 uses the camera parameter 310 to associate the person area 14 extracted from the first image and the person area 14 extracted from the second image with the same person corresponding to the position in the world coordinate system. . The person tracking unit 323 updates the color histogram of the person information 316 with the color histogram of the person area 14 for which the correspondence has been obtained, and adds the position of the person area 14 for which the position history of the person information 316 of the person has been matched. To do. Further, the person information 316 of a new person is generated for the person area for which no correspondence is obtained, and the person information 316 for which no correspondence is obtained is deleted as it disappears.

  The person tracking process is not limited to these, and various existing image tracking processes can be applied.

In step S <b> 6, a person color histogram is generated from the pixels in the person region 14. The histograms H _{1 to} H _K of the K persons existing in the vicinity of the door 5 are added and normalized, and the person color histogram H _P is included in the person information 316 and stored in the storage unit 31.

  In step S <b> 7, generation processing of the outside door / entire background information 313 is performed according to the flowchart of FIG. 7. The background information generation unit 321 sequentially sets each pixel of the first image of the current frame as a target pixel (Step S50), and executes the loop processing of Steps S50 to S55.

  In step S51, it is confirmed whether the state of the door 5 is an open state. If the opening angle detected by the door state detection unit 320 is other than 0 degrees, the background information generation unit 321 shifts the process to step S52 assuming that the door 5 is in the open state (YES in step S51). The process proceeds to S53 (NO in step S51).

  In step S52, it is confirmed whether or not the region attribute of the target pixel is the outside door region 13. If the region attribute set for the target pixel by the door state detection unit 320 is the outside door region 13, the background information generation unit 321 moves the process to step S 53 (YES in step S 52). The process proceeds to S55 (NO in step S52).

  In step S53, it is confirmed whether or not the pixel of interest is a pixel in the person area 14. The background information generation unit 321 refers to the person region 14 extracted from the first image by the person region extraction unit 322, and shifts the process to step S55 if the target pixel is a pixel of the person region 14 (in step S53). YES), otherwise, the process proceeds to step S54 (NO in step S53).

  In step S54, the background information generation unit 321 updates the background pixel value of the target pixel stored in the outside / overall background information 313 with the pixel value of the target pixel of the current frame.

ただし０＜α_G＜１であり、例えばα_G＝０．３と設定する Specifically, the background information generation unit 321 determines the pixel value I (x) of the target pixel of the current frame and the background pixel value B _G (x, t− of the outside / overall background information 313) according to Expression (8). A background pixel value B _G (x, t) obtained by weighting and averaging 1) at a ratio of α _G : (1−α _G ) is obtained and written in the outside / overall background information 313.

However, 0 <α _G <1, and for example, α _G = 0.3 is set.

  In step S55, the background information generation unit 321 checks whether all pixels of the first image have been processed as the target pixel. If all the pixels have not been processed, the process returns to step S50 to process the next pixel (NO in step S55). When all the pixels are processed as the target pixel, the process in FIG. 7 ends.

  As described above, the background information generation unit 321 uses the first image of the current frame for the area excluding the person area 14, the opening area 11, and the displacing doorboard area 12. The entire background information 313 is generated / updated.

  Further, the background information generation unit 321 generates the outside / overall background information 313 in the same manner for the second image. In other words, if the door is in the open state, the background information generation unit 321 corresponds to the pixel value of the outside area 13 where no person is detected in the second image of the current frame and the corresponding background pixel of the outside / overall background information 313. The weighted average value with the value is written in the storage unit 31 as a new background pixel value of the opening area 11 in the second image. Further, if the door 5 is in the closed state, the background information generation unit 321 determines the pixel value of the area where no person is detected in the second image of the current frame and the corresponding background pixel value of the outside / overall background information 313. Are written in the storage unit 31 as new background pixel values for the second image. This process corresponds to the process in the second viewpoint background generation unit.

  When the outside door / overall background information 313 is generated, the process proceeds to step S8 in FIG. 4, and the background information generation unit 321 generates the opening background information 314.

  In step S8, opening background information generation processing is performed according to the flowcharts shown in FIGS.

  First, the background information generation unit 321 generates an opening color histogram in the first image using the second image according to the flowchart of FIG.

  In the loop processing of steps S60 to S64, a color histogram in the second viewpoint background acquisition region 311 of the current frame is generated. In step S60, each pixel of the second image of the current frame is sequentially set as the target pixel. That is, the background information generation unit 321 sequentially selects the pixels included in the second image and sets them as the target pixel every time the process returns to step S50. At the start of the loop processing, the background information generation unit 321 prepares a storage area for storing the color histogram of the current frame, and initializes the frequency of each bin to zero.

  In step S61, it is confirmed whether or not the target pixel is a pixel in the second viewpoint background acquisition region 311. If the target pixel is a pixel in the second viewpoint background acquisition region 311 set for the second image, the background information generation unit 321 moves the process to step S62 (YES in step S61), otherwise. The process proceeds to step S66 (NO in step S61).

  In step S62, it is confirmed whether or not the target pixel is a pixel in the door plate region 12. The background information generation unit 321 shifts the process to step S66 if the pixel of interest is in the doorboard region 12 (YES in step S62), otherwise shifts the process to step S63 (NO in step S62). . By this determination, the background information generation unit 321 excludes the overlapping door panel area 12 from the second viewpoint background acquisition area 311 when the door 5 is opened forward as viewed from the second viewpoint. By this exclusion, the door plate 15 is not mixed with the background, so that the accuracy of the opening color histogram can be maintained high.

  In step S63, it is confirmed whether or not the target pixel is a pixel in the person region 14. The background information generation unit 321 refers to the person area 14 extracted from the second image by the person area extraction unit 322, and shifts the process to step S66 if the target pixel is in the person area 14 (YES in step S63). If not, the process proceeds to step S64 (NO in step S63).

  In step S64, a color conversion process is performed. The background information generation unit 321 uses a color conversion function set in advance so that the average luminance value and variance of the second image match the average luminance value and variance of the first image, and the pixel value of the target pixel of the second image Convert.

  In step S65, the value of the target pixel is accumulated in the color histogram of the current frame. The background information generation unit 321 increases the frequency of the bin corresponding to the color of the pixel of interest in the current frame color histogram prepared in the storage unit 31 by one.

  In step S66, the background information generation unit 321 confirms whether all the pixels of the second image have been processed as the target pixel. If all the pixels have not been processed, the process returns to step S60 to process the next pixel (NO in step 66), and if the process has been completed, the process proceeds to step S67.

ただし０＜α_O1＜１であり、例えばα_O1＝０．３と設定する。 In step S67, the background information generation unit 321 updates the color histogram stored in the opening background information 314 using the color histogram of the current frame. Specifically, the background information generation unit 321 normalizes the color histogram of the current frame with the number of pixels to be processed, and the value of each bin of the color histogram H of the current frame and the opening background information 314 according to Expression (9). the color histogram H _O (t-1) corresponding to the value of the bin alpha _O1: write to background information generation unit 321 obtains the (1-α _O1) color histogram H _O (t) weighted average rate.

However, 0 <α _O1 <1 and, for example, α _O1 = 0.3 is set.

If the color histogram is not stored in the opening background information 314, the color histogram H of the current frame is stored as H 2 _O as it is.

  Steps S60 to S66 for generating the background information of the shielding space 7 from the second image correspond to the processing of the second viewpoint background generation unit, and step S67 for generating the opening color histogram used in the first image from the background information is the first. This corresponds to the processing of the single viewpoint background generation unit. As described above, even when the door panel 15 (the shielding object) is not displaced, the second viewpoint background generation unit can adapt to the illumination variation of the shielding space 7 by generating the background information of the shielding space 7, and this background information is used. Then, the first viewpoint background generation unit generates the opening color histogram so that the door 15 is displaced and the shielding space 7 that has not been photographed from the first viewpoint until now is the opening area 11 (shielding cancellation area) of the first image. ) Can be detected from the first viewpoint, and the region of the person 10 (moving object) can be continuously detected with high accuracy using the aperture color histogram adapted to the illumination variation. In addition, since the background information of the shielding space 7 is not information on a pixel basis but information on a single region (histogram), errors in the camera parameters 310 and the color conversion function are diluted, and a reduction in accuracy can be prevented.

  Next, the background information generation unit 321 generates an opening background pixel value in the first image according to the flowchart of FIG.

  In step S70, it is confirmed whether or not the door 5 is open. If the opening angle detected by the door state detection unit 320 is other than 0 degrees, the background information generation unit 321 determines that the door 5 is in the open state and shifts the process to step S71 (YES in step S70). If so, the process proceeds to step S76 (NO in step S70).

  In step S71, the background information generation unit 321 sequentially sets each pixel of the first image of the current frame as a target pixel. In the loop processing in steps S71 to S75, the opening background pixel value is generated / updated from the first image of the current frame.

  In step S72, it is confirmed whether or not the target pixel is a pixel in the opening area 11. If the region attribute of the target pixel set for the first image by the door state detection unit 320 is the opening region 11, the background information generation unit 321 moves the process to step S73 (YES in step S72). If not, the process proceeds to step S75 (NO in step S72).

  In step S <b> 73, it is confirmed whether or not the target pixel is a pixel in the person area 14. The background information generation unit 321 refers to the person region 14 extracted from the first image by the person region extraction unit 322, and shifts the process to step S75 if the target pixel is in the pixel of the person region 14 (in step S73). YES), otherwise, the process proceeds to step S74 (NO in step S73).

ただし０＜α_O2＜１であり、例えばα_O2＝０．３と設定する。 In step S74, the background pixel value of the target pixel is updated. The background information generation unit 321 updates the background pixel value of the target pixel stored in the aperture background information 314 with the pixel value of the target pixel of the current frame. Specifically, the background information generation unit 321 calculates the pixel value I (x) of the pixel of interest in the current frame and the background pixel value B _O (x, t−1) of the opening background information 314 according to Expression (10). A background pixel value B _O (x, t) obtained by weighted averaging at a ratio of α _O2 : (1−α _O2 ) is obtained and written in the opening background information 314.

However, 0 <α _O2 <1 and, for example, α _O2 = 0.3 is set.

  In step S75, it is confirmed whether all pixels of the first image have been processed as the target pixel. If all the pixels have not been processed, background information generation unit 321 returns the process to step S71 to process the next pixel (NO in step S75), otherwise the process proceeds to step S7 in FIG. Return (YES in step S75). By the processing of steps S71 to S75, the background pixel value of the shielding space 7 is generated in a portion where no person appears in the opening area 11, and can be used instead of the color histogram.

  In step S76, it is confirmed whether or not the door 5 has changed from the open state to the closed state. If the opening angle of the previous frame stored in the storage unit 31 is other than 0 degree and the opening angle of the current frame is 0 degree, the background information generation unit 321 is in the open state in the current frame. If it is changed from the closed state to the closed state, the process proceeds to step S77 (YES in step S76). Otherwise, the process proceeds to step S8 in FIG. 4 (NO in step S76).

  In step S77, the aperture background pixel value is cleared. The background information generation unit 321 clears all the background pixel values of the opening portion background information 314. As a result, the background information of the shielded space 7 can be updated to the latest state without any error in coordinate conversion or color conversion, so that it is possible to detect an object with higher accuracy corresponding to illumination variations. That is, when it is determined in step S70 that the door 5 is in the closed state, the latest background pixel value in the opening region 11 cannot be obtained from the first image, and the illumination fluctuation during the closed state cannot be handled. While the closed state continues, the background information generation unit 321 does not generate the background pixel value in the opening region 11, and further clears the opening background pixel value so that the door 5 receives the opening background pixel value. It can be used only while it is open.

  The background information generation unit 321 also generates the opening background information 314 in the same manner for the opening area 11 in the second image. That is, if the door 5 is in the open state, the background information generation unit 321 calculates a color histogram of the current frame from the pixels of the opening area 11 where no person is detected in the first image of the current frame, and the color histogram And the weighted average value of the color histogram stored in the opening portion background information 314 are written in the storage unit 31 as a new color histogram of the opening region 11 in the second image. In addition, if the door 5 is in the open state, the background information generation unit 321 includes the pixel value of the opening area 11 in which no person is detected in the second image of the current frame and the corresponding background pixel value of the opening background information 314. Is written in the storage unit 31 as a new background pixel value of the opening area 11 in the second image. The background information generation unit 321 clears the background pixel value of the opening background information 314 when the door 5 changes from the open state to the closed state in the current frame.

  When the opening background information 314 is generated, the process proceeds to step S9 in FIG. 4, and the background information generation unit 321 generates the doorboard background information 315. The background information generation unit 321 executes the loop process of steps S80 to S84 to generate a color histogram in the door panel region 12 of the current frame.

  In step S80, the background information generation unit 321 sequentially sets each pixel of the first image of the current frame as a target pixel. Further, the background information generation unit 321 prepares a storage area for storing the color histogram of the current frame, and initializes the frequency of each bin to zero.

  In step S81, the background information generation unit 321 confirms whether or not the target pixel is a pixel of the doorboard region 12 set for the first image. If the pixel of interest is within door panel region 12, the process proceeds to step S82 (YES in step S81), otherwise the process proceeds to step S84 (NO in step S81).

  In step S <b> 82, the background information generation unit 321 refers to the person region 14 extracted from the first image by the person region extraction unit 322 and confirms whether the pixel of interest is a pixel in the person region 14. If the pixel of interest is in person area 14, the process proceeds to step S84 (YES in step S82), otherwise the process proceeds to step S83 (NO in step S82).

  In step S <b> 83, the background information generation unit 321 accumulates the color histogram H of the current frame in the storage unit 31 by increasing the frequency of the bin corresponding to the color of the target pixel by 1 by setting the target pixel as the target of color histogram generation. To do.

  In step S84, the background information generation unit 321 checks whether all pixels of the first image have been processed as the target pixel. If all the pixels have not been processed, the process returns to step S80 to process the next pixel (NO in step S84). If not, the process proceeds to step S85 (YES in step S84).

ただし０＜α_D＜１であり、例えばα_D＝０．３と設定する。 In step S85, the background information generation unit 321 updates the color histogram stored in the doorboard background information 315 using the color histogram of the current frame. Specifically, the background information generation unit 321 normalizes the color histogram of the current frame with the number of pixels targeted for processing, and the value of each bin of the color histogram H of the current frame and the doorboard background information according to the equation (11). A color histogram H _D (t) obtained by weighting and averaging the bin values corresponding to the color histogram H _D (t−1) 315 at a ratio of α _D1 : (1−α _D1 ) is written in the door panel background information 315. . If the color histogram is not stored in the door panel background information 315 immediately after the start-up, the color histogram H of the current frame is stored as _HD as it is.

However, 0 <α _D <1 and, for example, α _D = 0.3 is set.

  In addition, the background information generation unit 321 similarly generates the door panel background information 315 for the door panel region in the second image. That is, the background information generation unit 321 calculates a color histogram of the current frame from the pixels in the door panel area 12 where no person is detected in the second image of the current frame, and stores the color histogram and the door panel background information 315. The weighted average value with the current color histogram is written in the storage unit 31 as a new color histogram of the doorboard region 12 in the second image.

  When the above process is completed, the process returns to step S1 in FIG. 4 to process the next frame.

  Instead of the flowchart of FIG. 8, an opening color histogram in the first image may be generated using the second image according to the flowchart of FIG.

  In steps S90 to S94, each pixel of the outside / overall background information 313 generated from the second image is sequentially set as the pixel of interest, and an aperture color histogram is generated by loop processing for all the pixels. In step S90, the background information generation unit 321 sequentially sets the background pixel value of the outside / overall background information 313 generated from the second image of the current frame as the target pixel value. At the start of the loop processing, the background information generation unit 321 initializes the frequency of each bin in the opening color histogram to 0.

  In step S91, it is confirmed whether the pixel of interest is a pixel in the second viewpoint background acquisition region. In step S93, an opening color histogram is generated. The aperture color histogram is generated by increasing the frequency of the bin corresponding to the color of the target background pixel value by one.

  If the target pixel value is the pixel value of the second viewpoint background acquisition region 311 set for the second image, the background information generation unit 321 shifts the process to step S92 (YES in step S91), and so on. If not, the process proceeds to step S94 (NO in step S91).

  In step S92, color conversion is performed. The background information generation unit 321 converts the target background pixel value using a color conversion function from the second image to the first image.

  In step S94, it is confirmed whether all the pixels in the outside / overall background information 313 generated from the second image have been processed as the target pixel. If all the pixels have not been processed, the process returns to step S90 to process the next pixel (NO in step 94). If the process has been completed, the background information generation unit 321 normalizes the aperture color histogram. Then, the process proceeds to step S70.

  When the flowchart of FIG. 11 is followed, the process of the second viewpoint background generation unit that generates the background information of the shielded space 7 from the second image corresponds to the process according to FIG. The process of FIG. 11 for generating the opening color histogram from the outside / outdoor background information 313 of the two images corresponds to the process of the first viewpoint background generation unit.

  As described above, according to the present embodiment, when an object such as a person is detected in a space in which a displaceable shielding object such as a door is installed, the object is shielded with the displacement of the shielding object. The background information in which the space is reflected can be appropriately updated. Accordingly, it is possible to prevent the entire opening from becoming a change area when the background difference process is performed, and it is possible to appropriately detect an object in which a part or the whole is reflected in the opening. Moreover, even when there is a change in the environment such as illumination fluctuation in the shielded space, the background information of the opening can be appropriately updated, and the object can be detected. Furthermore, it is possible to continue tracking while the object passes through the opening, and the object tracking process can be reliably performed.

  In the present embodiment, an example of a hinged door that opens to the front is shown, but the present invention can be applied to various doors such as a hinged door, a sliding door, and a folding door that are opened to the back by adapting the door pattern 312. Furthermore, in addition to the door, the present invention can also be applied to a shield such as a container of an article storehouse or a train at a yard.

  Moreover, although this Embodiment demonstrated the aspect which implement | achieves the function of each part of a target object detection apparatus with one computer, it is not limited to this. The functions of each part of the object detection device can be realized by controlling a general computer by a program, and each function of these devices may be appropriately combined and processed by one computer. Distributed processing may be performed by a plurality of computers connected via a network or the like.

  2 shooting unit, 2a first shooting unit, 2a second shooting unit, 3 control unit, 4 display unit, 5 door, 6 rooms, 6a first room, 6b second room, 7 opening, 7 shielding space, 8 opening part, 9 floor, 10 person, 11 opening area, 12 doorboard area, 13 outdoor area, 14 person area, 30 image acquisition part, 31 storage part, 32 image processing part, 33 output part, 100 object detection device, 310 camera parameters, 311 second viewpoint background acquisition region, 312 door pattern, 313 outside / overall background information, 314 opening background information, 315 doorboard background information, 316 person information, 320 door state detection unit, 321 background information generation Part, 322 person area extraction part, 323 person tracking part.

Claims (6)

A background information generating device that generates background information of the monitoring space that does not include the object in order to detect a predetermined object in the monitoring space in which a shield that can be displaced from a steady position is installed,
A first photographing unit for photographing the monitoring space from a first viewpoint;
When the shielding object is in the steady position in the monitoring space, at least the shielding space captured from the first viewpoint when the shielding object is displaced from the steady position without being photographed from the first viewpoint. A second photographing unit for photographing from a second viewpoint that can be photographed when the object is in the stationary position;
A displacement detector for detecting the displacement of the shield;
A second viewpoint background generation unit that generates background information at the second viewpoint from the second image captured by the second imaging unit;
When no displacement is detected in the displacement detection unit, the first image capturing unit generates background information from the first image captured, and when the displacement detection unit detects displacement, A first viewpoint background generation unit that uses the background information generated by the second viewpoint background generation unit as background information at the first viewpoint;
A background information generating apparatus comprising: The background information generating apparatus according to claim 1,
The displacement detection unit detects a shielding object region in which the shielding object being displaced appears in the second image from a comparison between the second image and a shape characteristic of the shielding object region registered in advance.
The second viewpoint background generation unit generates background information at the second viewpoint by excluding the shield region from the second image. The background information generation apparatus according to claim 1 or 2,
The second viewpoint background generation unit generates a histogram of an image obtained by imaging the shielding space as background information. The background information generation device according to any one of claims 1 to 3,
The displacement detection unit compares the first image with a shape characteristic of the shield release area registered in advance in the first image and the shield release area where the shield space appears due to the displacement of the shield. Detect
The first viewpoint background generation unit generates background information of the shielding release area from background information generated by the second viewpoint background generation unit. The background information generation device according to claim 4,
The first viewpoint background generation unit generates background information outside the shielding release area from the first image when displacement of the shielding object is detected. The background information generation device according to any one of claims 1 to 5,
An object detection unit that compares the first image with the background information generated by the first viewpoint background generation unit and detects an area that differs from the background information by a predetermined amount or more in the first image as the area of the object. An object detection apparatus comprising:

Images