JP6568772B2 - Image processing apparatus, image processing system, operation method of image processing apparatus, and control program - Google Patents

Description

  The present invention relates to image processing.

  As described in Patent Documents 1 to 6, various image processing apparatuses have been conventionally proposed.

JP 2004-56488 A Japanese Patent Laying-Open No. 2015-53011 Japanese Patent Laying-Open No. 2015-70359 Japanese Patent No. 3423861 Japanese Patent No. 3423886 Japanese Patent No. 3569163

  In the image processing apparatus, it is desired to reduce the processing load.

  Therefore, the present invention has been made in view of the above points, and an object thereof is to provide a technique capable of reducing the processing load of an image processing apparatus.

An aspect of the image processing apparatus includes: a first processing unit that performs a fitting process for applying a figure to each of the plurality of input images in which the objects are captured at different times; A second processing unit that performs a tracking process for tracking the position of the object in the input image by tracking the position of the graphic on the input image based on the result of the fitting process ; The first processing unit applies a figure to the object shown in the input image, and the third processing unit applies a plurality of figures to one object. Includes a fourth processing unit that employs one new figure obtained by integrating the plurality of figures in place of the plurality of figures .

In one aspect of the image processing apparatus, a fifth processing unit is further provided for determining the number of the objects appearing in the input image based on the result of the tracking process.

  In one aspect of the image processing apparatus, the second processing unit generates graphic information related to the graphic, and the second processing unit updates the graphic information based on a result of the fitting process. I do.

  In one aspect of the image processing apparatus, the graphic information includes state information indicating a state of the graphic and graphic specifying information for specifying the graphic.

  Further, in one aspect of the image processing apparatus, the second processing unit performs the first time for each first graphic obtained in the fitting process for the first input image in which the object is captured at the first time. A correspondence specifying process for specifying a corresponding second graphic corresponding to the first graphic obtained in the fitting process for the second input image in which the object is captured at a second time later than the second time; The two processing units perform the update process based on the result of the correspondence specifying process.

  In one aspect of the image processing device, the second processing unit is obtained in the fitting process for the second input image when specifying the corresponding second graphic corresponding to the first graphic. Among the second graphics, the second graphics that are closest to the first graphics, and are second graphics whose first distance from the first graphics is equal to or less than a first threshold value. Is specified as the corresponding second graphic.

  Further, in one aspect of the image processing device, the second processing unit is configured to determine the first input image based on a result of the fitting process for the first and second input images in the correspondence specifying process. It is determined whether or not one second graphic is applied in the fitting process for the second input image with respect to a plurality of objects to which the plurality of first graphics obtained in the fitting process is applied.

  In one aspect of the image processing device, the second processing unit is obtained in the fitting process for the second input image when specifying the corresponding second graphic corresponding to the first graphic. Among the second graphics, the second graphics that are closest to the first graphics, and are second graphics whose first distance from the first graphics is equal to or less than a first threshold value. Is specified as the corresponding second graphic, and the second processing unit determines that there is an unspecified first graphic that is the first graphic in which the neighboring second graphic is not specified as the corresponding second graphic. When the second distance to the unspecified first graphic is less than or less than the second threshold value, the first graphic corresponding to the corresponding second graphic is applied. Not only the object but also the unspecified first figure Even for the object to be fitted, it is determined that the corresponding second figure is fitted.

  In one aspect of the image processing device, the second processing unit is obtained in the fitting process for the second input image when specifying the corresponding second graphic corresponding to the first graphic. Among the second graphics, the second graphics that are closest to the first graphics, and are second graphics whose first distance from the first graphics is equal to or less than a first threshold value. Is specified as the corresponding second graphic, and the second processing unit determines that there is an unspecified first graphic that is the first graphic in which the neighboring second graphic is not specified as the corresponding second graphic. Is not only the object to which the first graphic corresponding to the corresponding second graphic is applied, when there is the corresponding second graphic including at least a part of the region of the unspecified first graphic, The above-mentioned unidentified first figure is applied Even for elephants it was determined that the corresponding second figure is fitted.

  In one aspect of the image processing apparatus, the second processing unit may be configured to include an unspecified first graphic that is the first graphic in which the neighboring second graphic is not specified as the corresponding second graphic. When the second distance to the unspecified first graphic is less than or less than a second threshold value and there is the corresponding second graphic including at least a part of the region of the unspecified first graphic, The corresponding second graphic is applied not only to the object to which the first graphic corresponding to the corresponding second graphic is applied, but also to the object to which the unspecified first graphic is applied. judge.

  In one aspect of the image processing device, the input image is an image obtained by image processing of a captured image captured by a camera that captures the object, and the second processing unit includes: As the position of the first graphic moves away from the corresponding position corresponding to the position of the camera in the first input image, the first graphic used for identifying the corresponding second graphic corresponding to the first graphic is used. 1 Set a small threshold value.

  Further, in one aspect of the image processing device, the second processing unit causes the first processing unit to move away from a corresponding position corresponding to the position of the camera on the first input image. 2 Set the threshold value small.

  Further, in one aspect of the image processing device, the second processing unit is configured to determine the first input image based on a result of the fitting process for the first and second input images in the correspondence specifying process. It is determined whether or not a plurality of second graphics are applied in the fitting process for the second input image with respect to a plurality of objects to which one first graphic obtained in the fitting process is applied.

  In one aspect of the image processing apparatus, the second processing unit may perform the fitting process on the second input image that is closest in position to the first graphic from the size of the first graphic. The difference value obtained by subtracting the size of the second graphic obtained in step is larger than the third threshold value or greater than or equal to the third threshold value. In the case where at least a part of each of the plurality of second graphics obtained in the fitting process for the two-input image is included, for a plurality of objects corresponding to the one first graphics, It is determined that the plurality of second graphics are applied.

  In one aspect of the image processing apparatus, the input image is an image obtained by performing image processing on a captured image captured by a camera that captures an overhead view of the object, and the second processing unit includes: As the position of the one first figure moves away from the corresponding position corresponding to the position where the photographing depression angle of the person appearing in the photographed image becomes the maximum in the photographed image on the first input image, the third image is displayed. Increase the threshold value.

  Further, in one aspect of the image processing apparatus, the second processing unit, when the position of the one first graphic is within a first predetermined range from the corresponding position, The third threshold value is set to be larger as the position is farther from the corresponding position, and when the position of the first graphic is outside the first predetermined range, the position of the first graphic is The third threshold value is set smaller as the distance from the corresponding position increases.

  Further, in one aspect of the image processing device, the second processing unit corresponds to the third graphic that is the graphic applied to the object that is no longer included in the input image in the correspondence specifying process. The second graphic obtained in the fitting process for the two input images is identified, and the graphic information for the third graphic is updated based on the identification result, and the second processing unit adds the second graphic to the input image. The graphic information about the third graphic applied to the object that is no longer captured is erased after a predetermined time has elapsed since the object is not captured in the input image.

In one aspect of the image processing apparatus, the fourth processing unit has a third distance between two figures obtained by the third processing unit that is less than or less than a fourth threshold value, and the 2 If the size of one new figure obtained by integrating two figures is less than or less than the fifth threshold value, it is determined that the two figures have been applied to one object, and the new figure A simple figure is used instead of the two figures.

In one aspect of the image processing device, the input image is an image obtained by performing image processing on a captured image captured by a camera that captures an overhead view of the object, and the fourth processing unit includes: The two figures obtained by the third processing unit are one second extending from the corresponding position corresponding to the position where the shooting depression angle of the person appearing in the captured image is maximum in the captured image on the input image. The size of one new figure which is located on one straight line and the third distance between the two figures is less than or less than the fourth threshold and is obtained by integrating the two figures If the threshold is less than or less than the fifth threshold value, it is determined that the two figures are applied to the one object, and the new figure is adopted instead of the two figures.

Further, in one aspect of the image processing apparatus, the figure is an ellipse, and the fourth processing unit is configured to determine the two ellipses from the center of one of the two ellipses obtained by the third processing unit. The distance between the first intersection point where the second straight line connecting to the center of the other ellipse of the ellipse intersects the one ellipse and the second intersection point where the second straight line intersects the other ellipse 3 distances.

In one aspect of the image processing device, the input image is an image obtained by performing image processing on a captured image captured by a camera that captures an overhead view of the object, and the fourth processing unit includes: The position of the new figure obtained by integrating the two figures with the fourth threshold value compared with the third distance between the two figures has the largest shooting depression angle in the photographed image. The larger the distance from the corresponding position in the input image corresponding to the position, the larger the setting.

Further, according to one aspect of the image processing device, the fourth processing unit is configured to perform the processing when the position of a new figure obtained by integrating two figures is within the second predetermined range from the corresponding position. A new figure obtained by integrating the two figures by setting the fourth threshold value to be compared with the third distance between the two figures so that the position of the new figure is farther from the corresponding position. Is located outside the second predetermined range, the fourth threshold value to be compared with the third distance between the two figures is set to the position of the new figure from the corresponding position. Set smaller as you move away.

In one aspect of the image processing device, the input image is an image obtained by performing image processing on a captured image captured by a camera that captures an overhead view of the object, and the fourth processing unit includes: The fifth threshold value to be compared with the size of a new figure obtained by integrating two figures corresponds to the position where the new figure has the maximum shooting depression angle in the photographed image. The larger the distance from the corresponding position in the input image, the larger the setting.

Further, in one aspect of the image processing device, the fourth processing unit, when the position of a new figure obtained by integrating two figures is within the second predetermined range from the corresponding position, The fifth threshold value to be compared with the size of a new figure is set to increase as the position of the new figure moves away from the corresponding position, and the position of the new figure obtained by integrating two figures is the first value. 2 If it is outside the predetermined range, the fifth threshold value to be compared with the size of the new figure is set to be smaller as the position of the new figure is farther from the corresponding position.

Moreover, in one aspect of the image processing apparatus, the fourth processing unit has a difference between the position of one new figure obtained by integrating two figures and the corresponding position being less than or less than a sixth threshold value. The third distance between the two figures is less than or less than the fourth threshold value, regardless of whether the two figures are located on the first straight line, and When the size of the new figure is less than or less than the fifth threshold value, it is determined that the two figures have been applied to the one object, and the new figure is substituted for the two figures. To adopt.

  In one aspect of the image processing apparatus, a sixth processing unit that performs image processing on a captured image captured by a camera that captures the object and generates the input image is further provided.

  An aspect of the image processing system includes the above-described image processing apparatus and a camera that captures the captured image on which the sixth processing unit of the image processing apparatus performs the image processing.

The operation method of the image processing apparatus includes a first step of performing a fitting process for applying a figure to each of the plurality of input images in which the objects are captured at different times, with respect to the object in the input images. , based on a result of the fitting process, by tracking the position of the graphic in the input image, and a second step of performing tracking processing for tracking a position of the object in the input image, the In the first step, when a plurality of figures are applied to one object, one new figure obtained by integrating the plurality of figures is employed instead of the plurality of figures. .

Further, a control program for controlling the image processing apparatus is configured such that, for each of a plurality of input images in which objects at different times are captured, the image processing apparatus is configured to display a graphic for the object captured in the input image. And a tracking process for tracking the position of the object in the input image by tracking the position of the figure in the input image based on the result of the fitting process. And when a plurality of figures are applied to one object in the first step, one new figure obtained by integrating the plurality of figures and those of the order to execute the process employed in place of the plurality of figures.

  The processing load on the image processing apparatus can be reduced.

It is a figure which shows the structure of an image processing system. It is a figure which shows the example of installation of a camera. It is a figure which shows an example of a captured image. It is a figure which shows the structure of an image processing apparatus. It is a flowchart which shows operation | movement of an image processing apparatus. It is a figure which shows the structure of an image process part. It is a figure which shows an example of an input image. It is a figure which shows the structure of a fitting process part. It is a flowchart which shows a fitting process. It is a figure which shows a mode that the ellipse was applied with respect to the person reflected in an input image. It is a figure which shows a mode that the ellipse fitted with respect to the person reflected in the input image was superimposed on the captured image. It is a figure which shows a mode that the ellipse fitted with respect to the person reflected in the input image was superimposed on the captured image. It is a figure which shows an example of a picked-up image. It is a flowchart which shows an ellipse integration process. It is a figure for demonstrating an ellipse integration process. It is a figure for demonstrating an ellipse integration process. It is a figure for demonstrating an ellipse integration process. It is a figure for demonstrating an ellipse integration process. It is a figure for demonstrating an ellipse integration process. It is a figure for demonstrating an ellipse integration process. It is a figure for demonstrating an ellipse integration process. It is a figure for demonstrating an ellipse integration process. It is a figure for demonstrating an ellipse integration process. It is a figure for demonstrating an ellipse integration process. It is a figure for demonstrating an ellipse integration process. It is a figure for demonstrating an ellipse integration process. It is a figure for demonstrating an ellipse integration process. It is a figure for demonstrating an ellipse integration process. It is a figure for demonstrating an ellipse integration process. It is a figure for demonstrating an ellipse integration process. It is a figure which shows the structure of a tracking process. It is a figure which shows an example of tracking information. It is a flowchart which shows a tracking process. It is a figure which shows an example of a mode that a 1st ellipse isolate | separates into a some 2nd ellipse. It is a figure which shows an example of a mode that a 1st ellipse isolate | separates into a some 2nd ellipse. It is a figure for demonstrating a separation determination process. It is a figure for demonstrating a separation determination process. It is a figure for demonstrating a primary update process. It is a figure for demonstrating a primary update process. It is a figure for demonstrating a primary update process. It is a flowchart which shows a secondary specific process. It is a figure which shows an example of a mode that one 2nd ellipse is applied with respect to the some person to whom the some 1st ellipse was fitted. It is a figure which shows a mode that the 1st ellipse obtained by the fitting process in the front frame was superimposed on the captured image. It is a figure which shows a mode that the 2nd ellipse obtained by the fitting process in the present flame | frame was superimposed on the captured image. It is a figure which shows a mode that the 1st ellipse obtained by the fitting process in the front frame was superimposed on the captured image. It is a figure which shows a mode that the 2nd ellipse obtained by the fitting process in the present flame | frame was superimposed on the captured image. It is a figure for demonstrating operation | movement of a tracking process part. It is a figure for demonstrating operation | movement of a tracking process part. It is a figure for demonstrating operation | movement of a tracking process part. It is a figure for demonstrating operation | movement of a tracking process part.

<Configuration of image processing system>
FIG. 1 is a diagram showing a configuration of the image processing system 1. The image processing system 1 is a system that tracks the position of an object using image processing. The image processing system 1 is introduced into, for example, an office or a school. The image processing system 1 tracks the position of a person in a room such as an office. The object whose position is tracked may be other than a person.

  As shown in FIG. 1, the image processing system 1 includes a camera 2 and an image processing device 3. As shown in FIG. 2, the camera 2 is attached to, for example, a ceiling 310 in the room 300. The camera 2 is, for example, an omnidirectional camera and takes a bird's-eye view of the room 300. Therefore, the shooting range of the camera 2 is the room 300. The camera 2 captures and outputs an image at a predetermined frame rate (for example, 30 fps). The captured images 100 that are sequentially output from the camera 2 are sequentially input to the image processing apparatus 3. FIG. 2 also shows a person (object) 200 present in the room 300.

  In this example, the captured image 100 is a grayscale image, but may be a color image. The camera 2 may be attached to a place other than the ceiling 310. The camera 2 may be other than an omnidirectional camera.

  FIG. 3 is a diagram illustrating an example of the captured image 100. The captured image 100 is also called a fisheye image. As shown in FIG. 3, a plurality of desks 330 and a plurality of chairs 340 are arranged on the floor 320 of the room 300 that is captured in the captured image 100. There are a plurality of people 200 in the room 300. Specifically, in the room 300, there are two persons 200a and 200b standing beside the wall 350 and one person 200c sitting on a chair 340. Accordingly, the photographed image 100 includes three people 200a, 200b, and 200c.

  The image processing apparatus 3 tracks the position of the person 200 in the room 300 based on the input captured image 100. In other words, the image processing device 3 tracks the position of the person 200 in the shooting range of the camera 2 based on the input captured image 100. In other words, the image processing apparatus 3 tracks the position of the person 200 in the captured image 100 based on the input captured image 100.

  The image processing apparatus 3 is, for example, a kind of computer, and includes a CPU 4 (Central Processing Unit) and a storage unit 5 as shown in FIG. The storage unit 5 is configured by a non-transitory recording medium that can be read by the CPU 4, such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The storage unit 5 stores a control program 6 for controlling the image processing apparatus 3 (computer). Various function blocks are formed in the image processing apparatus 3 by the CPU 4 executing the control program 6 in the storage unit 5.

  Note that some or all of the functions of the image processing apparatus 3 may be realized by a hardware circuit including a logic circuit or the like that does not require a program (software) to realize the function. The storage unit 5 may include a computer-readable non-transitory recording medium other than the ROM and RAM. The storage unit 5 may include, for example, a small hard disk drive and an SSD (Solid State Drive).

<Configuration of image processing apparatus>
FIG. 4 is a diagram illustrating the configuration of the image processing apparatus 3. As shown in FIG. 4, the image processing apparatus 3 includes an image processing unit 10, a fitting processing unit 11, a tracking processing unit 12, and a counting unit 13. The image processing unit 10 performs image processing on the captured image 100 and outputs an image obtained thereby as an input image 140. The input image 140 shows a person 200 and includes an image showing the person 200. Hereinafter, an image showing the person 200 may be referred to as a “human area”.

  The fitting processing unit 11 performs a fitting process for fitting a figure to the person 200 shown in the input image 140. It can be said that the fitting process is a process of fitting a figure to a human region included in the input image 140. The fitting process can also be said to be a process of representing a human region included in the input image 140 with a graphic. In this example, the fitting processing unit 11 fits an ellipse, for example, to the person 200 shown in the input image 140 in the fitting process. Here, the ellipse in the present invention is a concept including a circle (perfect circle). Hereinafter, unless otherwise specified, “ellipse” means an ellipse that the fitting processing unit 11 has applied to the person 200 shown in the input image 140.

  The tracking processing unit 12 tracks the position of the person 200 on the input image 140 by tracking the position of the ellipse on the input image 140 based on the result of the fitting process in the fitting processing unit 11. I do. The tracking process can be said to be a process of tracking the position of the person 200 in the imaging region (room 300) of the camera 2 by tracking the position of the ellipse on the input image 140.

  The counting unit 13 performs a counting process for obtaining the number of people 200 shown in the input image 140 based on the result of the tracking process in the tracking processing unit 12. In other words, the count unit 13 obtains the number of people 200 that appear in the captured image 100 based on the result of the tracking process in the tracking processing unit 12. In other words, the count unit 13 obtains the number of people 200 existing in the shooting range (the room 300) of the camera 2 based on the result of the tracking process in the tracking processing unit 12.

  FIG. 5 is a flowchart showing the operation of the image processing apparatus 3. As shown in FIG. 5, when the image processing apparatus 3 acquires the captured image 100 from the camera 2 in step s1, the image processing unit 10 performs image processing on the captured image 100 in step s2, and inputs the captured image 100. An image 140 is generated. Specific processing in step s2 will be described later.

  After step s2, in step s3, the fitting processing unit 11 performs a fitting process for fitting an ellipse to the person 200 shown in the input image 140 generated in step s2. Thereafter, in step s4, the tracking processing unit 12 tracks the position of the person 200 on the input image 140 by tracking the position of the ellipse on the input image 140 based on the result of the fitting process in step s3. Follow the tracking process. Thereafter, in step s5, the counting unit 13 performs a counting process for obtaining the number of people 200 shown in the input image 140 generated in step s2, based on the result of the tracking process in step s4. That is, the count unit 13 obtains the number of people 200 shown in the captured image 100 acquired in step s1 based on the result of the tracking process in step s4.

  After that, when step s1 is executed again and a new captured image 100 (the captured image 100 of the next frame) is acquired by the image processing device 3, the image processing device 3 performs the same processing from steps s2 to s5. To run. Each time the captured image 100 is acquired (every time the captured image 100 is input), the image processing apparatus 3 executes the processes from steps s2 to s5 using the acquired captured image 100.

  Hereinafter, the input image 140 currently processed by the image processing apparatus 3 may be referred to as “current frame input image 140”. In the image processing apparatus 3, the input image 140 generated immediately before the input image 140 of the current frame may be referred to as “the input image 140 of the previous frame”.

<Details of image processing unit>
FIG. 6 is a diagram illustrating a configuration of the image processing unit 10. The image processing unit 10 performs a plurality of types of image processing on the captured image 100 to generate an input image 140. As illustrated in FIG. 6, the image processing unit 10 includes a resizing unit 20, a first noise removing unit 21, a background difference image generating unit 22, and a second noise removing unit 23.

  The resizing unit 20 reduces the size of the captured image 100 and outputs the reduced captured image 100 as a processed image 110. For example, the resizing unit 20 sets the number of vertical and horizontal pixels of the captured image 100 to 0.5 times. Hereinafter, the processed image 110 that is the reduced captured image 100 may be referred to as a “captured image 110”. The content that appears in the captured image 110 is the same as that that appears in the captured image 100. Therefore, it can be said that FIG. 3 described above is an example of the captured image 110.

  The first noise removing unit 21 performs predetermined image processing on the processed image 110 to remove noise included in the processed image 110. The first noise removing unit 21 smoothes the processed image 110 by performing a Gaussian filter on the processed image 110, for example. Thereby, the noise contained in the processed image 110 is removed. The first noise removing unit 21 outputs the processed image 110 after noise removal as the processed image 120.

  In addition, when there is no need to resize the captured image 100 because the number of pixels of the captured image 100 is small, the resizing unit 20 is not necessary. In this case, the first noise removing unit 21 performs a Gaussian filter on the captured image 100 and outputs the processed captured image 100 as the processed image 120.

  The background difference image generation unit 22 generates a background difference image 130 indicating a difference between the processed image 120 and a background image indicating only the background of the captured image 100. In this example, the background image is an image obtained by reducing an image obtained by the camera 2 taking an image of the room 300 in which no person 200 exists to the same size as the processed image 120. The background difference image generation unit 22 obtains a difference between the pixel value of the processed image 120 and the pixel value of the background image for each pixel, and binarizes the difference value of each pixel obtained thereby, thereby obtaining a background difference image. 130 is generated. Therefore, the background difference image 130 is a binarized image.

  The second noise removing unit 23 performs predetermined image processing on the background difference image 130 to remove noise included in the background difference image 130. For example, the second noise removing unit 23 repeatedly performs expansion processing and contraction processing on the background difference image 130 to remove noise included in the background difference image 130. The second noise removing unit 23 outputs the background difference image 130 after noise removal as the input image 140.

  FIG. 7 is a diagram illustrating an example of the input image 140 (background difference image 130 from which noise has been removed). In the example of FIG. 7, a portion having a pixel value “1” (high luminance region) is shown in black, and a portion having a pixel value “0” (low luminance region) is shown in white. The input image 140 illustrated in FIG. 7 corresponds to the captured image 100 (captured image 110) illustrated in FIG. 3 described above. That is, the input image 140 shown in FIG. 7 shows the difference between the processed image 120 and the background image generated based on the captured image 100 (captured image 110) shown in FIG.

  The input image 140 shown in FIG. 7 includes three people 200 and includes three person regions 230. Specifically, the input image 140 includes human areas 230a, 230b, and 230c that respectively indicate the persons 200a, 200b, and 200c shown in FIG. The human areas 230a, 230b, and 230c are high luminance areas. Since the input image 140 does not include a background image, it can be said that the image processing unit 10 detects the person in the room 300 by generating the input image 140.

  In the image processing unit 10 having the above configuration, when the image processing apparatus 3 acquires the captured image 100, the resize unit 20 resizes the captured image 100 in step s21 as shown in FIG. 5 described above. A processed image 110 is generated. In step s22, the first noise removing unit 21 removes noise included in the processed image 110 to generate the processed image 120. Thereafter, in step s23, the background difference image generation unit 22 generates a background difference image 130 indicating the difference between the processed image 120 and the background image. In step s24, the second noise removing unit 23 removes noise included in the background difference image 130 to generate the input image 140. When the input image 140 is generated in step s24, steps s3 to s5 are sequentially executed as described above.

  Hereinafter, the captured image 110 that is the basis of the input image 140 of the current frame may be referred to as “the captured image 110 of the current frame”. In addition, the captured image 110 that is the basis of the input image 140 of the previous frame may be referred to as “the captured image 110 of the previous frame”.

<Details of fitting processing section>
FIG. 8 is a diagram illustrating the configuration of the fitting processing unit 11. FIG. 9 is a flowchart showing details of the fitting process in step s3. As shown in FIG. 8, the fitting processing unit 11 includes a labeling unit 30, a contour extracting unit 31, an ellipse fitting unit 32, and an ellipse integrating unit 33. As shown in FIG. 9, in the fitting process, first, the process in the labeling unit 30 is executed in step s31, and then the process in the contour extracting unit 31 is executed in step s32. Next, in step s33, processing by the ellipse fitting unit 32 is executed, and finally processing by the ellipse integration unit 33 is executed. Specific processing in each configuration will be described below.

  The labeling unit 30 performs labeling using 4-connection or 8-connection on the high-luminance region included in the input image 140. And the labeling part 30 removes the independent area | region whose magnitude | size is smaller than a threshold value among the independent area | regions (connection area | region) obtained by labeling, or an independent area | region whose magnitude | size is below a threshold value. Thereby, a small independent area included in the input image 140 is removed. That is, noise included in the input image 140 is removed. The labeling unit 30 outputs the input image 140 from which noise has been removed as a processed image 150.

  The contour extraction unit 31 performs contour extraction of the high luminance region included in the processed image 150 and generates a contour extracted image 160 indicating the contour of the high luminance region included in the processed image 150. That is, the contour extraction unit 31 performs contour extraction of each human region included in the processed image 150 and generates a contour extracted image 160 indicating the contour of each human region included in the processed image 150. Hereinafter, the contour of the human region is referred to as “human contour”.

  The ellipse fitting unit 32 applies an ellipse to the human contour included in the contour extraction image 160. Thereby, an ellipse is applied to the person 200 shown in the input image 140. In other words, an ellipse is applied to the human region 230 included in the input image 140. The ellipse fitting unit 32 applies an ellipse to the human contour included in the contour extraction image 160 using, for example, the least square method.

  FIG. 10 is a diagram illustrating a state in which an ellipse 250 applied to the human contour included in the contour extraction image 160 generated from the input image 140 illustrated in FIG. 7 is superimposed on the input image 140. FIG. 11 is a diagram illustrating a state in which the ellipse 250 is superimposed on the captured image 110 obtained by resizing the captured image 100 illustrated in FIG. 3.

  As shown in FIG. 10, an ellipse 250 is applied to three people shown in the input image 140. Specifically, an ellipse 250a is applied to a person area 230a indicating the person 200a included in the input image 140. Further, an ellipse 250b is applied to a human area 230b indicating the person 200b included in the input image 140. Then, ellipses 250c1 and 250c2 are applied to a human area 230c indicating the person 200c included in the input image 140. Thus, as shown in FIG. 11, the ellipse 250a is applied to the person 200a shown in the photographed image 110, the ellipse 250b is applied to the person 200b shown in the photographed image 110, and the person 200c appears in the photographed image 110. Ellipses 250c1 and 250c2 are fitted to the above.

  When the ellipse fitting unit 32 applies a plurality of ellipses 250 to one person 200 shown in the input image 140 in the ellipse fitting unit 32, the ellipse integrating unit 33 integrates the plurality of ellipses to create one new ellipse. And the generated new ellipse is reapplied to the one person 200. That is, when the ellipse integration unit 33 applies a plurality of ellipses 250 to one person 200 shown in the input image 140 in the ellipse fitting unit 32, the ellipse integration unit 33 integrates the plurality of ellipses into one new one. A new ellipse is generated instead of the plurality of ellipses.

  Here, in the ellipse fitting unit 32, a plurality of ellipses 250 may be fitted to one person 200. For example, if the color of the jacket of the person 200 is the same as the background color, one ellipse 250 is fitted to the head of the person 200 and one ellipse 250 is fitted to the lower body of the person 200. May be. In the example of FIGS. 10 and 11 described above, two ellipses 250c1 and 250c2 are respectively applied to the upper and lower body of the person 200c sitting on the chair.

  On the other hand, as can be understood from the following description, the tracking processing unit 12 basically treats one ellipse 250 as one corresponding to one person 200. Therefore, when the plurality of ellipses 250 are applied to one person 200 in the fitting processing section 11, the tracking processing unit 12 treats the plurality of ellipses 250 as corresponding to the plurality of persons 200, respectively. It will be. As a result, the position of the person 200 in the room 300 may not be correctly tracked.

  Therefore, the ellipse integration unit 33, when the ellipse fitting unit 32 applies a plurality of ellipses 250 to one person 200, creates a new ellipse obtained by integrating the plurality of ellipses 250. Is adopted instead of the plurality of ellipses 250. As a result, even when a plurality of ellipses 250 are applied to one person 200 in the ellipse fitting unit 32, one ellipse is finally assigned to the one person 200. It can be applied.

  FIG. 12 is a diagram illustrating a state where a new ellipse 250c obtained by integrating a plurality of ellipses 250c1 and 250c2 applied to the person 200c shown in FIG. 11 is applied to the person 200c. As shown in FIG. 12, one ellipse 250c is applied to the person 200c by the action of the ellipse integration unit 33.

  Based on the result of fitting the ellipse 250 in the ellipse fitting unit 32 and the result of re-fitting a new ellipse in the ellipse integrating unit 33, the fitting processing unit 11 performs the final processing on the person 200 shown in the input image 140. The ellipse specifying information for specifying the ellipse that has been fitted is output to the tracking processing unit 12 as the fitting process result. The ellipse identification information includes the center coordinates, major axis length, minor axis length, and inclination of the ellipse. Therefore, the position and shape of the ellipse can be specified from the ellipse specifying information. The center coordinates of the ellipse included in the ellipse specifying information are the center coordinates of the ellipse on the input image 140. It can be said that the center coordinates of the ellipse included in the ellipse specifying information are the center coordinates of the ellipse on the captured image 110. The center coordinate of the ellipse included in the ellipse specifying information indicates the position of the person corresponding to the ellipse in the input image 140 (captured image 110).

<Details of ellipse integration processing>
Next, the ellipse integration process (the process of step s34) in the ellipse integration unit 33 will be described in detail. In this example, when the two ellipses satisfy all the first to third conditions, the ellipse integration unit 33 determines that the two ellipses are applied to one person, and integrates the two ellipses. One new ellipse obtained in this way is adopted instead of the two ellipses. Hereinafter, the new ellipse may be referred to as an “integrated ellipse”.

  The first condition is a condition that the distance between the two ellipses is not more than the first threshold value. When two ellipses are fitted to one person, the distance between the two ellipses is small. Therefore, when two ellipses satisfy the first condition, there is a possibility that the two ellipses are applied to one person. The first condition may be a condition that the distance between the two ellipses is less than the first threshold value. A specific example of the determination method for the first condition will be described later in detail.

  The second condition is a condition that the positions of two ellipses are on one straight line extending from a specific position on the input image 140. This specific position is a corresponding position on the input image 140 corresponding to a position where the bird's-eye view angle shown in the captured image 110 (captured image 100) is maximum in the captured image 110 (captured image 100).

  Here, the bird's-eye view angle shown in the photographed image indicates what kind of bird's-eye view angle (the depression angle) the person looks like. In other words, the bird's-eye view angle in the captured image means the certain bird's-eye view angle when the person is seen as if seen from a certain bird's-eye view angle. Therefore, the bird's-eye view angle shown in the photographed image means the angle of the person shown in the photographed image (how the person looks in the photographed image), and means the angle (posture) of the camera. is not. In the captured image, when a person is seen as if seen from directly above, the overhead angle of the person appearing in the photographed image is 90 degrees at the maximum.

  In the captured image obtained by the overhead view, the overhead angle of the person varies depending on the image position. In this example, since the camera 2 is an omnidirectional camera attached to the ceiling 310 of the room 300, the center of the captured image 110 is a position where the person's overhead angle in the captured image 110 is maximized. That is, the person's bird's-eye view angle at the center of the captured image 110 is 90 degrees. And in the picked-up image 110, a person's bird's-eye angle becomes small, so that it leaves | separates from a center. Therefore, as shown in FIG. 13, in the captured image 110, the person 200 (200 a to 200 c) located away from the center 111 appears to extend along the direction 112 from the center 111 toward the outside.

  In this example, since the center of the captured image 110 is the maximum position where the person's overhead angle is maximum, the corresponding position corresponding to the maximum position in the input image 140 is the center of the input image 140 (2 Intersection of two diagonals). Therefore, in this example, the second condition is a condition that two ellipses are located on one straight line extending from the center of the input image 140. From another point of view, the second condition can be said to be a condition that two ellipses are located on one straight line extending from the center on the contour extraction image 160 generated from the input image 140. Further, from another viewpoint, the second condition can be said to be a condition that two ellipses are located on one straight line extending from the center of the captured image 110. Hereinafter, in the captured image 110, the maximum position where the person's overhead angle is maximum may be referred to as “maximum overhead angle position”.

  In the photographed image 110, the person 200 appears to extend along the direction 112 extending outward from the center 111 of the photographed image 110, so that two ellipses obtained from the input image 140 are displayed on the input image 140. When arranged along the direction from the center to the outside, the two ellipses may be applied to one person (in the example of FIG. 10, applied to the person 200c). Two ellipses 250c1 and 250c2 are arranged on the input image 140 along the direction from the center to the outside). That is, when two ellipses obtained from the input image 140 are positioned on one straight line extending from the center of the input image 140, the two ellipses are represented by one person appearing in the input image 140. It may be applied to a person. Therefore, when two ellipses satisfy the second condition, there is a possibility that the two ellipses are applied to one person. A specific example of the determination method of the second condition will be described later in detail.

  The third condition is a condition that the size of the integrated ellipse obtained by integrating the two ellipses is not more than the second threshold value. As the size of the integrated ellipse, for example, the area of the integrated ellipse is adopted. When two ellipses are applied to one person, the size of the integrated ellipse obtained by integrating the two ellipses is close to the size of the person shown in the input image 140. Therefore, when the size of the integrated ellipse obtained by integrating the two ellipses is too large, there is a high possibility that the two ellipses are applied to different people. Therefore, when two ellipses satisfy the third condition, the two ellipses may be applied to one person. The third condition may be a condition that the size of the integrated ellipse obtained by integrating the two ellipses is less than the second threshold value. A specific example of the third condition determination method will be described later in detail.

  In this example, since the camera 2 is an omnidirectional camera attached to the ceiling 310 of the room 300, it can be said that the center of the captured image 110 is a position corresponding to the position of the camera 2 in the captured image 110. Similarly, it can be said that the center of the input image 140 is a position corresponding to the position of the camera 2 in the input image 140.

<Flow of ellipse integration processing>
FIG. 14 is a flowchart showing the ellipse integration process (the process of step s34). As shown in FIG. 14, in the ellipse integration process, first in step s341, the ellipse integration unit 33 sets a plurality of ellipses obtained by the ellipse fitting unit 32 as a first ellipse group, and an ellipse is included in the first ellipse group. It is determined whether there are two or more. If the ellipse integration unit 33 determines that there are two or more ellipses in the first ellipse group, in step s342, the ellipse integration unit 33 determines a first target ellipse to be processed from the first ellipse group. In step s343, the ellipse integration unit 33 deletes the first target ellipse determined in step s342 from the first ellipse group. On the other hand, if the ellipse integration unit 33 determines No in step s341, the ellipse integration process ends.

  After step s343, in step s344, the ellipse integration unit 33 sets the current first ellipse group as the second ellipse group and determines the second target ellipse from the second ellipse group. The second ellipse group does not include the first target ellipse determined in step s342. Therefore, the second target ellipse in step s344 is an ellipse different from the first target ellipse in step s342. After step s344, the ellipse integration unit 33 deletes the second target ellipse determined in step s344 from the second ellipse group in step s345.

  Next, in step s346, the ellipse integration unit 33 determines whether the first and second target ellipses satisfy the first condition. That is, the ellipse integration unit 33 determines whether the distance between the first and second target ellipses is equal to or less than the first threshold value.

  If the ellipse integration unit 33 determines that the first and second target ellipses satisfy the first condition, in step s347, the ellipse integration unit 33 determines whether the first and second target ellipses satisfy the second condition. That is, the ellipse integration unit 33 determines whether or not the first and second target ellipses are located on one straight line extending from the center of the input image 140. On the other hand, if the ellipse integration unit 33 determines that the first and second target ellipses do not satisfy the first condition, that is, the distance between the first and second target ellipses is greater than the first threshold value. If determined, step s351 described later is executed.

  If the ellipse integration unit 33 determines that the first and second target ellipses satisfy the second condition, in step s348, the ellipse integration unit 33 determines whether the first and second target ellipses satisfy the third condition. That is, the ellipse integration unit 33 determines whether the size of the integrated ellipse obtained by integrating the first and second target ellipses is equal to or smaller than the second threshold value. On the other hand, if the ellipse integration unit 33 determines that the first and second target ellipses do not satisfy the second condition, that is, the first and second target ellipses 1 extend from the center on the input image 140. If it is determined that the position is not on the straight line of the book, step s351 is executed.

  If the ellipse integration unit 33 determines that the first and second target ellipses satisfy the third condition, that is, determines that the first and second target ellipses satisfy all of the first to third conditions. In step s349, it is determined that the first and second target ellipses are applied to one person, and the integrated ellipse obtained by integrating the first and second target ellipses is defined as the first and second targets. Adopt instead of ellipse. Thereafter, in step s350, the ellipse integration unit 33 updates the first ellipse group. Specifically, the ellipse integration unit 33 deletes the second target ellipse from the first ellipse group and adds the integrated ellipse employed in step s349 to the first ellipse group.

  After step s350, the ellipse integration unit 33 executes step s341 again to determine whether two or more ellipses exist in the updated first ellipse group. If the ellipse integration unit 33 determines No, it ends the ellipse integration process. On the other hand, if the ellipse integration unit 33 determines Yes, in step s342, a new first target ellipse is determined from the first ellipse group. Then, in step s343, the ellipse integration unit 33 deletes the first target ellipse newly determined from the first ellipse group. Next, in step s344, the ellipse integration unit 33 sets the current first ellipse group as the second ellipse group and determines a new second target ellipse from the second ellipse group. The ellipse integration unit 33 operates similarly thereafter.

  In step s351 executed when it is determined No in steps s346, s347, and s348, the ellipse integration unit 33 determines whether an ellipse exists in the second ellipse group. If the ellipse integration unit 33 determines that an ellipse is present in the second ellipse group, it executes step s344 to determine a new second target ellipse from the second ellipse group. On the other hand, if the ellipse integration unit 33 determines that no ellipse exists in the second ellipse group, step s341 is executed. Thereafter, the ellipse integration unit 33 operates in the same manner.

  As the ellipse integration unit 33 operates as described above, a plurality of ellipses applied by the ellipse fitting unit 32 to one person shown in the input image 140 are integrated into one new ellipse. Then, the new ellipse is adopted instead of the plurality of ellipses, and reapplied to the one person.

<Example of ellipse integration processing>
Next, an example of the ellipse integration process shown in FIG. 14 will be described by taking as an example the case where the ellipse fitting unit 32 has obtained a plurality of ellipses 250 shown in FIG. In FIG. 15, a plurality of ellipses 250 e to 250 i obtained in the ellipse fitting unit 32 are arranged on the input image 140. In FIG. 15, the human area included in the input image 140 is not shown for convenience of explanation. Thereafter, in the figure, when the human area included in the input image 140 is unnecessary for the description, the human area is deleted to show the input image 140.

  First, the ellipse integration unit 33 sets the plurality of ellipses 250e to 250i obtained by the ellipse fitting unit 32 as a first ellipse group, and determines whether two or more ellipses exist in the first ellipse group (step s341). ). Here, since the first ellipse group includes five ellipses 250e to 250i, it is determined as Yes.

  Next, the ellipse integration unit 33 determines a first target ellipse from the first ellipse group (step s342). Here, for example, the ellipse 250e is the first target ellipse 250e. Then, the ellipse integration unit 33 deletes the first target ellipse 250e from the first ellipse group (step s343).

  Next, the ellipse integration unit 33 sets the current first ellipse group (ellipses 250f to 250i) as the second ellipse group, and determines the second target ellipse from the second ellipse group (step s344). Here, for example, the ellipse 250f is set as the second target ellipse 250f. Then, the ellipse integration unit 33 deletes the second target ellipse 250f from the second ellipse group (step s345).

  Next, the ellipse integration unit 33 determines whether or not the first target ellipse 250e and the second target ellipse 250f satisfy the first condition (step s346). As FIG. 16 shows, since the distance between the 1st object ellipse 250e and the 2nd object ellipse 250f is small, the 1st condition is satisfy | filled (Yes determination).

  Next, the ellipse integration unit 33 determines whether or not the first target ellipse 250e and the second target ellipse 250f satisfy the second condition (step s347). As shown in FIG. 16, the first target ellipse 250e and the second target ellipse 250f are located on the two straight lines 142 extending from the center 141 on the input image 140, that is, from the center 141. Since they are not lined up along the direction toward the outside, the second condition is not satisfied (No determination).

  Next, the ellipse integration unit 33 determines whether an ellipse exists in the second ellipse group (step s351). Here, since the ellipses 250g to 250i exist in the second ellipse group, the ellipse integration unit 33 determines Yes.

  Next, the ellipse integration unit 33 determines a new second target ellipse from the second ellipse group including the ellipses 250g to 250i (step s344). Here, for example, the ellipse 250g is set as the second target ellipse 250g. Then, the ellipse integration unit 33 deletes the second target ellipse 250g from the second ellipse group (step s345).

  Next, the ellipse integration unit 33 determines whether or not the first target ellipse 250e and the second target ellipse 250g satisfy the first condition (step s346). As FIG. 17 shows, since the distance between the 1st object ellipse 250e and the 2nd object ellipse 250g is large, the 1st condition is not satisfy | filled (No determination).

  Next, the ellipse integration unit 33 determines whether an ellipse exists in the second ellipse group (step s351). Here, since the ellipses 250h and 250i exist in the second ellipse group, the ellipse integration unit 33 determines Yes.

  Thereafter, when the ellipse integration unit 33 operates in the same manner and determines that there are no ellipses in the second ellipse group in step s351, the ellipse integration unit 33 determines whether two or more ellipses exist in the first ellipse group ( Step s341). Here, since the four ellipses 250f to 250i exist in the first ellipse group, it is determined as Yes.

  Next, the ellipse integration unit 33 determines a first target ellipse from the first ellipse group including the ellipses 250f to 250i (step s342). Here, for example, the ellipse 250f is the first target ellipse 250f. Then, the ellipse integration unit 33 deletes the first target ellipse 250f from the first ellipse group (step s343).

  Next, the ellipse integration unit 33 sets the current first ellipse group (ellipses 250g to 250i) as the second ellipse group, and determines the second target ellipse from the second ellipse group (step s344). Here, for example, the ellipse 250g is set as the second target ellipse 250g. Then, the ellipse integration unit 33 deletes the second target ellipse 250g from the second ellipse group (step s345).

  Next, the ellipse integration unit 33 determines whether or not the first target ellipse 250f and the second target ellipse 250g satisfy the first condition (step s346). As shown in FIG. 18, since the distance between the first target ellipse 250f and the second target ellipse 250g is large, the first condition is not satisfied (No determination).

  Next, the ellipse integration unit 33 determines whether an ellipse exists in the second ellipse group (step s351). Here, since the ellipses 250h and 250i exist in the second ellipse group, it is determined as Yes.

  Thereafter, when the ellipse integration unit 33 operates in the same manner and determines in step s351 that there are no ellipses in the second ellipse group, it determines whether there are two or more ellipses in the first ellipse group (step s341). ). Here, it is determined Yes because three ellipses 250g to 250i exist in the first ellipse group.

  Next, the ellipse integration unit 33 determines a first target ellipse from the first ellipse group including the ellipses 250g to 250i (step s342). Here, for example, the ellipse 250g is the first target ellipse 250g. Then, the ellipse integration unit 33 deletes the first target ellipse 250g from the first ellipse group (step s343).

  Next, the ellipse integration unit 33 sets the current first ellipse group (ellipses 250h and 250i) as the second ellipse group, and determines the second target ellipse from the second ellipse group (step s344). Here, for example, the ellipse 250h is set as the second target ellipse 250h. Then, the ellipse integration unit 33 deletes the second target ellipse 250h from the second ellipse group (step s345).

  Next, the ellipse integration unit 33 determines whether or not the first target ellipse 250g and the second target ellipse 250h satisfy the first condition (step s346). As FIG. 19 shows, since the distance between the 1st object ellipse 250g and the 2nd object ellipse 250h is small, the 1st condition is satisfy | filled (Yes determination).

  Next, the ellipse integration unit 33 determines whether or not the first target ellipse 250g and the second target ellipse 250h satisfy the second condition (step s347). As shown in FIG. 19, the first target ellipse 250 g and the second target ellipse 250 h are located on one straight line 142 extending from the center 141 on the input image 140, that is, outside the center 141. Therefore, the second condition is satisfied (Yes determination).

  Next, the ellipse integration unit 33 determines whether or not the first target ellipse 250g and the second target ellipse 250h satisfy the third condition (step s348). As shown in FIG. 20, since the size of the integrated ellipse 250j obtained by integrating the first target ellipse 250g and the second target ellipse 250h is small, the third condition is satisfied (Yes determination).

  When determining that the third condition is satisfied, the ellipse integration unit 33 determines that the first target ellipse 250g and the second target ellipse 250h are applied to one person (step s349). Then, the ellipse integration unit 33 employs the integrated ellipse 250j instead of the first target ellipse 250g and the second target ellipse 250h (step s349).

  Next, the ellipse integration unit 33 deletes the second target ellipse 250h from the first ellipse group (step s350). Further, the ellipse integration unit 33 adds the integrated ellipse 250j to the first ellipse group (step s350).

  Next, the ellipse integration unit 33 determines whether there are two or more ellipses in the first ellipse group (step s341). Here, since the first ellipse group includes two ellipses 250i and 250j, it is determined as Yes.

  Next, the ellipse integration unit 33 determines a new first target ellipse from the first ellipse group including the ellipses 250i and 250j (step s342). Here, for example, the ellipse 250j (integrated ellipse 250j) is the first target ellipse 250j. Then, the ellipse integration unit 33 deletes the first target ellipse 250j from the first ellipse group (step s343).

  Next, the ellipse integration unit 33 sets the current first ellipse group (ellipse 250i) as the second ellipse group, and determines the second target ellipse from the second ellipse group (step s344). Here, the ellipse 250i is the second target ellipse 250i. Then, the ellipse integration unit 33 deletes the second target ellipse 250i from the second ellipse group (step s345).

  Next, the ellipse integration unit 33 determines whether or not the first target ellipse 250j and the second target ellipse 250i satisfy the first condition (step s346). As FIG. 21 shows, since the distance between the 1st object ellipse 250j and the 2nd object ellipse 250i is small, the 1st condition is satisfy | filled (Yes determination).

  Next, the ellipse integration unit 33 determines whether or not the first target ellipse 250j and the second target ellipse 250i satisfy the second condition (step s347). As shown in FIG. 21, the first target ellipse 250j and the second target ellipse 250i are located on a single straight line 142 extending from the center 141 on the input image 140, that is, outside the center 141. Therefore, the second condition is satisfied (Yes determination).

  Next, the ellipse integration unit 33 determines whether or not the first target ellipse 250j and the second target ellipse 250i satisfy the third condition (step s348). As FIG. 22 shows, since the magnitude | size of the integrated ellipse 250k obtained by integrating the 1st object ellipse 250j and the 2nd object ellipse 250i is small, the 3rd condition is satisfy | filled (Yes determination).

  Next, the ellipse integration unit 33 determines that the first target ellipse 250j and the second target ellipse 250i have been applied to one person (step s349). Then, the ellipse integration unit 33 employs the integrated ellipse 250k instead of the first target ellipse 250j and the second target ellipse 250i (step s349).

  Next, the ellipse integration unit 33 deletes the second target ellipse 250i from the first ellipse group (step s350). The ellipse integration unit 33 adds the integrated ellipse 250k to the first ellipse group (step s350). Then, the ellipse integration unit 33 determines whether there are two or more ellipses in the first ellipse group (step s341). Here, it is determined No because only one ellipse 250k exists in the first ellipse group. If the ellipse integration unit 33 determines No, it ends the ellipse integration process.

  15-22, three ellipses 250g to 250i fitted to one person are integrated, and a new ellipse 250k obtained thereby is adopted instead of the three ellipses 250g to 250i. . Therefore, after the ellipse integration process, as shown in FIG. 22, an ellipse 250e, an ellipse 250f, and an ellipse 250k remain. The fitting processing unit 11 outputs the fitting processing result including the ellipse specifying information regarding the ellipses 250e, 250f, and 250k to the tracking processing unit 12.

<Details of first condition determination method>
When the ellipse integration unit 33 determines whether or not two ellipses satisfy the first condition, the ellipse integration unit 33 determines from the center of one of the two ellipses the other ellipse of the two ellipses. The distance between the first intersection where the second straight line connecting to the center of the first intersection intersects with the one ellipse and the second intersection where the second straight line intersects with the other ellipse is defined as the distance between the two ellipses. Ask.

  For example, consider a case where the distance between two ellipses 250m and 250n shown in FIG. The ellipse integration unit 33 obtains a straight line (line segment) 260 connecting the center 251m of the ellipse 250m to the center 251n of the ellipse 250n. The ellipse integration unit 33 sets the distance dx between the intersection 252m between the straight line 260 and the ellipse 250m and the intersection 252n between the straight line 260 and the ellipse 250n as the distance between the ellipses 250m and 250n. In step s346, the ellipse integration unit 33 determines whether or not the ellipses 250m and 250n satisfy the first condition by determining whether or not the distance dx is equal to or less than the first threshold value.

  Note that the distance between the centers of the two ellipses may be adopted as the distance between the two ellipses. In this case, the first condition may not be satisfied even though the two ellipses are close to each other. For example, as shown in FIG. 23 described above, when the distance dn between the center 251n and the intersection 252n is large in at least one of the two ellipses 250m and 250n, the distance between the centers 251m and 251n. dmn increases. Therefore, the first condition may not be satisfied even though the ellipses 250m and 250n are close to each other. On the other hand, when the above-described distance dx is adopted as the distance between the ellipses 250m and 250n, it can be appropriately determined that the ellipses 250m and 250n are close to each other without such a situation. Therefore, it can be more correctly determined whether or not the ellipses 250m and 250n are applied to one person.

<Details of second condition determination method>
FIG. 24 is a diagram for explaining an example of a method by which the ellipse integration unit 33 determines whether or not two ellipses are located on one straight line extending from the center of the input image 140. is there. In FIG. 24, two ellipses 250p and 250q are arranged on the input image 140.

  When the ellipse integration unit 33 determines whether or not the two ellipses 250p and 250q are located on one straight line extending from the center 141 on the input image 140, on the input image 140, A first vector extending from the center 141 to the center of one of the two ellipses 250p and 250q is obtained. In addition, the ellipse integration unit 33 obtains a second vector extending from the center of the one ellipse 250 to the center of the other ellipse 250 of the two ellipses 250p and 250q on the input image 140. In the example of FIG. 24, a vector extending from the center 141 to the center 251q of the ellipse 250q is a first vector 271. A vector extending from the center 251q of the ellipse 250q to the center 251p of the ellipse 250p is defined as a second vector 272.

  Next, the ellipse integration unit 33 obtains an angle α formed by the first vector 271 and the second vector 272. FIG. 25 is a diagram showing the angle α. Next, the ellipse integration unit 33 determines whether or not the angle α is 90 degrees or less. If α ≦ 90 degrees, the ellipse integration unit 33 sets the angle α as the determination angle β. On the other hand, if α> 90 degrees, the ellipse integration unit 33 sets (180 ° −α) as the determination angle β. Since the angle α shown in FIG. 25 is 90 degrees or less, the angle α becomes the determination angle β as it is.

  Next, the ellipse integration unit 33 determines whether the determination angle β is equal to or smaller than the third threshold value. For example, the third threshold value is set to several degrees or less. If the determination angle β is equal to or smaller than the third threshold value, the ellipse integration unit 33 determines that the two ellipses 250p and 250q are located on one straight line extending from the center 141 on the input image 140. To do. That is, the ellipse integration unit 33 determines that the two ellipses 250p and 250q are arranged along the direction from the center 141 toward the outside on the input image 140. On the other hand, if the determination angle β is larger than the third threshold value, the ellipse integration unit 33 positions the two ellipses 250p and 250q on one straight line extending from the center 141 on the input image 140. Judge that there is no. That is, the ellipse integration unit 33 determines that the two ellipses 250p and 250q are not aligned along the direction from the center 141 toward the outside on the input image 140. 24 and 25, the determination angle β is equal to or smaller than the third threshold value, and two ellipses 250p and 250q extend from the center 141 on the input image 140 as shown in FIG. 142 is determined to be located on the top. That is, it is determined that the second condition of the two ellipses 250p and 250q is satisfied.

  In the ellipse integration unit 33, if the determination angle β is less than the third threshold value, the two ellipses 250p and 250q are positioned on one straight line extending from the center 141 on the input image 140. If the determination angle β is equal to or greater than the third threshold value, it is determined that the two ellipses 250p and 250q are not located on one straight line extending from the center 141 on the input image 140. Also good.

  27, a vector extending from the center 141 to the center 251p of the ellipse 250p is a first vector 271 and a vector extending from the center 251p of the ellipse 250p to the center 251q of the ellipse 250q is a second vector 272. good. In this case, as shown in FIG. 28, the angle α formed by the first vector 271 and the second vector 272 is greater than 90 degrees. Therefore, the determination angle β is (180 degrees−α).

  In the above example, the determination angle β is determined from the angle α formed by the first vector 271 and the second vector 272. However, as shown in FIG. 29, two ellipses are formed from the center 141 of the input image 140. An angle of 180 ° or less formed by two vectors 273 extending to the centers 251p and 251q of 250p and 250q, respectively, may be adopted as the determination angle β.

<Details of third condition determination method>
FIG. 30 is a diagram for explaining an example of a method by which the ellipse integration unit 33 integrates two ellipses. FIG. 30 shows two ellipses 250r and 250s (broken lines) and an integrated ellipse 250t (solid line) obtained by integrating them.

  When integrating the two ellipses 250r and 250s, the ellipse integration unit 33 obtains a straight line 280 passing through the centers 251r and 251s of the ellipses 250r and 250s. Then, the ellipse integration unit 33 obtains an intersection 253s (an intersection far from the ellipse 250s) on the side opposite to the ellipse 250s among the two intersections where the ellipse 250r and the straight line 280 intersect. In addition, the ellipse integration unit 33 obtains an intersection 253s (an intersection far from the ellipse 250r) on the opposite side to the ellipse 250r among the two intersections where the ellipse 250s and the straight line 280 intersect. Then, the ellipse integration unit 33 obtains an ellipse 250t having a long axis 255t as a line segment connecting the intersection 253r and the intersection 253s. At this time, the ellipse integration unit 33 sets a predetermined ratio of the length of the major axis 255t as the length of the minor axis 256t of the ellipse 250t. This predetermined ratio is set to the same value as, for example, the ratio of the height of the person to the width of a general person. The ellipse integration unit 33 sets the obtained ellipse 250t as an integrated ellipse of the two ellipses 250s and 250r. Note that the method for obtaining the integrated ellipse may be other than this.

  The ellipse integration unit 33 determines whether or not the two ellipses 250r and 250s satisfy the third condition by determining whether or not the size of the obtained ellipse 250t is greater than or equal to the second threshold value. . The ellipse integration unit 33 adopts the area of the ellipse 250t as the size of the ellipse 250t. The ellipse integration unit 33 may adopt the major axis length of the ellipse 250t as the size of the ellipse 250t.

<Details of the tracking processor>
Next, the tracking processing unit 12 will be described in detail. FIG. 31 is a diagram illustrating a configuration of the tracking processing unit 12. As shown in FIG. 31, the tracking processing unit 12 includes a correspondence specifying unit 40 and an updating unit 44. The correspondence specifying unit 40 includes a primary specifying unit 41, a secondary specifying unit 42, and a tertiary specifying unit 43. In the tracking process, the tracking processing unit 12 tracks the position of the ellipse on the input image 140 by updating the tracking information 50 in the storage unit 5 based on the fitting process result 170 obtained by the fitting processing unit 11. . The tracking information 50 is updated for each frame. That is, the tracking information 50 is updated every time the captured image 100 is input from the camera 2 to the image processing apparatus 3, in other words, every time the input image 140 is generated by the image processing unit 10.

<About tracking information>
The tracking information 50 includes graphic information 500 regarding an ellipse applied to a person shown in the input image 140 (a person shown in the captured image 110). The tracking processing unit 12 tracks the position of the ellipse on the input image 140 by updating the graphic information 500 for the ellipse for each frame. Basically, one graphic information 500 exists for one ellipse.

  FIG. 32 is a diagram showing an example of the graphic information 500. As shown in FIG. 32, the graphic information 500 includes an ID 501, state information 502, and ellipse specifying information 503. Further, the graphic information 500 may include a disappearance time 504. Further, the graphic information 500 may include a merge partner ID 505 and a merge number 506.

  The ID 501 included in the graphic information 500 is identification information for identifying an ellipse corresponding to the graphic information 500. It can be said that the ID 501 is identification information for identifying the graphic information 500. It can also be said that the ID 501 is identification information for identifying a person corresponding to the ellipse corresponding to the graphic information 500.

  The ellipse specifying information 503 included in the graphic information 500 is information for specifying an ellipse corresponding to the graphic information 500. The ellipse identification information 503 includes the center coordinates, major axis length, minor axis length, and inclination of the ellipse. Therefore, the ellipse specifying information 503 specifies the position and shape of the ellipse. It can be said that the ellipse specifying information 503 included in the graphic information 500 indicates the latest position on the input image 140 for the ellipse corresponding to the graphic information 500. In other words, it can be said that the ellipse specifying information 503 included in the graphic information 500 indicates the latest position on the photographed image 110 for the person corresponding to the ellipse corresponding to the graphic information 500. In other words, it can be said that the ellipse specifying information 503 included in the graphic information 500 indicates the latest position (current position) in the room 300 of the person corresponding to the ellipse corresponding to the graphic information 500. The tracking processing unit 12 tracks the position of the ellipse corresponding to the ellipse specifying information 503 by updating the ellipse specifying information 503.

  The state information 502 included in the graphic information 500 is information indicating the state of the ellipse corresponding to the graphic information 500 detected by the tracking processing unit 12. The tracking processing unit 12 detects the state of the ellipse on the input image 140 based on the fitting processing result 170, and updates the state information 502 included in the graphic information 500 about the ellipse based on the detection result. There are five types of status information 502: “appearance”, “existence”, “disappearance”, “merge”, and “merge”.

  Here, in this example, for convenience, it is considered that an ellipse appears on the input image 140 as a result of fitting the ellipse to the person shown in the input image 140. When one new ellipse is applied to a plurality of people, one previous ellipse among the plurality of previous ellipses applied to the plurality of people becomes the one new ellipse, Consider that the one new ellipse merges the remaining previous ellipses of the plurality of previous ellipses.

  When the tracking processing unit 12 determines that an ellipse appears on the input screen 140 for the first time, the tracking processing unit 12 sets the state information 502 included in the graphic information 500 of the certain ellipse to “appear”. Therefore, the state information 502 indicating “appearance” included in the graphic information 500 indicates that an ellipse corresponding to the graphic information 500 has appeared on the input screen 140 for the first time. The first appearance of an ellipse on the input image 140 means that a person corresponding to the ellipse has first appeared on the input screen 140 (captured image 110). Therefore, the state information 502 indicating “appearance” included in the graphic information 500 indicates that a person corresponding to the graphic information 500 first appears on the input screen 140 (the photographed image 110).

  When it is determined that the ellipse that has appeared on the input screen 140 no longer appears on the input screen 140, the tracking processing unit 12 sets the state information 502 included in the graphic information 500 of the ellipse to “disappear”. . Therefore, the state information 502 indicating “disappearance” included in the graphic information 500 indicates that an ellipse corresponding to the graphic information 500 does not appear on the input screen 140. The fact that the ellipse that appeared on the input image 140 does not appear on the input image 140 means that the person corresponding to the ellipse is no longer shown on the input screen 140 (the photographed image 110). Therefore, the state information 502 indicating “disappearance” included in the graphic information 500 indicates that the person corresponding to the graphic information 500 is no longer displayed on the input screen 140 (captured image 110).

  When it is determined that a certain ellipse continues to appear on the input screen 140, the tracking processing unit 12 sets the state information 502 included in the graphic information 500 of the certain ellipse to “exist”. Therefore, the state information 502 indicating “existence” included in the graphic information 500 indicates that an ellipse corresponding to the graphic information 500 continuously appears on the input screen 140. In this example, when a certain ellipse appears on the input image 140 continuously for two or more frames, the state information 502 included in the graphic information 500 for the certain ellipse is set to “exist”. However, when a certain ellipse does not appear on the input screen 140 again and appears again in the input image 140 within a predetermined time, the tracking processing unit 12 includes a state included in the graphic information 500 regarding the certain ellipse. The information 502 is changed from “disappearance” to “existence” instead of “appearance”. The continuous appearance of an ellipse on the input image 140 means that a person corresponding to the ellipse is continuously captured on the input screen 140 (the captured image 110). Therefore, the state information 502 indicating “existence” included in the graphic information 500 indicates that the person corresponding to the graphic information 500 is continuously reflected in the input screen 140 (the photographed image 110).

  When it is determined that a certain ellipse merges another ellipse, the tracking processing unit 12 sets the state information 502 included in the graphic information 500 of the certain ellipse to “merged”. Therefore, the state information 502 indicating “merged” included in the graphic information 500 indicates that the ellipse corresponding to the graphic information 500 is merged with another ellipse.

  When it is determined that a certain ellipse has been merged with another ellipse, the tracking processing unit 12 sets the state information 502 included in the graphic information 500 of the certain ellipse to “merged”. Therefore, the state information 502 indicating “merge” included in the graphic information 500 indicates that the ellipse corresponding to the graphic information 500 is merged with another ellipse.

  In this example, “merge” and “merge” simply mean that one ellipse is applied to a plurality of people, and the state information 502 is an ellipse indicating “merge”. And the state information 502 is not used to distinguish the state on the input image 140 between the ellipse indicating “merged”. An ellipse whose state is “merged” and an ellipse whose state is “merged” are the same, but in this example, they are treated as separate ellipses for convenience. It will become clear from the following description how the status information 502 of the graphic information 500 is set to “merged” or “merged”.

  Here, in the case where a plurality of people appear close to each other in the input image 140, there is a high possibility that a plurality of person areas indicating the plurality of persons included in the input image 140 are connected. Therefore, in the fitting processing unit 11, there is a high possibility that one ellipse is fitted to a plurality of people appearing close to each other in the input image 140. Therefore, the state information 502 indicating “merge” or “merge” included in the graphic information 500 indicates that the person corresponding to the graphic information 500 is reflected in the input image 140 near another person. It can be said that.

  As can be understood from the above description, the state information 502 included in the graphic information 500 is information indicating the state of the ellipse on the input image 140 and information indicating the state of the person on the input image 140. . It can be said that the state information 502 is information indicating the state of a person on the captured image 110, or can be said to be information indicating the state of a person in the room 300.

  The disappearance time 504 included in the graphic information 500 indicates the time when the tracking processing unit 12 determines that the ellipse that appeared on the input screen 140 no longer appears on the input screen 140. Therefore, when the state information 502 included in the graphic information 500 indicates “disappearance”, the disappearance time 504 is included in the graphic information 500. On the other hand, when the state information 502 included in the graphic information 500 indicates other than “disappearance”, the disappearance time 504 is not included in the graphic information 500. As will be described later, when the disappearance time 504 included in the graphic information 500 is more than a predetermined time before the current time, the graphic information 500 is deleted from the tracking information 50. That is, the graphic information 500 of the ellipse is deleted after a predetermined time has elapsed since the ellipse no longer appears on the input image 140. In other words, the ellipse graphic information 500 corresponding to the person is deleted after a predetermined time has passed since the person was not shown in the input image 140.

  The merge partner ID 505 included in the graphic information 500 is included in the graphic information 500 of the ellipse in which the ellipses corresponding to the graphic information 500 are merged when the state information 502 included in the graphic information 500 indicates “merged”. ID 501 is shown. When the ellipse corresponding to the graphic information 500 merges a plurality of ellipses, the merge partner ID 505 included in the graphic information 500 includes a plurality of IDs 501. When the status information 502 included in the graphic information 500 indicates other than “merged”, the graphic partner 500 does not include the merge partner ID 505.

  The number of merges 506 included in the graphic information 500 indicates the number of ellipses in which the ellipses corresponding to the graphic information 500 are merged when the state information 502 included in the graphic information 500 indicates “merge”. . When the number of merges 506 included in the graphic information 500 indicates N (N ≧ 1), the ellipse corresponding to the graphic information 500 is merged with N ellipses. When the number of merges 506 included in the graphic information 500 indicates N, it can be said that the ellipse corresponding to the graphic information 500 is applied to (N + 1) people. When the state information 502 included in the graphic information 500 indicates other than “merged”, the graphic information 500 does not include the merged number 506.

  Hereinafter, an ellipse whose state indicates “merged”, that is, an ellipse that is merged with another ellipse may be referred to as a “merging main ellipse”. In addition, an ellipse whose state indicates “merged”, that is, an ellipse merged with another ellipse may be referred to as a “merged ellipse”. An ellipse group including a merged ellipse and a merged ellipse in which the merged ellipse is merged may be referred to as a “merged ellipse group”. In addition, an ellipse whose state indicates “appearance” may be referred to as “appearance ellipse”, and an ellipse whose state indicates “disappearance” may be referred to as “disappearance ellipse”.

<About each component of the tracking processor>
Next, each configuration shown in FIG. 31 for the tracking processing unit 12 will be described. Hereinafter, the fitting process in the previous frame means the fitting process for the input image 140 of the previous frame, and the fitting process in the current frame means the fitting process for the input image 140 in the current frame. . In addition, the ellipse obtained in the fitting process in the previous frame may be referred to as a “first ellipse”. In addition, the ellipse obtained in the fitting process in the current frame may be referred to as a “second ellipse”. Further, the first ellipse of the merged subject ellipse may be referred to as “merged subject first ellipse”, and the second ellipse of the merged subject ellipse may be referred to as “merged subject second ellipse”. The first ellipse of the merged ellipse may be referred to as “merge first ellipse”, and the second ellipse of the merged ellipse may be referred to as “merge second ellipse”.

  The fitting process result 170 obtained by the fitting processing unit 11 includes ellipse specifying information indicating the second ellipse obtained by the fitting process in the current frame. On the other hand, the tracking information 50 in the storage unit 5 when the fitting process result 170 is input to the tracking processing unit 12 includes graphic information 500 indicating the first ellipse obtained by the fitting process in the previous frame. ing. In other words, the tracking information 50 before being updated based on the fitting process result 170 includes graphic information 500 indicating the first ellipse obtained by the fitting process in the previous frame. Therefore, it can be said that the tracking information 50 before the update indicates the result of the fitting process in the previous frame. Further, the tracking information 50 includes the ellipse graphic information 500 that is an ellipse obtained by the fitting process prior to the fitting process in the previous frame and that has already disappeared on the input image 140. Sometimes.

  In this example, every time the input image 140 is generated by the image processing unit 10, the tracking processing unit 12 is based on the fitting processing result 170 for the input image 140 and the tracking information 50 in the storage unit 50. For each first ellipse obtained by the fitting process in the previous frame, a first correspondence identifying process for identifying the second ellipse corresponding to the first ellipse obtained by the fitting process in the current frame is performed. The second ellipse corresponding to the first ellipse is a second ellipse applied to the same person as the person to whom the first ellipse was applied. Further, when the tracking information 50 includes the disappearing ellipse graphic information 500, the tracking processing unit 12 selects a second ellipse corresponding to the disappearing ellipse indicated by the graphic information 500 based on the graphic information 500. A second correspondence specifying process for specifying is performed. The second ellipse corresponding to the disappearing ellipse is a second ellipse applied to the same person as the person who has applied the disappearing ellipse. Thereby, the tracking processing unit 12 can track the position of the ellipse on the input image 140. Hereinafter, the first correspondence specifying process and the second correspondence specifying process may be collectively referred to as “correspondence specifying process”.

  In the tracking processing unit 12, the correspondence specifying unit 40 performs a correspondence specifying process. Then, the update unit 44 updates the tracking information 50 in the storage unit 5 based on the result of the correspondence specifying process in the correspondence specifying unit 40. Thereby, the position of the ellipse on the input image 140 is tracked. That is, the position of a person on the input image 140 is tracked.

  FIG. 33 is a flowchart showing the tracking process in the tracking processing unit 12. As shown in FIG. 33, in step s41, the primary specifying unit 41 of the correspondence specifying unit 40 performs a primary specifying process. Next, in step s42, the update unit 44 performs a primary update process for updating the tracking information 50 based on the result of the primary specifying process. Hereinafter, the primary identification process and the primary update process are collectively referred to as “primary process”.

  Next, in step s43, the secondary specifying unit 42 of the correspondence specifying unit 40 performs a secondary specifying process. Next, in step s44, the update unit 44 performs a secondary update process for updating the tracking information 50 based on the result of the secondary specifying process. Hereinafter, the secondary specifying process and the secondary update process are collectively referred to as “secondary process”.

  Next, in step s45, the update unit 44 determines whether or not the tracking information 50 includes the disappearing ellipse graphic information 500. When the tracking information 50 includes the disappearing ellipse graphic information 500, the update unit 44 determines whether the disappearance time 504 included in the disappearing ellipse graphic information 500 is a predetermined time or more before the current time. Determine whether. The update unit 44 deletes the graphic information 500 from the tracking information 50 when the disappearance time 504 included in the graphic information 500 is more than a predetermined time before the current time. As a result, the graphic information 500 of the ellipse is deleted from the tracking information 50 after a predetermined time has elapsed since the ellipse no longer appears on the input image 140. In other words, the ellipse graphic information 500 corresponding to the person is deleted after a predetermined time has elapsed since the person is no longer shown in the input image 140. When the tracking information 50 includes a plurality of disappearing ellipse graphic information 500, the update unit 44 performs the same processing for each graphic information 500. Hereinafter, the process in step s45 may be referred to as “2.5th order process” or “2.5th order update process”.

  Next, in step s46, the tertiary specifying unit 43 of the correspondence specifying unit 40 performs the tertiary specifying process. Next, in step s47, the update unit 44 performs a tertiary update process for updating the tracking information 50 based on the result of the tertiary specific process. When the tertiary update process is executed, the tracking process ends. Hereinafter, the tertiary identification process and the tertiary update process are collectively referred to as “tertiary process”.

  Note that when the fitting processing unit 11 performs a fitting process for the first time after the image processing apparatus 3 is activated, the update unit 44 generates graphic information 500 for each ellipse obtained by the fitting process. The generated graphic information 500 is stored in the storage unit 5 as the tracking information 50. The state of each ellipse obtained by the fitting process performed for the first time by the fitting processing unit 11 after the image processing apparatus 3 is activated is “appearance”. The update unit 44 updates the tracking information 50 in the storage unit 5 every time the fitting process result 170 is input to the tracking processing unit 12.

<Details of primary processing>
Next, the primary process will be described in detail. In the primary specifying process of the primary process, the primary specifying unit 41 first includes, in the tracking information 50, graphic information 500 (hereinafter, “merge” of the merged subject first ellipse (the first ellipse whose state indicates “merge”)). It is determined whether or not it is called “subject graphic information 500”. In other words, the primary specifying unit 41 determines whether or not the merged subject first ellipse is included in the first ellipse obtained in the fitting process in the previous frame. When the tracking information 50 does not include the merged subject graphic information 500, the primary specifying process ends. In this case, the primary update process is not performed.

  If the primary identification unit 41 determines that the tracking information 50 includes the merged subject graphic information 500, whether or not the merged subject first ellipse indicated by the merged subject graphic information 500 is separated into a plurality of second ellipses. A separation determination process is performed to determine The separation determination process will be described later in detail. When determining that the merged main first ellipse has been separated into a plurality of second ellipses, the primary specifying unit 41 specifies the plurality of second ellipses as second ellipses corresponding to the merged main first ellipse. When the tracking information 50 includes a plurality of merged subject graphic information 500, the primary specifying unit 41 performs the same process for each of the plurality of merged graphic information 500. Thereby, the 2nd ellipse corresponding to each merge main 1st ellipse contained in the 1st ellipse obtained in the fitting process in the previous frame is specified.

<Details of separation determination processing>
In the separation determination process, the primary identification unit 41 first applies a plurality of second ellipses in the fitting process in the current frame to a plurality of persons to which the merged subject first ellipse is fitted in the fitting process in the previous frame. It is determined whether or not is applied. The primary identification unit 41 determines that a plurality of second ellipses are applied in the fitting process in the current frame to a plurality of persons in which the merged subject first ellipse is applied in the fitting process in the previous frame. Then, it is determined that the merged main first ellipse is separated into the plurality of second ellipses. When the primary specifying unit 41 determines that the merged main first ellipse is separated into a plurality of second ellipses, the primary specifying unit 41 specifies the plurality of second ellipses as second ellipses corresponding to the merged main first ellipse. On the other hand, the primary identification unit 41 applies a plurality of second ellipses in the fitting process in the current frame to a plurality of persons to which the merging subject first ellipse is fitted in the fitting process in the previous frame. If it is determined that there is not, the primary specifying unit 41 does not specify the second ellipse corresponding to the merged subject first ellipse. Hereinafter, the number of the plurality of second ellipses when it is determined that the merging main first ellipse is separated into the plurality of second ellipses is referred to as “ellipse separation number”.

  34 and 35 are diagrams showing an example of a state in which the merged main first ellipse 250x is separated into a plurality of second ellipses 250y. 34 and 35, the merged subject first ellipse 250x is applied to three persons 200. FIG.

  In the fitting process in the previous frame, there are roughly two types of patterns for separating the merged main first ellipse 250x into a plurality of second ellipses 250y applied to three persons 200. As shown in FIG. 34, the first eye pattern is a pattern in which the merged main first ellipse 250x is separated into three second ellipses 250y in the fitting process in the current frame (the ellipse separation number is “3”). "). In this pattern, one second ellipse 250y is applied to each of the three persons 200 to which the first ellipse 250x is applied. In the first pattern, the merged main first ellipse 250x corresponds to the three second ellipses 250y.

  As shown in FIG. 35, the second pattern is a pattern in which the merging main first ellipse 250x is separated into two second ellipses 250y in the fitting process in the current frame (the ellipse separation number is “2”). "). In this pattern, out of the three persons 200 to which the merging subject first ellipse 250x is applied, one second ellipse 250y is applied to the two persons 200, and the remaining one person 200 is compared. One second ellipse 250y is applied. In the second pattern, the merged main first ellipse 250x corresponds to the two second ellipses 250y.

  The merged subject first ellipse is the same ellipse as the merged first ellipse included in the merged ellipse group to which it belongs. Therefore, it can be said that the primary specifying unit 41 determines whether or not the merged first ellipse is separated into a plurality of second ellipses. Further, it can be said that the plurality of second ellipses to which the merged subject first ellipse corresponds correspond to the merged first ellipse included in the merged ellipse group to which the merged subject first ellipse belongs. In addition, it can be said that the plurality of second ellipses to which the merged subject first ellipse corresponds correspond to each first ellipse constituting the merged ellipse group to which the merged subject first ellipse belongs.

  Next, a specific example of the separation determination process will be described. For example, when the merged subject first ellipse satisfies two conditions, the primary specifying unit 41 determines that the merged subject first ellipse is divided into a plurality of ellipses.

  The first condition is that the size of the second ellipse obtained by the fitting process in the current frame closest to the position of the merged subject first ellipse is sufficiently smaller than the size of the merged subject first ellipse. It is a condition. Hereinafter, the first condition may be referred to as a “separation first condition”.

  The second condition is a condition that at least a part of each of the plurality of second ellipses obtained by the fitting process in the current frame is included in the merged main first ellipse. Hereinafter, the second condition may be referred to as a “separation second condition”.

<Determination method of first separation condition>
When determining whether or not the separation first condition is satisfied, the primary identification unit 41 firstly is closest to the merged subject first ellipse among the second ellipses obtained by the fitting process in the current frame. The second ellipse at the position is specified as the nearest second ellipse. The primary specifying unit 41 can specify the nearest second ellipse based on the fitting processing result 170 and the tracking information 50 obtained by the fitting processing unit 11.

  Next, the primary specifying unit 41 determines whether or not the difference value obtained by subtracting the size of the nearest second ellipse from the size of the merged subject first ellipse is larger than the fourth threshold value. To do. For example, the primary specifying unit 41 employs the area of the merged subject first ellipse as the size of the merged subject first ellipse, and employs the area of the nearest second ellipse as the size of the nearest second ellipse. . The primary specifying unit 41 can obtain the size of the neighboring second ellipse from the ellipse specifying information about the nearest second ellipse included in the fitting process result 170. Further, the primary specifying unit 41 can determine the size of the merged subject first ellipse from the ellipse specifying information 503 of the merged subject graphic information 500 included in the tracking information 50.

  When the difference value is larger than the fourth threshold value, the primary specifying unit 41 determines that the size of the nearest second ellipse is sufficiently smaller than the size of the merged subject first ellipse. That is, the primary specifying unit 41 determines that the first separation condition is satisfied. On the other hand, the primary specific | specification part 41 determines with not satisfy | filling 1st separation conditions, when a difference value is below a 4th threshold value.

  The primary identification unit 41 determines that the separation first condition is satisfied when the difference value is equal to or greater than the fourth threshold value, and the separation first value when the difference value is less than the fourth threshold value. It may be determined that the condition is not satisfied.

<Method for determining separation second condition>
When the primary identification unit 41 determines whether or not the second separation condition is satisfied, first, at least a part of the second ellipse obtained by the fitting process in the current frame is: The second ellipse included in the merged main first ellipse is specified. Hereinafter, the second ellipse in which at least a part of the area is included in the merged main first ellipse is referred to as “encapsulated second ellipse”. For example, when the center of the second ellipse exists within the merged main first ellipse, the primary specifying unit 41 sets the second ellipse as the second ellipse to be included. The primary specifying unit 41 can specify the second ellipse to be included based on the fitting process result 170 and the tracking information 50. Then, the primary specifying unit 41 determines whether or not the number of contained ellipses included in the second ellipse obtained by the fitting process in the current frame is two or more.

  When the primary identification unit 41 determines that the number of the included ellipses included in the second ellipse obtained by the fitting process in the current frame is two or more, the primary specifying unit 41 has a plurality of pieces obtained by the fitting process in the current frame. It is determined that at least a part of the second ellipse is included in the merged main first ellipse. That is, the primary specifying unit 41 determines that the second separation condition is satisfied. On the other hand, if it is determined that the number of second ellipses to be included included in the second ellipse obtained by the fitting process in the current frame is one, it is determined that the second separation condition is not satisfied.

  As described above, the primary specifying unit 41 determines whether or not the merged subject first ellipse satisfies the separation first condition and the separation second condition. When the merged subject first ellipse satisfies the separation first condition and the separation second condition, the primary identification unit 41 applies the current frame to a plurality of persons to which the merged subject first ellipse is applied. It is determined that a plurality of second ellipses are fitted in the fitting process at. That is, the primary specifying unit 41 determines that the merging main first ellipse is separated into a plurality of second ellipses. Specifically, the primary specifying unit 41 determines that the merging main first ellipse is separated into a plurality of encapsulated second ellipses specified by the separation second condition. And the primary specific | specification part 41 specifies a several to-be-enclosed 2nd ellipse as a 2nd ellipse corresponding to the merged main body 1st ellipse. Since the merged subject first ellipse is the same ellipse as the merged ellipse to which it is merged, the primary specifying unit 41 constitutes a plurality of contained second ellipses and a merged ellipse group including the merged subject first ellipses. It can be said that it identifies as the 2nd ellipse corresponding to each 1st ellipse to do.

  In this way, by determining whether or not a plurality of second ellipses are applied in the fitting process in the current frame for a plurality of people to which the merging subject first ellipse is applied, Can be appropriately associated. Therefore, by tracking the position of the ellipse on the input image 140, it is possible to appropriately track the position of the person on the input image 140.

  The primary identification unit 41 may determine whether or not the second separation condition is satisfied by a method different from the above. Hereinafter, an example of another determination method for the second separation condition will be described.

  The primary specifying unit 41 uses four intersection points 257y for the second ellipse 250y as shown in FIG. The four intersection points 257y are composed of two intersection points 257y where the second ellipse 250y and the major axis 255y intersect, and two intersection points 257y where the second ellipse 250y and the minor axis 256y intersect. The primary specifying unit 41 selects a second ellipse having a predetermined number or more of the intersection points 257y out of the four intersection points 257y in the merged subject first ellipse among the second ellipses obtained by the fitting process in the current frame. , Specify as the second ellipse to be encapsulated. This predetermined number is set to a value between 1 and 4. FIG. 37 is a diagram illustrating an example of a second ellipse 250y in which two of the four intersections 257y exist in the merged main first ellipse 250x. When the predetermined number is set to “2”, for example, the second ellipse 250y shown in FIG. 37 is an encapsulated second ellipse 250y in which at least a part of the second ellipse 250y is included in the merged main ellipse 250x.

  When the primary specifying unit 41 specifies the second encapsulated ellipse, the primary specifying unit 41 operates in the same manner as described above, and the number of included ellipses included in the second ellipse obtained by the fitting process in the current frame is two or more. It is determined whether or not there is. If the primary specifying unit 41 determines that the number of contained ellipses included in the second ellipse obtained by the fitting process in the current frame is two or more, it determines that the second separation condition is satisfied. On the other hand, if it is determined that the number of second ellipses to be included included in the second ellipse obtained by the fitting process in the current frame is one, it is determined that the second separation condition is not satisfied.

  The first ellipse in which the corresponding second ellipse is specified may be referred to as “specified first ellipse”. The second ellipse specified as corresponding to the first ellipse may be referred to as “specified second ellipse”. In addition, a first ellipse for which the corresponding second ellipse is not specified may be referred to as an “unspecified first ellipse”. A second ellipse that is not specified as corresponding to the first ellipse may be referred to as an “unspecified second ellipse”.

  When the primary specifying unit 41 specifies a plurality of contained second ellipses as the second ellipse corresponding to the merged main first ellipse, the plurality of contained second ellipses become the specified second ellipses, and merged. The subject first ellipse becomes the identified first ellipse. Further, the merged first ellipse merged with the merged main first ellipse is also the identified first ellipse.

  Hereinafter, the merged ellipse group to which the merged subject first ellipse that is the identified first ellipse may be referred to as a “specified merged first ellipse group”. Each first ellipse included in the identified merged first ellipse group is a specified first ellipse. Further, the total number of ellipses of the specified merged first ellipse group means the total number of first ellipses constituting the specified merged first ellipse group. The total number of ellipses of the specified merged first ellipse group is a value obtained by adding “1” to the merge number 506 included in the graphic information 500 of the merged subject first ellipse belonging to the specified merged first ellipse group. It becomes. Speaking of the number of ellipse separations corresponding to the specified merged first ellipse group, the number of ellipse separations of the merged subject first ellipse belonging to the specified merged first ellipse group, that is, corresponding to the merged subject first ellipse. It means the number of a plurality of second ellipses. Speaking of a plurality of specified second ellipses corresponding to the specified merged first ellipse group means a plurality of second ellipses corresponding to the merged subject first ellipse belonging to the specified merged first ellipse group.

<Details of the primary update process>
Next, the primary update process will be described in detail. In the primary update process, the update unit 44 updates the graphic information 500 of each first ellipse of the specified merged first ellipse group obtained in the primary specification process. That is, the update unit 44 updates the merged subject first ellipse graphic information 500 in which the corresponding second ellipse is specified, and the merged first ellipse graphic information 500 merged with the merged subject first ellipse. When the plurality of specified merged first ellipse groups are obtained in the primary specifying process, the updating unit 44 sets the specified merged first ellipse group for each of the plurality of specified merged first ellipse groups. The graphic information 500 of each first ellipse is updated. If the identified merged first ellipse group is not obtained in the primary identification process, the primary update process is skipped.

  The identified merged first ellipse group is divided into a case where the total number of ellipses of the identified merged first ellipse group and the number of ellipse separations corresponding to the identified merged first ellipse group are not matched. A method for updating the graphic information 500 of each first ellipse will be described.

<When the total number of ellipses matches the number of ellipses separated>
When the total number of ellipses of the specified merged first ellipse group matches the number of ellipse separations corresponding to the specified merged first ellipse group, the updating unit 44 performs a plurality of operations corresponding to the specified merged first ellipse group. The graphic information 500 of a plurality of first ellipses (specified first ellipses) constituting the specified merged first ellipse group is associated with each specified second ellipse. And the update part 44 updates the graphic information 500 of each 1st ellipse of the identified merged 1st ellipse group based on the ellipse specific information of the identified 2nd ellipse which matched it. Specifically, the updating unit 44 uses the ellipse specifying information 503 of the graphic information 500 of the first ellipse included in the specified merged first ellipse group as the ellipse of the specified second ellipse associated with the graphic information 500. The ellipse identification information 503 is updated so as to match the identification information. Then, the update unit 44 sets the state information 502 of the graphic information 500 to “exist”. Thereby, the graphic information 500 of each first ellipse of the identified merged first ellipse group is updated. The updated graphic information 500 about the first ellipse becomes the graphic information 500 of the second ellipse corresponding to the first ellipse. Note that the ellipse specifying information of the specified second ellipse is included in the fitting process result 170.

  FIG. 38 shows an example of correspondence between graphic information 500 of a plurality of first ellipses constituting a specified merged first ellipse group and a plurality of specified second ellipses corresponding to the specified merged first ellipse group. FIG. In the example of FIG. 38, the identified merged first ellipse group is composed of four first ellipses. Therefore, the total number of ellipses of the specified merged first ellipse group is “4”. The merged main first ellipse included in the specified merged first ellipse group is separated into four second ellipses 250ya to 250yd. Therefore, the ellipse separation number is “4”, which is equal to the total number of ellipses. In the example of FIG. 38, four graphic information 500a to 500d respectively corresponding to the four first ellipses constituting the identified merged first ellipse group are associated with the four second ellipses 250ya to 250yd. Yes. By updating the graphic information 500a to 500d by the updating unit 44, the graphic information 500a becomes information on the second ellipse 250ya, the graphic information 500b becomes information on the second ellipse 250yb, and the graphic information 500c is information on the second ellipse 250yc. Thus, the graphic information 500d is information related to the second ellipse 250yd.

<When the total number of ellipses does not match the number of ellipse separation>
Next, update when the total number of ellipses of the specified merged first ellipse group does not match the number of divisions corresponding to the specified merged first ellipse group, that is, when the total number of ellipses is greater than the number of ellipse separations The operation of the unit 44 will be described. In this case, the updating unit 44 associates the same number of graphic information 500 as much as possible with each of the plurality of specified second ellipses corresponding to the specified merged first ellipse group and is different from each other. , And at least one of the plurality of first ellipse graphic information 500 constituting the specified merged first ellipse group so that there is no specified second ellipse to which the graphic information 500 is not associated. Associate.

  When the updating unit 44 associates one graphic information 500 with one specified second ellipse, the ellipse specifying information 503 of the graphic information 500 corresponds to the graphic information 500 as described above. The ellipse specification information 503 is updated so as to match the ellipse specification information of the specified second ellipse. Then, the update unit 44 sets the state information 502 of the graphic information 500 to “exist”. Thereby, the graphic information 500 is updated.

  On the other hand, when the updating unit 44 associates M pieces (M ≧ 2) of graphic information 500 with one specified second ellipse, the one specified second ellipse is M persons. Let it be the second ellipse fitted to 200. Then, the update unit 44 sets the state information 502 included in one graphic information 500 to “merged” in the M pieces of graphic information 500, and sets the state information included in the remaining graphic information 500 to “merged”. Set to. Then, the update unit 44 includes the ID 501 of the graphic information 500 in which the status information 502 is set to “merged” in the merged partner ID 505 of the graphic information 500 in which the status information 502 is set to “merged”. Further, the updating unit 44 sets (M−1) as the merge number 505 of the graphic information 500 in which the state information 502 is set to “merged”. Then, for each of the M pieces of graphic information 500, the update unit 44, like the above, the ellipse specifying information 503 of the graphic information 500 is the ellipse of the specified second ellipse associated with the graphic information 500. The ellipse identification information 503 is updated so as to match the identification information. Thereby, the M pieces of graphic information 500 are updated.

  39 and 40 show the correspondence between the graphic information 500 of the plurality of first ellipses constituting the specified merged first ellipse group and the plurality of specified second ellipses corresponding to the specified merged first ellipse group. It is a figure which shows an example. In the examples of FIGS. 39 and 40, the number of ellipse separations is different from the example of FIG. 38 described above.

  In the example of FIG. 39, the merged subject first ellipse included in the identified merged first ellipse group is separated into two second ellipses 250ya and 250yb, and the ellipse separation number is “2”. In the example of FIG. 39, two pieces of graphic information 500a and 500b are associated with the second ellipse 250ya, and two pieces of graphic information 500c and 500d are associated with the second ellipse 250yb. In the example of FIG. 39, the update unit 44 sets the state information 502 of one graphic information 500 to “merged” in the two graphic information 500a and 500b, and sets the state information 502 of the other graphic information 500 to “merged”. To "". Here, for example, it is assumed that the state information 502 of the graphic information 500a is set to “merge” and the state information 502 of the graphic information 500b is set to “merge”. The update unit 44 includes the ID 501 of the graphic information 500b in which the status information 502 is set to “merged” in the merge partner ID 505 of the graphic information 500a in which the status information 502 is set to “merged”. Further, the updating unit 44 sets the merged number 505 of the graphic information 500a to “1”. Then, the update unit 44 updates the ellipse specifying information 503 so that the ellipse specifying information 503 of the graphic information 500a matches the ellipse specifying information of the second ellipse 250ya, and the ellipse specifying information 503 of the graphic information 500b The ellipse specifying information 503 is updated so as to coincide with the ellipse specifying information of the second ellipse 250ya. Similarly, the update unit 44 updates two pieces of graphic information 500c and 500d associated with the second ellipse 250yb.

  In the example of FIG. 40, the merged subject first ellipse included in the identified merged first ellipse group is separated into three second ellipses 250ya, 250yb, 250yc, and the ellipse separation number is “3”. In the example of FIG. 40, one graphic information 500a is associated with the second ellipse 250ya, one graphic information 500b is associated with the second ellipse 250yb, and two graphic information 500c, 500d is associated with the second ellipse 250yc. Are associated. In the example of FIG. 40, the updating unit 44 changes the ellipse specifying information 503 so that the ellipse specifying information 503 of the graphic information 500a associated with the second ellipse 250ya matches the ellipse specifying information of the second ellipse 250ya. Update. Then, the update unit 44 sets the state information 502 of the graphic information 500a to “exist”. The updating unit 44 similarly updates the graphic information 500b associated with the second ellipse 250yb. Further, the update unit 44 sets the state information 502 of one graphic information 500 to “merged” in the graphic information 500c and 500d associated with the second ellipse 250yc, and sets the state information 502 of the other graphic information 500 to Set to “merged”. Then, the update unit 44 includes the ID 501 of the graphic information 500 in which the status information 502 is set to “merged” in the merged partner ID 505 of the graphic information 500 in which the status information 502 is set to “merged”. Further, the updating unit 44 sets the merged number 505 of the graphic information 500 in which the state information 502 is set to “merged” to “1”. Then, the update unit 44 updates the ellipse specifying information 503 so that the ellipse specifying information 503 of the graphic information 500c matches the ellipse specifying information of the second ellipse 250yc, and the ellipse specifying information 503 of the graphic information 500d The ellipse specifying information 503 is updated so as to coincide with the ellipse specifying information of the second ellipse 250yc. As a result, the graphic information 500a to 500d is updated.

  As described above, in the primary update process, the graphic information 500 of the plurality of first ellipses constituting the first merged first ellipse group obtained in the primary identification process is updated.

<Details of secondary processing>
Next, the secondary processing will be described in detail. In the secondary specifying process of the secondary process, the secondary specifying unit 42 first sets the first ellipse in which the corresponding second ellipse is not specified in the primary specifying process in the tracking information 50 that has been subjected to the primary update process. The graphic information 500 (hereinafter, sometimes referred to as “unspecified graphic information 500”) is included. In other words, the secondary specifying unit 42 performs the tracking information 50 on which the primary update processing has been performed. Whether or not the graphic information 500 of the unspecified first ellipse is included, and in other words, the secondary specifying unit 42 is not updated to the tracking information 50 subjected to the primary update process. It is determined whether information 500 is included.

  When the secondary specifying unit 42 determines that the unspecified graphic information 500 is included in the tracking information 50 that has been subjected to the primary update process, the state information included in the tracking information 50 that has been subjected to the primary update process Based on the predetermined information, the second ellipse corresponding to the unspecified first ellipse indicated by the unspecified graphic information 500 in which 502 indicates other than “merge” is specified. The unspecified graphic information 500 whose status information 502 indicates other than “merged” is particularly referred to as “target unspecified graphic information 500”. When the tracking information 50 subjected to the primary update process includes the graphic information 500 of the disappearing ellipse, the secondary specifying unit 42 specifies the second ellipse corresponding to the disappearing ellipse. I do. When the tracking information 50 includes a plurality of target unspecified graphic information 500, the secondary specifying unit 41 performs the same process for each of the plurality of target unspecified graphic information 500. If the secondary identification unit 42 determines that the unspecified graphic information 500 is not included in the tracking information 50 that has undergone the primary update process, the secondary identification process ends. In this case, the secondary update process is not performed.

  Hereinafter, the unspecified first ellipse indicated by each target unspecified graphic information 500 included in the tracking information 50 that has been subjected to the primary update process may be referred to as a “second ellipse to be processed secondarily”. The first ellipse of the secondary process does not include the first ellipse whose state is “merged”. Further, among the second ellipses obtained in the fitting process in the current frame, the second ellipses other than the identified second ellipse obtained in the primary identification process are referred to as “second ellipses to be subjected to secondary processing”. There is. The secondary specifying unit 42 specifies the second ellipse corresponding to the first ellipse to be subjected to the secondary processing from the second ellipse to be subjected to the secondary processing using, for example, the nearest neighbor method (Nearest Neighbor). FIG. 41 is a flowchart showing the secondary specifying process.

  As shown in FIG. 41, in step s431, the secondary specifying unit 42 determines whether or not the unspecified graphic information 500 is included in the tracking information 50 that has undergone the primary update process. If the secondary specifying unit 41 determines that the unspecified graphic information 500 is not included in the tracking information 50 that has undergone the primary update process, the secondary specifying process ends. On the other hand, when determining that the unspecified graphic information 500 is included in the tracking information 50 that has been subjected to the primary update process, the secondary specifying unit 41 executes step s432.

  In step s432, the secondary specifying unit 42 selects one first ellipse from the first ellipses to be subjected to the secondary processing, and sets the selected first ellipse as the selected first ellipse. After step s432, in step s433, the secondary specifying unit 42 selects the second ellipse closest to the selected first ellipse from the unspecified second ellipse included in the second ellipse to be subjected to the secondary processing. Identify as 2 ellipses. In step s433, the secondary specifying unit 42 sets, for example, the unspecified second ellipse having the center closest to the center of the selected first ellipse as the nearest second ellipse.

  Next, in step s434, the secondary specifying unit 42 obtains the distance between the selected first ellipse and the nearest second ellipse. For example, the secondary specifying unit 42 sets the distance between the center of the selected first ellipse and the center of the nearest second ellipse as the distance between the selected first ellipse and the nearest second ellipse. And the secondary specific | specification part 42 determines whether the calculated | required distance is below a 5th threshold value.

  If the secondary specifying unit 42 determines in step s434 that the obtained distance is equal to or smaller than the fifth threshold value, in step s435, the nearest second ellipse is determined as the second ellipse corresponding to the selected first ellipse. To do. As a result, the selected first ellipse becomes the identified first ellipse. Further, the nearest second ellipse becomes the specified second ellipse, and the specified second ellipse is included in the second ellipse to be subjected to the secondary processing. After step s435, the secondary specifying unit 42 executes step s436. On the other hand, when determining that the obtained distance is larger than the fifth threshold value in step s434, the secondary specifying unit 42 executes step s436. If it is determined that the obtained distance is greater than the fifth threshold, the nearest second ellipse remains the unspecified second ellipse.

  In step s436, the secondary specifying unit 42 determines whether or not all of the first ellipses to be processed are processed. That is, the secondary specifying unit 42 determines whether or not all the first ellipses to be subjected to the secondary processing are selected first ellipses. If it determines with Yes in step s436, a secondary specific process will be complete | finished. On the other hand, if it determines with No in step s436, the secondary specific | specification part 42 will perform step s432 again. Here, among the first ellipses to be subjected to the secondary processing, the first ellipse that has not yet been processed, that is, the first ellipse that has not yet been selected, is selected as the selected first ellipse. Next, the secondary specifying unit 41 executes step s433, and selects the second ellipse closest to the selected first ellipse from the unspecified second ellipse included in the second ellipse to be subjected to the secondary processing. The second ellipse is specified. Thereafter, the secondary specifying unit 41 operates in the same manner.

  As described above, the secondary specifying unit 42 is the second ellipse to be processed secondarily closest to the second ellipse to be processed, and the distance from the first ellipse is equal to or less than the fifth threshold value. Is identified as the second ellipse corresponding to the first ellipse.

  In step s434 described above, the secondary specifying unit 42 may determine whether the distance between the selected first ellipse and the nearest second ellipse is less than the fifth threshold value. In this case, the secondary specifying unit 42 is the second ellipse to be secondary-processed closest to the first ellipse to be secondary-processed, and the distance from the first ellipse is the fifth threshold value. The second ellipse to be subjected to the secondary processing is less than the second ellipse corresponding to the first ellipse.

<Details of secondary update processing>
Next, the secondary update process will be described in detail. In the secondary update process, the update unit 44 sets the graphic information 500 of each identified first ellipse newly obtained in the secondary specification process as the graphic information 500 to be secondary updated. Then, the updating unit 44 updates each piece of graphic information 500 to be subjected to secondary update based on the ellipse specifying information of the second ellipse corresponding to the specified first ellipse indicated by the graphic information 500.

  The method for updating the graphic information 500 in the secondary update process differs depending on the value of the state information 502 included in the graphic information 500.

  For example, when the state information 502 of the graphic information 500 that is the secondary update target indicates “appearance”, the update unit 44 continuously displays an ellipse (person) corresponding to the graphic information 500 on the input image 140. And the status information 502 is changed to “exist”. Then, the updating unit 44 sets the ellipse specifying information 503 so that the ellipse specifying information 503 of the graphic information 500 matches the ellipse specifying information of the second ellipse corresponding to the specified first ellipse indicated by the graphic information 500. Update.

  When the state information 502 of the graphic information 500 that is the secondary update target indicates “exist”, the update unit 44 keeps the state information 502 “exist”. Then, the updating unit 44 sets the ellipse specifying information 503 so that the ellipse specifying information 503 of the graphic information 500 matches the ellipse specifying information of the second ellipse corresponding to the specified first ellipse indicated by the graphic information 500. Update.

  When the state information 502 of the graphic information 500 that is the secondary update target indicates “merge”, the update unit 44 keeps the state information 502 “merge”. Then, the updating unit 44 causes the ellipse specifying information 503 of the graphic information 500 to match the ellipse specifying information of the second ellipse corresponding to the specified first ellipse (the merged main first ellipse) indicated by the graphic information 500. The ellipse specifying information 503 is updated.

  In addition, when the state information 502 of the graphic information 500 that is the secondary update target indicates “merged”, the updating unit 44 merges each merged first ellipse merged with the merged subject first ellipse indicated by the graphic information 500. The ellipse graphic information 500 is also updated. That is, when the state information 502 of the graphic information 500 that is the secondary update target indicates “merge”, the update unit 44 also updates the graphic information 500 including the ID 501 indicated by the merge partner ID 505 included in the graphic information 500. . Specifically, the update unit 44 includes the ellipse specifying information 503 of the merged first ellipse graphic information 500, and the ellipse specifying information of the second ellipse corresponding to the merged first ellipse that merges the merged first ellipse. The ellipse identification information 503 is updated so as to match. The state information 502 of the graphic information 500 of the merged first ellipse is not changed from “merge”. Thereby, according to the update of the graphic information 500 of the merged subject ellipse, the graphic information 500 of each merged ellipse to which the merged subject ellipse merges is updated.

  When the state information 502 of the graphic information 500 to be subjected to the secondary update process indicates “disappearance”, the update unit 44 determines that an ellipse (person) corresponding to the graphic information 500 has appeared on the input image 140. The status information 502 is changed to “exist”. The update unit 44 also updates the ellipse specifying information 503 so that the ellipse specifying information 503 of the graphic information 500 matches the ellipse specifying information of the second ellipse corresponding to the specified first ellipse indicated by the graphic information 500. Update. Then, the update unit 44 deletes the disappearance time 504 included in the graphic 500.

  As described above, the tracking information 50 subjected to the primary update process is updated based on the result of the secondary specifying process.

  The update unit 44 performs 2.5th order processing (2.5th order update processing) after the secondary processing. That is, the update unit 44 determines whether or not the tracking information 50 that has undergone the primary update process and the secondary update process includes the graphic information 500 of the lost ellipse. When the tracking information 50 includes the disappearing ellipse graphic information 500, the update unit 44 determines whether the disappearance time 504 included in the disappearing ellipse graphic information 500 is a predetermined time or more before the current time. Determine whether. The update unit 44 deletes the graphic information 500 from the tracking information 50 when the disappearance time 504 included in the graphic information 500 is more than a predetermined time before the current time.

  Thus, in this example, the ellipse graphic information 500 of the disappearing ellipse, that is, the ellipse graphic information 500 corresponding to the person who no longer appears in the input image 140 is not deleted immediately, but a certain time has passed since the person disappeared. Have been deleted after. This allows the same graphic information for the person to appear before and after it disappears when it becomes visible immediately after it disappears, such as when a person is temporarily hidden behind objects. 500 can be associated. Therefore, even when a person temporarily disappears in the input image 140, the position of the person can be tracked.

<Details of tertiary processing>
As described above, in the secondary specifying process, the second ellipse associated with the first ellipse is subsequently excluded from candidates for specifying the second ellipse corresponding to another first ellipse. Therefore, when a common second ellipse is applied to a plurality of people who have been fitted with a plurality of first ellipses, the second ellipse is a second corresponding to all of the plurality of first ellipses. In spite of being an ellipse, in the secondary specifying process, the second ellipse is associated with only one first ellipse among the plurality of first ellipses.

  FIG. 42 is a diagram illustrating an example of a state in which a common second ellipse is applied to a plurality of persons to which a plurality of first ellipses are respectively applied. In the example of FIG. 42, one common second ellipse 250y is applied to the three persons 200 to which the three first ellipses 250x1, 250x2, and 250x3 are respectively applied. In the example of FIG. 42, although the second ellipse 250y is a second ellipse corresponding to all of the first ellipses 250x1, 250x2, and 250x3, in the secondary specifying process, the second ellipse 250y is the first ellipse. It is specified as a second ellipse corresponding to only one first ellipse of 250x1, 250x2, 250x3. Therefore, after the secondary specifying process, the one first ellipse becomes the specified first ellipse, the second ellipse 250y becomes the specified second ellipse, and the remaining two of the first ellipses 250x1, 250x2, 250x3. The first ellipse is an unspecified first ellipse. Therefore, after the secondary update process, the remaining two first ellipse graphic information 500 is not updated.

  In this example, when one second ellipse is applied to a plurality of people who have been fitted with a plurality of first ellipses, for convenience, the second ellipse in the secondary process among the plurality of first ellipses. It is considered that the specified first ellipse associated with is transitioned to the second ellipse, and the remaining unspecified first ellipses are merged with the second ellipse.

  In the tertiary specifying process, the tertiary specifying unit 43 includes the unspecified first ellipse graphic information 500 (unspecified graphic information 500) included in the tracking information 50 that has been subjected to the 2.5th update process. It is determined whether the first ellipse is merged with the identified second ellipse. Then, when the unspecified first ellipse is merged with the specified second ellipse, the tertiary specifying unit 43 converts the specified second ellipse into the second ellipse corresponding to the unspecified first ellipse. As specified. Thereby, when one second ellipse is applied to a plurality of persons to which a plurality of first ellipses are applied, the second ellipse is associated in the secondary process among the plurality of first ellipses. The unspecified first ellipse that did not exist can be appropriately associated with the second ellipse. Hereinafter, the tertiary specifying process will be described in detail.

<Details of tertiary identification processing>
In the tertiary specifying process, the tertiary specifying unit 43 determines whether or not the unspecified graphic information 500 is included in the tracking information 50 that has been subjected to the 2.5th update process. In other words, the tertiary specifying unit 43 determines whether the graphic information 500 that has not yet been updated is included in the tracking information 50 that has undergone the primary update process, the secondary update process, and the 2.5th update process. judge.

  When the tertiary specifying unit 43 determines that the unspecified graphic information 500 is included in the tracking information 50 that has been subjected to the 2.5th order update process, It is determined whether or not the unspecified first ellipse indicated by the unspecified graphic information 500 (target unspecified graphic information 500) whose state information 502 is other than “merge” is merged with the specified second ellipse. Perform merge determination processing. The merge determination process will be described later. And if the tertiary specific | specification part 43 determines with the unspecified 1st ellipse being merged with the specified 2nd ellipse, the said specified 2nd ellipse will be a 2nd ellipse corresponding to an unspecified 1st ellipse. Identifies it. When the tracking information 50 that has undergone the 2.5th update process includes a plurality of target unspecified graphic information 500, the tertiary specifying unit 43 sets each of the plurality of target unspecified graphic information 500. The same processing is performed. If the tertiary identification unit 44 determines that the unspecified graphic information 500 is not included in the tracking information 50 that has undergone the 2.5th-order update process, the tertiary identification process ends. Note that the unspecified first ellipse that is determined not to be merged with the specified second ellipse becomes a lost ellipse in the tertiary update process.

  Hereinafter, the unspecified first ellipse for which it is determined whether or not it has been merged with the specified second ellipse may be referred to as a “first ellipse to be tertiary processed”.

<Details of merge determination process>
In the merge determination process, the tertiary specifying unit 43 determines that the first ellipse is merged with the specified second ellipse when the first ellipse to be subjected to the tertiary process satisfies two conditions.

  The first condition is that there is a specified second ellipse whose position is close to the first ellipse to be subjected to the tertiary processing. Hereinafter, the first condition may be referred to as “merging first condition”. Further, the specified second ellipse whose position is close to the first ellipse to be subjected to the tertiary processing is referred to as “neighbor specified second ellipse”.

  The second condition is a condition that the second ellipse whose neighborhood has been specified includes at least a part of the first ellipse to be subjected to the tertiary processing. Hereinafter, the second condition may be referred to as “merge second condition”.

<Judgment method of merger first condition>
When the tertiary specifying unit 43 determines whether or not the first condition for merging is satisfied, the second ellipse whose neighborhood has been specified exists in the specified second ellipse obtained by the primary process and the secondary process. It is determined whether or not to do. For example, the tertiary specifying unit 43 sets the specified second ellipse having a center whose distance from the center of the first ellipse to be subjected to the tertiary processing is equal to or less than a sixth threshold as the neighborhood specified second ellipse. That is, the tertiary specifying unit 43 has a center whose distance from the center of the first ellipse to be subjected to the tertiary processing is the sixth threshold value or less in the specified second ellipse obtained by the primary processing and the secondary processing. It is determined whether or not the specified second ellipse exists. If the tertiary specifying unit 44 determines that the second ellipse in the vicinity specified in the specified second ellipse obtained by the primary process and the secondary process exists, it determines that the merged first condition is satisfied. On the other hand, if the tertiary specifying unit 43 determines that there is no neighboring specified second ellipse in the specified second ellipse obtained by the primary processing and the secondary processing, it determines that the merged first condition is not satisfied. . Note that the specified second ellipse having a center whose distance from the center of the first ellipse to be third-processed is less than the sixth threshold value may be set as the second ellipse whose neighborhood has been specified.

<Judgment method of merger second condition>
The tertiary specifying unit 43 determines whether or not the merged second condition is satisfied for the first ellipse for the third processing target only when the merged first condition is satisfied for the first ellipse for the third processing target. . In the identified second ellipse obtained by the primary processing and the secondary processing, the tertiary processing unit 43, when there is only one neighborhood identified second ellipse, the neighborhood identified second ellipse is tertiary. It is determined whether or not at least a part of the first ellipse to be processed is included. On the other hand, in the identified second ellipse obtained by the primary process and the secondary process, the tertiary processing unit 43, when there are a plurality of neighborhood identified second ellipses, the plurality of neighborhood identified second ellipses. Among them, the second ellipse whose neighborhood has been identified that is closest to the first ellipse to be subjected to the tertiary processing is used. Then, the tertiary specifying unit 43 determines whether or not the second ellipse whose vicinity has been specified includes at least a part of the first ellipse to be processed. The tertiary processing unit 43 uses, for example, the second identified ellipse whose distance between the center of the plurality of second identified ellipses and the center of the first ellipse to be third-processed is the smallest as the target of use. To do.

  When determining whether or not the merged second condition is satisfied, the tertiary specifying unit 43 uses, for example, four intersections of the first ellipse to be subjected to the tertiary processing and its long and short axes. (See FIG. 36 above). The tertiary specifying unit 43, when a predetermined number or more of the four intersections are present in the second ellipse whose neighborhood has been specified, the second ellipse whose neighborhood has been specified is at least a part of the first ellipse to be subjected to the tertiary processing Is determined to be included. That is, the tertiary identification unit 43 determines that the merged second condition is satisfied. On the other hand, the tertiary specification unit 43 determines that the merged second condition is not satisfied when only the intersections less than the predetermined number among the four intersections exist in the second ellipse whose neighborhood has been specified.

  As described above, the tertiary specifying unit 43 determines whether or not the first ellipse to be subjected to the tertiary processing satisfies the merge first condition and the merge second condition. Then, the tertiary specifying unit 43 merges the first ellipse into the second ellipse whose neighborhood has been specified when the first ellipse to be subjected to the tertiary processing satisfies the merged first condition and the merged second condition. It is determined that Then, the tertiary specifying unit 43 specifies the second ellipse whose neighborhood has been specified as the second ellipse corresponding to the first ellipse to be subjected to the tertiary processing. Thereby, the first ellipse to be subjected to the tertiary processing becomes the identified first ellipse.

  In the case shown in FIG. 42 described above, the same specified first ellipse may be specified as the second ellipse corresponding to the plurality of first ellipses to be tertiary processed. For example, in the example of FIG. 42, if the first ellipse to be subjected to the tertiary processing is the first ellipse 250x1, 250x2, the second ellipse corresponding to the two first ellipses 250x1, 250x2 in the tertiary specifying process. As a result, the second ellipse 250y is specified.

  The tertiary identification unit 43 may determine whether or not the second merged condition is satisfied by a method different from the above. For example, when the center of the first ellipse to be subjected to the third processing exists in the second ellipse whose neighborhood has been specified, the tertiary specifying unit 43 has at least a part of the first ellipse as the second ellipse whose neighborhood has been specified It may be determined that it is included.

  In addition, the tertiary specifying unit 43 does not use the merged second condition, and when the first ellipse to be subjected to the tertiary processing satisfies the merged first condition, the first ellipse is determined as the neighborhood specified second ellipse. It may be determined that they have been merged.

  The tertiary specifying unit 43 includes a specified second ellipse that includes at least a part of the first ellipse to be processed in the specified second ellipse obtained by the primary processing and the secondary processing. In this case, it may be determined that the first ellipse is merged with the specified second ellipse.

  In this way, by determining whether or not the first ellipse to be subjected to the tertiary processing is merged with the specified second ellipse, it is possible to appropriately associate the person and the ellipse. Therefore, by tracking the position of the ellipse on the input image 140, it is possible to appropriately track the position of the person on the input image 140.

  In addition, when the first ellipse subject to the tertiary processing satisfies both the merged first condition and the merged second condition, it is determined that the first ellipse has been merged into the neighborhood-specified second ellipse. And an ellipse can be associated more appropriately. Therefore, the position of the person in the input image 140 can be tracked more appropriately.

<Details of tertiary update processing>
Next, the tertiary update process will be described in detail. In the tertiary update process, the update unit 44 updates the graphic information 500 of each specified first ellipse newly obtained in the tertiary specification process. Hereinafter, the graphic information 500 of the specified first ellipse to be subjected to the tertiary update process may be referred to as “first elliptic graphic information 500 to be subjected to the tertiary update”.

  Further, the updating unit 44 updates the graphic information 500 of the specified second ellipse corresponding to the specified first ellipse newly obtained by the tertiary specifying process. The graphic information 500 is graphic information 500 that has been subjected to secondary update processing. Hereinafter, the graphic information 500 of the specified second ellipse to be subjected to the tertiary update process may be referred to as “second elliptic graphic information 500 to be subjected to the tertiary update”.

  The updating unit 44 sets the state information 502 of the first ellipse graphic information 500 to “merged” when updating the first ellipse graphic information 500 to be tertiary updated. The update unit 44 also specifies the ellipse specification of the specified second ellipse corresponding to the specified first ellipse indicated by the first ellipse graphic information 500 in the ellipse specification information 503 of the first ellipse graphic information 500 to be subjected to the tertiary update. The ellipse identification information 503 is updated so as to match the information.

  The update unit 44 updates the second ellipse graphic information 500 that is the third update target when the state information 502 of the second ellipse graphic information 500 is “merged” or a value other than “merged”. And different.

  First, the operation of the updating unit 44 when the state information 502 of the second ellipse graphic information 500 to be tertiary updated is “merged” will be described. In this case, the update unit 44 keeps the state information 502 of the second ellipse graphic information 500 that is the third update target as “merged”. Further, the updating unit 44 does not change the ellipse specifying information 503 of the second ellipse graphic information 500 that is the third update target. Then, the update unit 44 updates the merge partner ID 505 and the merge number 506 of the second ellipse graphic information 500 that is the third update target. Specifically, the updating unit 44 determines that the number of identified first ellipses obtained by the tertiary identifying process corresponding to the identified second ellipse indicated by the second ellipse graphic information 500 to be tertiary updated is P ( If ≧ 1), the IDs 501 of the P identified first ellipse graphic information 500 are included in the merge partner ID 505 and the value of the merge number 506 is increased by “P”. As a result, the second ellipse graphic information 500 that is the third update target that has been subjected to the secondary update process is subjected to the tertiary update process.

  Next, the operation of the updating unit 44 when the state information 502 of the second ellipse graphic information 500 to be thirdarily updated is a value other than “merged” will be described. In this case, the update unit 44 changes the state information 502 of the second ellipse graphic information 500 that is the third update target to “merged”. Further, the updating unit 44 does not change the ellipse specifying information 503 of the second ellipse graphic information 500 that is the third update target. Then, the update unit 44 adds the merge partner ID 505 and the merge number 506 to the second ellipse graphic information 500 that is the third update target. The update unit 44 merges when the number of identified first ellipses obtained by the tertiary identification process corresponding to the identified second ellipse indicated by the second elliptic figure information 500 to be thirdarily updated is P. The partner ID 505 includes the ID 501 of the P identified first ellipse graphic information 500 and sets the value of the merged number 506 to “P”. As a result, the second ellipse graphic information 500 that is the third update target that has been subjected to the secondary update process is subjected to the tertiary update process.

  In addition, in the tertiary update process, when the graphic information 500 of the unspecified first ellipse is included in the tracking information 50 after the 2.5th update process, the update unit 44 sets the unspecified first ellipse to the unspecified first ellipse. It is determined that the corresponding second ellipse does not exist. That is, the update unit 44 determines that the ellipse corresponding to the graphic information 500 no longer appears on the input image 140. Then, the update unit 44 sets the state information 502 of the graphic information 500 to “disappear”. Further, the update unit 44 includes the current time as the disappearance time 504 in the graphic information 500.

  Further, the update unit 44 generates graphic information 500 of an unspecified second ellipse that is not yet associated with the first ellipse among the second ellipses obtained by the fitting process in the current frame. When a person who was not shown in the input image 140 of the previous frame comes to be shown in the input image of the current frame, the second ellipse applied to the person is not displayed after the primary to tertiary specifying process is completed. It is a specific second ellipse. The update unit 44 indicates that the state information 502 indicates “appearance” as the graphic information 500 of the unspecified second ellipse that is not yet associated with the first ellipse, and the ellipse specifying information 503 indicates the ellipse specifying the unspecified second ellipse. Graphic information 500 indicating the information is generated. Then, the update unit 44 adds the generated graphic information 500 to the tracking information 50.

  As described above, the tracking processing unit 12 performs the primary processing, the secondary processing, the 2.5th processing, and the tertiary processing, so that each first ellipse obtained by the fitting processing in the previous frame is The first correspondence specifying process for specifying the second ellipse corresponding to the first ellipse and the second correspondence specifying process for specifying the second ellipse corresponding to the lost ellipse obtained by the fitting process in the current frame are performed. Is called. Then, each time the fitting processing result 170 is input, the tracking processing unit 12 performs the primary processing, the secondary processing, the 2.5th processing, and the tertiary processing, so that the tracking information 50 is sequentially updated and input. The position of the ellipse on the image 140 is tracked. The center coordinates included in the ellipse specifying information 503 of the graphic information 500 included in the tracking information 50 indicate the current position of the ellipse corresponding to the graphic information 500 on the input image 140. It can be said that the position of the person on the input image 140 (captured image 110) is tracked by sequentially updating the tracking information 50. Moreover, it can be said that the position of the person in the room 300 is tracked by sequentially updating the tracking information 50. The center coordinates included in the ellipse specifying information 503 of the graphic information 500 indicate the current position on the input image 140 of the person corresponding to the graphic information 500, and further, imaging of the person corresponding to the graphic information 500 The current position on the image 110 is shown.

  Further, by referring to the state information 502 included in the graphic information 500, it is possible to grasp the state of the person corresponding to the graphic information 500 in the room 300. For example, when the state information 502 of the graphic information 500 indicates “appearance”, it indicates that a person corresponding to the graphic information 500 has entered the room 300. Further, when the state information 502 of the graphic information 500 included in the tracking information 50 indicates “exist”, it indicates that a person corresponding to the graphic information 500 continues to exist in the room 300. Further, when the state information 502 of the graphic information 500 included in the tracking information 50 indicates “disappearance”, it indicates that the person corresponding to the graphic information 500 has left the room 300. In addition, when the state information 502 of the graphic information 500 included in the tracking information 50 indicates “merged” or “merge”, the person corresponding to the graphic information 500 exists in the room 300 near another person. It shows that.

<Details of the counting unit>
When the tracking processing in the tracking processing unit 12 is completed, the counting unit 13 obtains the number of people that appear in the input image 140 of the current frame based on the result of the tracking processing. Specifically, the count unit 13 obtains the number of people who appear in the input image 140 of the current frame based on the tracking information 50 updated in the tracking process. Hereinafter, a method for obtaining the number of people shown in the input image 140 based on the tracking information 50 will be described.

  In this example, when the count unit 13 obtains the number of persons shown in the input image 140, among the tracking information 50, the graphic information 500 in which the state information 502 indicates “appearance” and the state information 502 indicates “existence”. Graphic information 500 to be shown and graphic information 500 whose status information 502 indicates “merged” are used.

  Here, K1 is the number of graphic information 500 whose status information 502 indicates “appearance” included in the tracking information 50, and K2 is the number of graphic information 500 whose status information 502 is “existence” included in the tracking information 50. The number of graphic information 500 included in the tracking information 50 and indicating that the status information 502 is “merged” is represented by K3. Also, numbers 1 to K3 are assigned to the graphic information 500 whose status information 502 indicates “merged” included in the tracking information 50. The value of the merge number 506 included in the graphic information 500 of the kth (1 ≦ k ≦ K3) is Jk.

  The counting unit 13 obtains the total number Ktotal of people appearing in the input image 140 using the following formula (1).

  The graphic information 500 whose status information 502 indicates “appearance” corresponds to one person. Further, the graphic information 500 whose status information 502 indicates “exist” corresponds to one person. The graphic information 500 in which the status information 502 indicates “merged” corresponds to a number of people that is one larger than the value of the merged number 506 included in the graphic information 500. Therefore, the total number Ktotal of people appearing in the input image 140 of the current frame can be expressed as in Expression (1).

  Note that the count unit 13 may obtain the number of people shown in the input image 140 by a method different from the above. For example, the count unit 13 may use the number of graphic information 500 included in the tracking information 50 indicating that the status information 502 is other than “disappeared” as the total number of people in the input image 140.

  In this example, since it is unlikely that a person who first appears in a certain input image 140 will not appear in the next generated input image 140, the counting unit 13 determines the number of persons appearing in the input image 140. When obtaining, the graphic information 500 indicating that the status information 502 indicates “appearance” may not be used. In this case, the count unit 13 obtains the total number Ktotal of people appearing in the input image 140 of the current frame using, for example, the following equation (2).

  Further, since the center coordinates included in the ellipse specifying information 503 of the graphic information 500 indicate the position of the person corresponding to the graphic information 500, the counting unit 13 obtains the number of people for each area in the room 300. Can do. For example, when the room 300 is divided into four areas, the count unit 13 can specify how many people are in each of the four areas.

  FIG. 43 is a diagram illustrating an example of a state in which the first ellipse obtained by the fitting process performed on the input image 140 of the previous frame is superimposed on the captured image 110x of the previous frame. FIG. 44 is a diagram illustrating an example of a state in which the second ellipse obtained by the fitting process performed on the input image 140 of the current frame is superimposed on the captured image 110y of the current frame.

  FIG. 43 shows a first ellipse 250x12 that is commonly applied to the persons 200z1 and 200z2, and a first ellipse 250x3 that is applied to the person 200z3. FIG. 43 also shows an erasure ellipse 250x4 indicated by the graphic information 500 whose state information 502 indicates “disappearance”, which is included in the tracking information 50 updated based on the fitting process result 170 for the input image 140 of the previous frame. Has been.

  44 shows a second ellipse 250y1 fitted to the person 200z1, a second ellipse 250y2 fitted to the person 200z2, a second ellipse 250y3 fitted to the person 200z3, and a person 200z4. A fitted second ellipse 250y4 is shown.

  The first ellipse 250x12 shown in FIG. 43 is the merged main first ellipse 250x12 (also referred to as the merged first ellipse 250x12). The merged main first ellipse 250x12 is separated into two second ellipses 250y1 and 250y2 shown in FIG. Further, the disappearing ellipse 250x4 is an ellipse that has been applied to the person 200z4, and the person 200z4 that is not shown in the input image 140 is shown in the input image 140 of the current frame.

  In the example of FIGS. 43 and 44, in the primary specifying process, the two second ellipses 250y1 and 250y2 are specified as the second ellipses corresponding to the merged main first ellipse 250x12. Therefore, in the primary update process, both the graphic information 500 of the merged subject first ellipse 250x12 and the state information 502 of the graphic information 500 of the merged first ellipse 250x12 are set to “exist”.

  In the secondary specifying process, the second ellipse corresponding to the first ellipse 250x3 and the disappearing ellipse 250x4 as the secondary processing target is specified from the second ellipses 250y3 and 250y4 as the secondary processing target. Here, the second ellipse 250y3 is specified as the second ellipse corresponding to the first ellipse 250x3, and the second ellipse 250y4 is specified as the second ellipse corresponding to the disappearing ellipse 250x4. In the secondary update process, the state information 502 of the graphic information 500 of the disappearing ellipse 250x4 is changed from “disappearing” to “existing”. In the example of FIGS. 43 and 44, the first ellipse to be subjected to the tertiary processing does not exist.

  FIG. 45 is a diagram illustrating another example of a state in which the first ellipse obtained by the fitting process performed on the input image 140 of the previous frame is overlaid on the captured image 110x of the previous frame. FIG. 46 is a diagram illustrating another example of a state in which the second ellipse obtained by the fitting process performed on the input image 140 of the current frame is overlaid on the captured image 110y of the current frame.

  FIG. 45 shows a first ellipse 250x1 fitted to the person 200z1, a first ellipse 250x2 fitted to the person 200z2, a first ellipse 250x3 fitted to the person 200z3, and a person 200z4. A first ellipse 250x4 fitted to it is shown.

  In FIG. 46, a second ellipse 250y12 that is commonly applied to two persons 200z1 and z2, a second ellipse 250y3 that is applied to person 200z3, and a second ellipse 250y3 that is applied to person 200z5. An ellipse 250y5 is shown. FIG. 46 also shows an erasure ellipse 250y4 indicated by the graphic information 500 in which the state information 502 indicates “disappearance”, which is included in the tracking information 50 updated based on the fitting process result 170 for the input image 140 of the current frame. Has been.

  45 and 46, since the first ellipse obtained by the fitting process in the previous frame does not include the merged subject first ellipse, the primary process is not performed.

  In the secondary specifying process, the second ellipse corresponding to the first ellipse 250x1 to 250x4 that is the secondary processing target is specified from the second ellipses 250y12, 250y3, and 250y5 that are the secondary processing target. In this example, the second ellipse 250y3 is specified as the second ellipse corresponding to the first ellipse 250x3. Further, the second ellipse 250y12 is specified as one of the first ellipses 250x1 and 250x2, for example, the second ellipse corresponding to the first ellipse 250x1. In the secondary update process, the graphic information 500 of the first ellipse 250x1 is updated, and the graphic information 500 becomes information regarding the second ellipse y12. Since there is no second ellipse close to the first ellipse 250x4, the second ellipse corresponding to the first ellipse 250x4 is not specified in the secondary specifying process. Further, the other of the first ellipses 250x1 and 250x2, that is, the second ellipse corresponding to the first ellipse 250x2 is not specified.

  In the tertiary specifying process, the second ellipse corresponding to the first ellipse 250x2 and 250x4 to be subjected to the tertiary process is specified from the specified second ellipses 250y12 and 250y3. In this example, the identified second ellipse 250y12 is identified as the second ellipse corresponding to the first ellipse 250x2. Since there is no specified second ellipse close to the first ellipse 250x4, the second ellipse corresponding to the first ellipse 250x4 is not specified in the tertiary specifying process.

  In the tertiary update process, the state information 502 of the graphic information 500 of the identified first ellipse 250x1 newly obtained in the tertiary identification process is set to “merged”. Further, the state information 502 of the graphic information 500 of the second ellipse y12 corresponding to the first ellipse 250x1 (the graphic information 500 of the first ellipse 250x1 updated by the secondary update process) is set to “merged”.

  In the tertiary update process, the state information 502 of the graphic information 500 of the unspecified first ellipse 250x4 included in the tracking information 50 after the tertiary update process is set to “disappear”. In the tertiary update process, the graphic information 500 of the unspecified second ellipse 250y5 that is not yet associated with the first ellipse among the second ellipses obtained by the fitting process in the current frame is generated and tracked. Added to information 50. The state information 502 of the graphic information 500 is set to “appearance”.

  In the above example, when the ellipse specifying information 503 of the figure information 500 of the merged subject ellipse is updated in the primary update process, the figure information 500 of the merged ellipse to which the merged subject ellipse is merged accordingly. The ellipse specifying information 503 of the figure information 500 of the merged ellipse to which the merged main ellipse is merged may be performed at the end, that is, in the tertiary update process. Similarly, in the secondary update process, when the ellipse specifying information 503 of the merged subject ellipse graphic information 500 is updated, the ellipse specifying information 503 of the merged ellipse graphic information 500 to which the merged subject ellipse is merged is updated to the last. That is, it may be performed by a tertiary update process.

  In the above example, the ellipse specifying information 503 of the merged ellipse graphic information 500 to which the merged main ellipse is merged is updated according to the update of the ellipse specifying information 503 of the merged main ellipse graphic information 500. The ellipse specifying information 503 of the figure information 500 of the merged ellipse need not be updated. In this case, when it is necessary to use the ellipse specifying information 503 of the figure information 500 of the merged ellipse, instead of the ellipse specifying information 503 that is not updated, the figure of the integrated main ellipse that integrates the merged ellipse The ellipse specifying information 503 of the information 500 is used.

  In the above example, an ellipse is applied to the person shown in the input image 140, but other figures may be applied. For example, a rectangle may be applied to a person shown in the input image 140. When tracking the position of the object on the input image 140, it is preferable to determine a figure to be applied to the object according to the shape of the object shown in the input image 140.

  In addition, the order of execution of the primary process, the secondary process, the 2.5th process, and the tertiary process can be switched. For example, the 2.5th order process may be executed before the primary process. Further, the secondary process may be executed before the primary process.

<Usage example of image processing system>
The image processing system 1 as described above can be used for various purposes. For example, the image processing system 1 can be used for air conditioning control of the room 300. For example, when the counting unit 13 calculates the total number of people in the room 300, when the total number of people calculated by the counting unit 13 becomes zero, each air conditioner in the room 300 can be stopped. Moreover, when the total number which the count part 13 calculates | requires changes from zero to 1 or more, each air conditioner of the room | chamber interior 300 can be operated. Further, when the counting unit 13 calculates the number of people in each area of the room 300, the temperature of the area with a large number of people can be lowered, or the temperature of the area with a small number of people can be raised.

  The image processing system 1 can be used for lighting control of the room 300. For example, when the counting unit 13 determines the total number of people in the room 300, each lighting device in the room 300 can be turned off when the total number of people calculated by the counting unit 13 becomes zero. Moreover, when the total number which the count part 13 calculates | requires changes from zero to 1 or more, each lighting fixture in the room 300 can be lighted. Further, when the counting unit 13 calculates the number of people in each area of the room 300, it is possible to turn off the lighting fixtures only in the area where the number of people becomes zero.

  The image processing system 1 can be used for room entry / exit management. For example, when the counting unit 13 determines the total number of people in the room 300, when the total number of people calculated by the counting unit 13 becomes zero, it is determined that all the people have left the room 300, and the security for monitoring the room 300 is detected. The system can be operated.

  Further, the image processing system 1 can be used to control a cleaning robot that automatically cleans the room 300. For example, the tracking information 50 updated at any time is monitored, and the area in which the person has existed for many hours in the room 300 is specified from the monitoring result. Then, the cleaning robot can be controlled so as to cleanly clean the area.

  The image processing system 1 can be used for marketing. For example, when the camera 2 is installed in an art museum and the image processing apparatus 3 tracks the position of a person in the art museum, by monitoring the tracking information 50, an exhibit in which people often gather, that is, a popular exhibit. Can be specified.

  The image processing system 1 can be used for parking management. For example, when the camera 2 is installed in a parking lot and the image processing apparatus 3 tracks the position of the car in the parking lot, the count unit 13 can determine the number of cars in the parking lot. Thereby, it can be determined automatically whether there is a vacancy in a parking lot.

  The image processing system 1 can be used in aquaculture. For example, the state in the cage in the farm can be photographed with the camera 2, and the image processing device 3 can obtain the number of farmed fish in the cage based on the photographed image in the camera 2.

  Further, the image processing system 1 can be used in a ranch. For example, the state of the ranch can be photographed by the camera 2, and the image processing device 3 can obtain the number of horses or sheep on the ranch based on the image captured by the camera 2.

  As described above, the image processing apparatus 3 applies the figure to the person shown in the input image 140 and tracks the position of the figure on the input image 140, thereby determining the position of the person on the input image 140. Tracking. Therefore, the processing load on the image processing apparatus 3 can be reduced.

<Various modifications>
Various modifications will be described below.

<First Modification>
Even if the person moves the room 300 by the same distance, the moving distance of the person becomes smaller on the input image 140 (captured image 110) as the distance between the camera 2 and the person increases. Therefore, the movement distance of the person on the input image 140 becomes smaller as the distance from the position corresponding to the position of the camera 2 in the input image 140, that is, from the center 141 of the input image 140 is increased. That is, the movement distance of the person on the input image 140 increases at a position close to the center 141, but decreases at a position far from the center 141. Therefore, at a position close to the center of the input image 140, even if the distance between the person shown in the input image 140 of the previous frame and the person shown in the input image 140 of the current frame is large to some extent, Although it can be said that they are the same person, at a position far from the center of the input image 140, the person cannot be said to be the same person unless the distance is somewhat small.

  Therefore, in the present modification, in the secondary specifying process, the selected first ellipse and the nearest second ellipse are separated as the position of the selected first ellipse (for example, the center of the selected first ellipse) is further away from the center 141 of the input image 140. The fifth threshold value compared with the distance between is reduced. Thereby, the 2nd ellipse applied to the same person as the person to whom the selection 1st ellipse was applied can be specified appropriately.

  In the present modification, the fifth threshold value changes in a plurality of stages. For example, the fifth threshold value is converted into four levels of values A1 to A4. Here, A1 <A2 <A3 <A4. As shown in FIG. 47, when the position of the selected first ellipse exists in the area AR1 whose distance from the center 141 is up to d1, the fifth threshold value is set to A4. When the position of the selected first ellipse is in the area AR2 from the center 141 to the distance d1 to d2 (> d1), the fifth threshold value is set to A3. When the position of the selected first ellipse is within the area AR3 from the distance d2 to d3 (> d2) from the center 141, the fifth threshold value is set to A2. Then, when the position of the selected first ellipse exists in the area AR4 whose distance from the center 141 is larger than d3, the fifth threshold value is set to A1. The mode of change of the fifth threshold value is not limited to this. For example, the fifth threshold value may change in two stages.

  If the camera 2 is a camera with a horizontal angle of view of 180 degrees, the position 145 corresponding to the position of the camera 2 in the input image 140 is as shown in FIG. The position 145 exists in the center of the input image 140 in the longitudinal direction of one side 146 extending in the horizontal direction. In such a case, the areas AR1 to AR5 are as shown in FIG. The position 145 is also a corresponding position corresponding to a position (maximum bird's-eye angle position) in the captured image 110 where the overhead angle of the person shown in the captured image 110 is the maximum on the input image 140.

<Second Modification>
As described above, the moving distance of the person on the input image 140 decreases as the distance from the center 141 of the input image 140 increases. Therefore, as shown in FIG. 42 described above, when one second ellipse 250y is applied to a plurality of people 200 to which a plurality of first ellipses 250x1 to 250x3 is applied, The positions of the plurality of first ellipses 250x1 to 250x3 are closer to the position of the second ellipse 250y on the input image 140 as the distance from the center 141 increases.

  Therefore, in the present modification, in the tertiary specifying process, the position of the first ellipse to be processed (for example, the center of the first ellipse) is specified closer to the first ellipse as the distance from the center 141 of the input image 140 increases. The sixth threshold value used in determining whether or not the second ellipse is present is reduced. Thereby, it can be determined more appropriately whether the 1st ellipse of the tertiary processing object is merged with the specified 2nd ellipse.

  In the present modification, the sixth threshold value is converted into, for example, four stages of values B1 to B4, similarly to the fifth threshold value in the first modification example. Here, B1 <B2 <B3 <B4. As shown in FIGS. 47 and 48 described above, when the position of the first ellipse to be subjected to the tertiary processing is present in the area AR1, the sixth threshold value is set to B4. When the position of the first ellipse to be subjected to the tertiary processing exists in the area AR2, the sixth threshold value is set to B3. When the position of the first ellipse to be subjected to the tertiary processing is present in the area AR3, the sixth threshold value is set to B2. When the position of the first ellipse to be subjected to the tertiary processing exists in the area AR4, the sixth threshold value is set to B1. The mode of change of the sixth threshold value is not limited to this.

<Third Modification>
In the captured image 110, as the distance from the maximum bird's-eye angle position increases, the person appears to extend along the direction from the maximum bird's-eye angle position toward the outside. Therefore, in the fitting process, when a plurality of ellipses are applied to one person shown in the input image 140, the distance between the ellipses increases as the distance from the position corresponding to the maximum overhead angle position in the input image 140 increases. The distance tends to increase. In the above example, the position corresponding to the maximum overhead angle position in the input image 140 is the center 141 of the input image 140. Therefore, in the fitting process, when a plurality of ellipses are fitted to one person shown in the input image 140, the distance between the plurality of ellipses tends to increase as the distance from the center 141 of the input image 140 increases. is there.

  Therefore, in this modification, in the ellipse integration process, the position of the integrated ellipse obtained by integrating the first and second target ellipses (for example, the position of the center of the integrated ellipse) is farther from the center 141 of the input image 140. The first threshold value used when determining whether or not the first and second target ellipses satisfy the first condition is increased. In this case, the integrated ellipse is obtained before step s346 in the flowchart shown in FIG. Thereby, it can be judged more appropriately whether the 1st and 2nd object ellipse is applied to one person.

  In the present modification, the first threshold value is converted into, for example, four levels of values C1 to C4. Here, C1 <C2 <C3 <C4. As shown in FIGS. 47 and 48 described above, when the position of the integrated ellipse exists in the area AR1, the first threshold value is set to C1. When the position of the integrated ellipse exists in the area AR2, the first threshold value is set to C2. When the position of the integrated ellipse exists in the area AR3, the first threshold value is set to C3. When the position of the integrated ellipse exists in the area AR4, the first threshold value is set to C4. The mode of change of the first threshold value is not limited to this.

  Note that the size of the integrated ellipse increases as the distance from the center of the input image 140 increases. Therefore, in the ellipse integration process, when determining whether or not the third condition is satisfied, the second threshold value compared with the size of the integrated ellipse may be changed in the same manner as the first threshold value. good.

  In addition, when the distance between the camera 2 and the person is large, the person appears small in the captured image 110. When the shooting range of the camera 2 is narrow, the person is seen to extend as the distance from the maximum overhead angle position increases in the captured image 110 without being affected by the distance between the camera 2 and the person. On the other hand, when the shooting range of the camera 2 is wide, the camera 2 is influenced by the distance between the camera 2 and the person, if it is more than a certain distance from the maximum overhead angle position (position corresponding to the position of the camera 2). The farther away from the maximum bird's-eye angle position, the more the person will appear to be captured.

  In such a case, when the position of the integrated ellipse is within a predetermined range from the center 141 of the input image 140, the first threshold value is increased as the position of the integrated ellipse is further away from the center 141. If the position of the integrated ellipse is outside the predetermined range, the first threshold value is decreased as the position of the integrated ellipse is farther from the center 141. For example, as shown in FIG. 49, when the position of the integrated ellipse is within a predetermined range 149 from the center 141 of the input image 140 to the distance d14, the position of the integrated ellipse increases as the distance from the center 141 increases. 1 Increase the threshold. For example, when the position of the integrated ellipse exists in the area AR11 whose distance from the center 141 is d11, the first threshold value is set to C1. When the position of the integrated ellipse exists in the area AR12 whose distance from the center 141 is from d11 to d12 (> d11), the first threshold value is set to C2. When the position of the integrated ellipse exists in the area AR13 whose distance from the center 141 is from d12 to d13 (> d12), the first threshold value is set to C3. When the position of the integrated ellipse exists in the area AR14 from the center 141 to the distance d13 to d14 (> d13), the first threshold value is set to C4. In FIG. 49, the predetermined range 149 is indicated by hatching.

  On the other hand, when the position of the integrated ellipse is outside the predetermined range 149, the first threshold value is decreased as the position of the integrated ellipse is farther from the center 141. For example, when the position of the integrated ellipse is in the area AR15 whose distance from the center 141 is from d14 to d15 (> d15), the first threshold value is set to C3. When the position of the integrated ellipse exists in the area AR16 whose distance from the center 141 is larger than d15, the first threshold value is set to C2.

  In this way, by changing the first threshold value according to the position of the integrated ellipse, if the distance from the maximum overhead angle position is more than a certain extent, the overhead angle is adversely affected by the distance between the camera 2 and the person. Even when the person is shrunk as the distance from the maximum position decreases, it can be more appropriately determined whether or not the first and second target ellipses are applied to one person.

  When the camera 2 is a camera having a horizontal angle of view of 180 degrees, the predetermined range 149 and the areas AR11 to AR16 are as shown in FIG. 50, for example.

  In addition, if the person is more distant from the maximum bird's-eye angle position due to the influence of the distance between the camera 2 and the person and, on the contrary, the more the person gets closer to the bird's-eye angle position, the second is shown. The threshold value may be changed in the same manner as the first threshold value.

<Fourth Modification>
As described above, in the captured image 110, the person appears to extend as the distance from the maximum overhead angle position increases. Therefore, when two second ellipses are respectively applied to two people who have been fitted with one first ellipse, the difference between the size of the first ellipse and the size of each second ellipse is: The distance from the center 141 of the input image 140 increases.

  Therefore, in the present modification, in the primary specifying process, the size of the nearest second ellipse is determined from the size of the merged subject first ellipse as the position of the merged subject first ellipse is further away from the center of the input image 140. The fourth threshold value to be compared with the difference value obtained by subtracting is increased. Thereby, it can determine more appropriately that the merging main 1st ellipse separated into a plurality of ellipses. For example, the fourth threshold value can be changed in the same manner as the first threshold value in the third modified example.

  In addition, as described above, when the user is away from the maximum bird's-eye angle position to some extent, the person may appear to be shrunk as the distance from the highest bird's-eye angle position is influenced by the distance between the camera 2 and the person. In such a case, as in the third modified example, if the position of the merged ellipse is within a predetermined range from the center 141 of the input image 140, the position of the merged ellipse is separated from the center 141. The fourth threshold is increased as the time increases. When the position of the merged ellipse is outside the predetermined range, the fourth threshold value is decreased as the position of the merged ellipse moves away from the center 141.

<Fifth Modification>
In the captured image 110, at a position where the person's overhead angle is large, the person may not appear to extend along a direction away from the maximum overhead angle position.

  Therefore, in this modification, in the ellipse integration process, the difference between the position of the integrated ellipse obtained by integrating the first target ellipse and the second target ellipse (for example, the center of the integrated ellipse) and the center 141 of the input image 140. Is less than the seventh threshold value, the second condition is not used. That is, when the difference between the position of the integrated ellipse obtained by integrating the first target ellipse and the second target ellipse and the center 141 of the input image 140 is equal to or smaller than the seventh threshold value, the second condition is satisfied. Regardless of whether or not to do so, when the first condition and the third condition are satisfied, it is determined that the first and second target ellipses are applied to one person. In other words, if the integrated ellipse obtained by integrating the first target ellipse and the second target ellipse is located within a predetermined range from the center 141 of the input image 140, whether or not the second condition is satisfied. Regardless, when the first condition and the third condition are satisfied, it is determined that the first and second target ellipses are applied to one person. On the other hand, when the difference between the position of the integrated ellipse obtained by integrating the first target ellipse and the second target ellipse and the center 141 of the input image 140 is equal to or greater than the seventh threshold, the same as above. When the first condition, the second condition, and the third condition are satisfied, it is determined that the first and second target ellipses are applied to one person. Thereby, it can be judged more appropriately whether the 1st and 2nd object ellipse is applied to one person.

  Note that the second condition may not be used when the difference between the position of the integrated ellipse and the center 141 of the input image 140 is less than the seventh threshold value.

  As described above, the image processing system 1 has been described in detail. However, the above description is illustrative in all aspects, and the present invention is not limited thereto. The various modifications described above can be applied in combination as long as they do not contradict each other. And it is understood that the countless modification which is not illustrated can be assumed without deviating from the scope of the present invention.

DESCRIPTION OF SYMBOLS 1 Image processing system 2 Camera 3 Image processing apparatus 6 Control program 10 Image processing part 11 Fitting process part 12 Tracking process part 13 Count part

Claims (29)

A first processing unit that performs a fitting process for fitting a figure to each of the plurality of input images in which objects at different times are captured,
A second processing unit that performs a tracking process for tracking the position of the object in the input image by tracking the position of the graphic on the input image based on the result of the fitting process ;
The first processing unit includes:
A third processing unit for applying a figure to the object shown in the input image;
In the third processing unit, when a plurality of figures are applied to one object, one new figure obtained by integrating the plurality of figures is used instead of the plurality of figures. A fourth processing unit to be adopted
An image processing apparatus. The image processing apparatus according to claim 1,
An image processing apparatus, further comprising: a fifth processing unit that obtains the number of objects in the input image based on a result of the tracking process. An image processing apparatus according to any one of claims 1 and 2,
The second processing unit generates graphic information regarding the graphic,
The image processing apparatus, wherein the second processing unit performs an update process for updating the graphic information based on a result of the fitting process. The image processing apparatus according to claim 3,
The graphic information is an image processing apparatus including state information indicating a state of the graphic and graphic specifying information for specifying the graphic. An image processing apparatus according to any one of claims 3 and 4,
The second processing unit, for each first graphic obtained in the fitting process for the first input image in which the object is captured at the first time, at the second time after the first time. Performing a correspondence specifying process for specifying a corresponding second graphic corresponding to the first graphic obtained in the fitting process for the second input image in which the object is captured;
The image processing apparatus, wherein the second processing unit performs the update process based on a result of the correspondence specifying process. The image processing apparatus according to claim 5,
The second processing unit includes:
When specifying the corresponding second graphic corresponding to the first graphic,
Among the second figures obtained in the fitting process for the second input image, the second figure is closest to the first figure, and the first distance from the first figure is the first threshold. An image processing apparatus that identifies a neighboring second graphic that is a second graphic less than or less than a value as the corresponding second graphic. The image processing apparatus according to claim 5,
In the correspondence specifying process, the second processing unit includes:
Based on the result of the fitting process for the first and second input images,
One second figure is applied in the fitting process for the second input image to a plurality of objects to which the plurality of first figures obtained in the fitting process for the first input image are applied. An image processing apparatus for determining whether or not The image processing apparatus according to claim 7,
The second processing unit includes:
When specifying the corresponding second graphic corresponding to the first graphic,
Among the second figures obtained in the fitting process for the second input image, the second figure is closest to the first figure, and the first distance from the first figure is the first threshold. A neighboring second graphic that is a second graphic less than or less than the value is identified as the corresponding second graphic;
When there is an unspecified first graphic that is the first graphic for which the neighboring second graphic has not been specified as the corresponding second graphic, the second processing unit determines whether the second specific graphic with the unspecified first graphic is When there is the corresponding second graphic whose two distances are less than or less than the second threshold value, not only the object to which the first graphic corresponding to the corresponding second graphic is applied, but also the unspecified first graphic. An image processing apparatus that determines that the corresponding second graphic is also applied to the object to which one graphic is applied. The image processing apparatus according to claim 7,
The second processing unit includes:
When specifying the corresponding second graphic corresponding to the first graphic,
Among the second figures obtained in the fitting process for the second input image, the second figure is closest to the first figure, and the first distance from the first figure is the first threshold. A neighboring second graphic that is a second graphic less than or less than the value is identified as the corresponding second graphic;
In the case where there is an unspecified first graphic that is the first graphic for which the neighboring second graphic is not specified as the corresponding second graphic, the second processing unit includes at least one of the unspecified first graphic. When the corresponding second graphic including the area of the part exists, not only the object to which the first graphic corresponding to the corresponding second graphic is applied, but also the unspecified first graphic is applied. An image processing apparatus that determines that the corresponding second graphic is also applied to an object. The image processing apparatus according to claim 9,
When there is an unspecified first graphic that is the first graphic for which the neighboring second graphic has not been specified as the corresponding second graphic, the second processing unit determines whether the second specific graphic with the unspecified first graphic is When two corresponding distances are less than or less than a second threshold and there is the corresponding second graphic including at least a part of the unspecified first graphic, the second corresponding to the corresponding second graphic An image processing apparatus that determines that the corresponding second graphic is applied not only to the object to which one graphic is applied, but also to the object to which the unspecified first graphic is applied. An image processing apparatus according to any one of claims 6, 8, 9, and 10,
The input image is an image obtained by image processing of a captured image captured by a camera that captures the object,
The second processing unit identifies the corresponding second graphic corresponding to the first graphic as the position of the first graphic is further away from the corresponding position corresponding to the position of the camera in the first input image. An image processing apparatus that sets the first threshold value used at the time to be small. An image processing apparatus according to any one of claims 8 and 10,
The second processing unit sets the second threshold value to be smaller as the position of the unspecified first graphic moves away from the corresponding position corresponding to the position of the camera on the first input image. Processing equipment. An image processing apparatus according to any one of claims 5 to 12,
In the correspondence specifying process, the second processing unit includes:
Based on the result of the fitting process for the first and second input images,
A plurality of second figures are applied in the fitting process for the second input image to a plurality of objects to which one first figure obtained in the fitting process for the first input image is applied. An image processing apparatus for determining whether or not the The image processing apparatus according to claim 13,
The second processing unit determines the size of the second graphic obtained in the fitting process for the second input image that is closest in position to the one first graphic from the size of the one first graphic. The difference value obtained by subtracting is greater than or equal to the third threshold value, and is obtained in the fitting process for the second input image in the one first graphic. When at least a part of each of the plurality of second figures is included, the plurality of second figures are applied to a plurality of objects corresponding to the one first figure. An image processing apparatus for determining. The image processing apparatus according to claim 14,
The input image is an image obtained by performing image processing on a photographed image photographed by a camera that photographs the object in an overhead view,
The second processing unit moves the position of the one first figure away from a corresponding position corresponding to a position where a shooting depression angle of a person appearing in the captured image is maximum in the captured image on the first input image. The image processing apparatus sets the third threshold value to be larger. The image processing apparatus according to claim 15, wherein
The second processing unit includes:
When the position of the first graphic is within a first predetermined range from the corresponding position, the third threshold value is set to be larger as the position of the first graphic is farther from the corresponding position. And
Image processing in which, when the position of the first graphic is outside the first predetermined range, the third threshold value is set smaller as the position of the first graphic is farther from the corresponding position. apparatus. An image processing apparatus according to any one of claims 5 to 16, comprising:
The second processing unit is obtained in the fitting process for the second input image corresponding to the third graphic that is the graphic applied to the object that is no longer shown in the input image in the correspondence specifying process. The second graphic is identified, and based on the identification result, the graphic information about the third graphic is updated,
The second processing unit erases the graphic information about the third graphic applied to the object that is no longer shown in the input image after a predetermined time has elapsed since the target no longer appears in the input image. An image processing apparatus. An image processing apparatus according to any one of claims 1 to 17 ,
The fourth processing unit includes:
The third distance between the two graphics obtained by the third processing unit is less than or less than the fourth threshold value, and the size of one new graphic obtained by integrating the two graphics is increased. If below or below the fifth threshold,
An image processing apparatus that determines that the two graphics are applied to one object and employs the new graphics instead of the two graphics. The image processing apparatus according to claim 18 , wherein
The input image is an image obtained by performing image processing on a photographed image photographed by a camera that photographs the object in an overhead view,
The fourth processing unit includes:
Two first figures obtained by the third processing unit extend from corresponding positions corresponding to positions where a shooting depression angle of a person appearing in the captured image is maximum in the captured image on the input image. One new figure that is located on a straight line and the third distance between the two figures is less than or less than the fourth threshold and that is obtained by integrating the two figures Is less than or less than the fifth threshold,
An image processing apparatus that determines that the two graphics are applied to one object and employs the new graphics instead of the two graphics. An image processing apparatus according to any one of claims 18 and 19 ,
The figure is an ellipse,
The fourth processing unit includes a second straight line connecting the center of one ellipse of the two ellipses obtained by the third processing unit to the center of the other ellipse of the two ellipses. An image processing apparatus, wherein a distance between a first intersection that intersects the ellipse and a second intersection that intersects the other ellipse is the third distance. The image processing apparatus according to claim 18 , wherein
The input image is an image obtained by performing image processing on a photographed image photographed by a camera that photographs the object in an overhead view,
The fourth processing unit sets the fourth threshold value to be compared with the third distance between two figures, and the position of a new figure obtained by integrating the two figures in the captured image. An image processing apparatus that is set to increase as the distance from the corresponding position in the input image corresponding to the position where the shooting depression angle is maximized. The image processing apparatus according to claim 21 ,
The fourth processing unit includes:
When the position of a new figure obtained by integrating two figures is within a second predetermined range from the corresponding position, the fourth distance is compared with the third distance between the two figures. Set the threshold value so that the position of the new figure is farther from the corresponding position,
When the position of a new figure obtained by integrating two figures is outside the second predetermined range, the fourth threshold value to be compared with the third distance between the two figures is set. An image processing apparatus that sets the position of the new figure so small that the position of the new figure is far from the corresponding position. The image processing apparatus according to claim 18 , wherein
The input image is an image obtained by performing image processing on a photographed image photographed by a camera that photographs the object in an overhead view,
The fourth processing unit sets the fifth threshold value to be compared with the size of a new figure obtained by integrating two figures, the position of the new figure being the maximum shooting depression angle in the photographed image. The image processing apparatus is set so as to increase as the distance from the corresponding position in the input image corresponding to the position becomes. The image processing apparatus according to claim 23 , wherein
The fourth processing unit includes:
When the position of a new figure obtained by integrating two figures is within a second predetermined range from the corresponding position, the fifth threshold value to be compared with the size of the new figure is set to Set the position of the new figure as far away from the corresponding position,
When the position of a new figure obtained by integrating two figures is outside the second predetermined range, the fifth threshold value to be compared with the size of the new figure is set as the new figure. An image processing apparatus that sets the position to be smaller as the position of the distance from the corresponding position increases. The image processing apparatus according to claim 19 ,
When the difference between the position of one new figure obtained by integrating two figures and the corresponding position is less than or less than a sixth threshold value, the fourth processing unit Regardless of whether or not it is located on the first straight line, the third distance between the two figures is less than or less than the fourth threshold value, and the size of the new figure is the fifth. An image processing apparatus that determines that the two graphics are applied to one of the objects when the threshold is less than or less than a threshold, and adopts the new graphics instead of the two graphics. An image processing apparatus according to any one of claims 1 to 25 , wherein:
An image processing apparatus, further comprising: a sixth processing unit that performs image processing on a captured image captured by a camera that captures the object and generates the input image. An image processing device according to claim 26 ;
An image processing system, wherein the sixth processing unit of the image processing apparatus includes a camera that captures the captured image that performs the image processing. A first step of performing a fitting process for fitting a figure to each of the plurality of input images in which the objects at different times are captured,
A second step of performing a tracking process for tracking the position of the object in the input image by tracking the position of the graphic in the input image based on the result of the fitting process ,
In the first step, when a plurality of figures are applied to one object, one new figure obtained by integrating the plurality of figures is used instead of the plurality of figures. A method of operating the image processing apparatus. A control program for controlling an image processing apparatus,
In the image processing apparatus,
A first step of performing a fitting process for fitting a figure to each of the plurality of input images in which the objects at different times are captured,
Performing a tracking process for tracking the position of the object in the input image by tracking the position of the figure in the input image based on the result of the fitting process ; and
In the first step, when a plurality of figures are applied to one object, one new figure obtained by integrating the plurality of figures is used instead of the plurality of figures. control program order processing by the execution of.

Images