JP6982263B2 - Image processing system, image processing system control method, and program - Google Patents

Description

The present invention relates to an image processing system, a control method of the image processing system, and a program.

In recent years, a system for counting the number of people using images from a network camera has been proposed. For example, Patent Document 1 proposes a mechanism for calculating the number of people according to the number of faces in the photographed image by acquiring the photographed image output from the camera and performing face recognition on the photographed image. ing.

Further, Patent Document 2 proposes a mechanism for counting the number of people who have passed through a specific place by taking a picture of the specific place and performing image processing on the shot image.

Japanese Unexamined Patent Publication No. 2011-221906 Japanese Unexamined Patent Publication No. 2008-217205

By analyzing the image (video) of the network camera using the mechanism of Patent Document 1 and Patent Document 2, the number of people in the image and the number of people who have passed a predetermined position can be calculated. Can be done.

However, when the technology proposed in these patent documents is used, for example, when it is desired to count the number of people who participated in an event venue, the number of people who are not participants in the event such as staff members will be counted. ..

Further, there is also a problem that the staying time for each user cannot be calculated because the people who stayed for a long time or those who stayed only for a short time are counted in the same manner.

Further, there is a problem that the correct number of people cannot be calculated when the actual number of people is to be counted because the same user is counted many times.

Therefore, in view of these problems, it is an object of the present invention to provide a mechanism capable of calculating the number of people suitable for a purpose when calculating the number of people who have passed a predetermined position from an image.

The image processing system according to the present invention is
A counting means that counts the number of people who have passed a predetermined position based on the captured image of the person.
A person identification means for identifying a person who is the same person in the photographed image and has passed the predetermined position in a different direction within a predetermined time after passing the predetermined position.
It is characterized by having .

According to the present invention, it is possible to provide a mechanism capable of calculating the number of people suitable for a purpose when calculating the number of people who have passed a predetermined position from an image.

It is a system configuration diagram which shows an example of the system configuration of the image processing system in embodiment of this invention. It is a figure which shows an example of the hardware composition of the information processing apparatus applicable to the server 100 and the client PC 101 of the image processing system in embodiment of this invention. It is a figure which shows an example of the hardware configuration of the network camera 102 in embodiment of this invention. It is a functional block diagram which shows an example of the functional block of the image processing system in embodiment of this invention. It is a figure which shows an example of the image used for the number of people counting in embodiment of this invention. It is a figure which shows an example of the image used for the passing person count in embodiment of this invention. It is a flowchart which shows an example of the image preservation processing of the image processing system in embodiment of this invention. It is a flowchart which shows an example of the passing person count processing of the image processing system in embodiment of this invention. It is a flowchart which shows an example of the number of people count processing of the image processing system in embodiment of this invention. It is a flowchart which shows an example of the face recognition processing of the image processing system in embodiment of this invention. It is a flowchart which shows an example of the browsing process of the image processing system in embodiment of this invention. It is a figure which shows an example of the person DB of the image processing system in embodiment of this invention. It is a figure which shows an example of the browsing screen of the number of people count in embodiment of this invention. It is a figure which shows an example of the browsing screen of the passing person count in embodiment of this invention. It is a flowchart which shows an example of the passing person count processing of the image processing system which concerns on another embodiment. It is a figure which shows an example of the person position information DB. It is a figure which shows an example of the boundary crossing information DB. It is a figure which shows an example of a line counter DB. It is a flowchart which shows an example of the count processing of the number of people which appeared in the image in the image processing system which concerns on another Embodiment. It is a figure which shows an example of the browsing screen of the number of people count.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The embodiment described below is an example of a specific embodiment of the present invention, and is one of the specific embodiments of the configuration described in the claims.

FIG. 1 is a system configuration diagram showing an example of a system configuration of an image processing system according to an embodiment of the present invention.

In the image processing system (information processing system) according to the embodiment of the present invention, the server 100, the client PC 101, and the network camera 102, which is an image pickup device, are connected to each other via a wireless (or wired) LAN 120 so as to be able to communicate with each other. The system configuration shown in FIG. 1 is an example, and there are various configuration examples depending on the application and purpose.

The server 100 stores and manages image data captured by the network camera 102. Further, the server 100 has a function of analyzing the image data stored and managed by the server 100, counting the number of users included in the image data, and counting the number of people who have passed through a predetermined area.

The image management system according to the embodiment of the present invention includes a plurality of network cameras 102 [network cameras 102 (A) and (B)], and communicates using the TCP / IP protocol. A fixed IP address is assigned independently, and an external port number assigned to each network group is assigned.

Therefore, the server 100 can identify from which network camera 102 the received image data is transmitted, and can determine which network camera 102 the received image data is transmitted from. It is possible to associate and manage the memory.

The network camera 102 is provided for each meeting space or entertainment venue (hereinafter, also referred to as “predetermined area”) on the floor of a company building, for example, the network camera 102 (A) is a still image or a still image of the entire predetermined area. A moving image is taken, and the network camera 102 (B) shoots a still image or a moving image for identifying the person photographed there.

Examples of predetermined regions are region 106, region A103, region B104, and the like, which indicate the entire region. The position 105 provided between the areas A103 and the area B104 is a reference position when counting the number of users (USERs) who have passed the position 105. That is, this position is a boundary that separates the area A103 and the area B104. This position may be set in a linear shape, may be set in a circular shape, or may be set in a rectangular shape.

The client PC 101 is used for the user to browse the image data (video of the network camera 102) managed by the server 100. The user can view the image data by logging in to the monitoring system via the client PC 101.

Next, an example of the hardware configuration of the information processing apparatus applicable to the server 100 and the client PC 101 shown in FIG. 1 will be described with reference to FIG.

FIG. 2 is a diagram showing an example of a hardware configuration of an information processing device applicable to the server 100 and the client PC 101 of the image processing system according to the embodiment of the present invention.

In FIG. 2, 201 is a CPU that comprehensively controls each device and controller connected to the system bus 204. Further, in the ROM 202 or the external memory 211, a BIOS (Basic Input / Output System) which is a control program of the CPU 201, an operating system program (hereinafter, OS), and various functions necessary for realizing a function executed by the PC, which will be described later, are used. The program etc. are stored.

Reference numeral 203 denotes a RAM, which functions as a main memory, a work area, and the like of the CPU 201. The CPU 201 realizes various operations by loading a program or the like necessary for executing a process from the ROM 202 or the external memory 211 into the RAM 203 and executing the loaded program.

Further, 205 is an input controller, which controls input from a pointing device such as a keyboard (KB) 209. Reference numeral 206 denotes a video controller, which controls the display on a display such as a CRT display (CRT) 210. Although it is described as CRT210 in FIG. 2, the display may be not only the CRT but also another display such as a liquid crystal display.

The 207 is a memory controller that can be used in an external storage device (hard disk (HD)) that stores boot programs, various applications, font data, user files, edit files, various data, etc., a flexible disk (FD), or a PCMCIA card slot. It controls access to an external memory 211 such as a CF memory connected via an adapter.

Reference numeral 208 denotes a communication I / F controller, which connects and communicates with an external device via a network, and executes communication control processing on the network. For example, communication using TCP / IP is possible.

The CPU 201 enables display on the CRT 210 by, for example, executing an outline font expansion (rasterization) process in the display information area in the RAM 203. Further, the CPU 201 enables a user instruction with a mouse cursor or the like (not shown) on the CRT 210.

Various programs described later for realizing the present invention are recorded in the external memory 211, and are executed by the CPU 201 by being loaded into the RAM 203 as needed. Further, a setting file or the like used when executing the above program is also stored in the external memory 211, and detailed description of these will be described later. This is the end of the description of FIG.

Next, an example of the hardware configuration of the network camera 102 shown in FIG. 1 will be described with reference to FIG.

FIG. 3 is a diagram showing an example of the hardware configuration of the network camera 102 according to the embodiment of the present invention.

The CPU 301 comprehensively controls each device and controller connected to the system bus 304. Further, the ROM 302 or the external memory 305 is required to realize the functions executed by the BIOS (Basic Input / Output System), the operating system program (hereinafter, OS), which is the control program of the CPU 301, and the image processing server 108. Various programs described later are stored. The RAM 303 functions as the main memory of the CPU 301, a work area, and the like.

The CPU 301 realizes various operations by loading a program or the like necessary for executing a process into the RAM 303 and executing the program.

The memory controller (MC) 306 is a CF connected to a hard disk (HD) or a PCMCIA card slot that stores a boot program, various applications, font data, user files, edit files, various data, image data, etc. via an adapter. Controls access to external memory 305 such as memory and SmartMedia®.

The camera unit 307 is connected to the image processing unit 308, and after performing photoelectric conversion of the light obtained through the lens directed to the monitored object by a light receiving cell such as a CCD or CMOS, an RGB signal is obtained. And the complementary color signal is output to the image processing unit 308.

The image processing unit 308 performs white balance adjustment, gamma processing, and sharpness processing based on the RGB signal and the color capture signal, and further performs YC signal processing to generate a brightness signal Y and a chroma signal (hereinafter, YC signal). Then, the YC signal is compressed in a predetermined compression format (for example, JPEG format, MotionPEG format, etc.), and the compressed data is temporarily stored in the external memory 305 as image data.

The communication I / F controller (communication I / FC) 309 connects and communicates with an external device via a network, executes communication control processing on the network, and is an image stored in the external memory 305. The data is transmitted to the external device by the communication I / F controller 309. This is the end of the description of FIG.

FIG. 4 is a functional configuration diagram showing an example of the functional configuration of the image processing system according to the embodiment of the present invention.

In the image processing system according to the present embodiment, the server 100 has each function of an image acquisition unit 411, a number of people counting unit 412, a person recognition unit 413, a person exclusion unit 414, and a result screen display control unit 415. The server 100 is not limited to the functional configuration shown in FIG. 4, and for example, a storage unit that stores a person determined to be a count exclusion target, and passage direction information indicating a direction in which the person has passed a predetermined position. It may have various other functional configurations such as a passage direction acquisition unit to be acquired, a stay time calculation means for calculating the time that a person is staying in a predetermined area, and the like.

The image acquisition unit 411 acquires an image taken by the network camera 102 as described with reference to FIG. 7. Images include still images and moving images. In the embodiment, the network camera 102 (A) captures a predetermined area, and the network camera 102 (B) recognizes the identified person (for example, face recognition) and uses it to identify an individual.

The number of people counting unit 412 uses an image taken by the network camera 102 (A) to count the number of people in a predetermined area as described with reference to FIG. 9, and is predetermined as described with reference to FIG. Count the number of people who have passed the position of.

The person recognition unit 413 uses the image taken by the network camera 102 (B) to authenticate the person in the image. An example of the authentication process will be described with reference to FIG.

Based on the image acquired by the image acquisition unit 411, the person exclusion unit 414 determines whether or not the person in the image is a count exclusion target to be excluded from the count of the number of people by the person count unit 412. The person exclusion unit 414 performs a process of specifying a person to be excluded from the number of people count by the number of people counting unit 412, and storing the authenticated person in the person table or the passing person table shown in FIG. The person exclusion unit 414 determines, for example, whether or not the person who has passed the predetermined position is the count exclusion target, based on the passage direction information indicating the passage direction in which the person in the image has passed the predetermined position. Further, the person exclusion unit 414 determines whether or not the person shown in the image is a count exclusion target based on the staying time of the person shown in the image in a predetermined area.

The result screen display control unit 415 performs a process of controlling the image processing result to be displayed on the user's terminal according to the user's request.

FIG. 5 is a diagram showing an example of an image used for counting the number of people in the embodiment of the present invention.

Image A501 is an image in which a predetermined area at a certain place is photographed, and image B502 is an image in which the same point and the same area as image A are photographed at different times. These images are managed as images to be compared.

In FIG. 5, five people from Mr. A to Mr. E are shown in image A. For example, in image B, which is an image after 5 minutes, even if the same five people are present, Mr. D and Mr. E disappear. , Mr. F and Mr. G are in the picture. Here, for example, Mr. A, Mr. B, and Mr. C are excluded from the number of people count because they are the users who were also shown in the image B 5 minutes ago. That is, the person exclusion unit 414 determines Mr. A, Mr. B, and Mr. C who have a staying time of 5 minutes or more in the predetermined area 501 as count exclusion targets. Then, the person counting unit 412 excludes the person determined to be the count exclusion target by the person exclusion unit 414 and counts the number of people.

It is also possible to exclude Mr. A from the number of people count because he is a staff member, for example. That is, the person exclusion unit 414 may determine that Mr. A is a count exclusion target based on the fact that he is a default person. The number counting unit 412 is not limited to the person determined to be the count exclusion target by the person exclusion unit 414, and may count the number of people by excluding the person set in advance by the administrator or the like.

In the present embodiment, the change status of the person photographed here can be displayed on the screen shown in FIG.

FIG. 6 is a diagram showing an example of an image used for counting the number of passing people in the embodiment of the present invention.

A predetermined area in a certain place is an image taken, and the time has elapsed in the order of image A601 → image B602 → image C602, respectively.

Here, the number of people who have moved from the area A to the area B (or vice versa) based on the predetermined position 604 is counted, and the same person or the default user who moves multiple times in the same direction. Can be excluded from the count. Here, the same direction is not limited to exactly the same direction on a straight line from one point to another, and means, for example, the same direction as the direction from the first region to the second region. For convenience, the case where a person moves to the area B with reference to the area A is counted as "OUT", and the case where the person moves from the area B to the area A is counted as "IN". In this case, the same direction means that both are "OUT" (or both are "IN").

Image A shows that Mr. C is moving from region A to region B (moving a predetermined position 604 from region A to region B) (OUT).

In the image B, Mr. C moves from the area B to the area A, and Mr. D moves from the area B to the area A (IN).

In the image C, Mr. C moves from the area A to the area B again, and Mr. A moves from the area A to the area B (OUT). Here, Mr. C is excluded from the number of people count because it is the second move in the same direction. That is, the person exclusion unit 414 determines that Mr. C, who passes through the predetermined position 604 a plurality of times in the same passing direction, is the count exclusion target. Then, the person counting unit 412 excludes the person determined to be the count exclusion target by the person exclusion unit 414 and counts the number of people.

Also, since it is default that Mr. A will be excluded in advance, even if it is the first move, it will be excluded. That is, the person exclusion unit 414 may determine that Mr. A is a count exclusion target based on the fact that he is a default person. The number counting unit 412 is not limited to the person determined to be the count exclusion target by the person exclusion unit 414, and may count the number of people by excluding the person set in advance by the administrator or the like.

FIG. 7 is a flowchart showing an example of image storage processing of the image processing system according to the embodiment of the present invention.

This process is a process of transmitting an image taken by the network camera 102 to the server 100 and storing the image.

In step S701, the network camera 102 photographs the monitoring area (area 106) and transmits the image captured in step S702 to the server.

In step S703, the server 100 receives an image from the network camera 102, and in step S704, the server 100 saves the image received as a recording file in the external memory 211 or the like of the server, and ends the process.

Using the image saved here, the image processing system counts the number of users or the number of passing users in the image.

FIG. 8 is a flowchart showing an example of the number of passing people counting process of the image processing system according to the embodiment of the present invention.

This process is a process of counting how many users have passed a predetermined position (for example, 604) set in a predetermined area (for example, 601), and is a process of counting an image of a predetermined time from an image stored in a server. Process as a target.

In step S801, the process is repeated for a predetermined time specified by the user or the administrator. In this embodiment, it is assumed that 5 minutes from 9:00 to 9:05 are designated. Further, in reality, a large number of images are composed to make a moving image for 5 minutes, but in this embodiment, three images will be used for convenience.

It is assumed that the image 601 is taken at 9:00, the image 602 is taken at 9:03, and the image 603 is taken at 9:05.

In step S802, the line passage detection area information is read. Specifically, it is determined whether or not there is a person who has passed the predetermined position 604.

In step S803, if the person can be identified (recognized), the process proceeds to step S804, and if the person cannot be identified (recognized), the process proceeds to step S811 to determine the loop (whether a predetermined time has elapsed).

In step S804, a person determined to have passed a predetermined position in step S802 by image analysis processing or the like acquires passage direction information indicating a direction in which the person has passed the predetermined position (passing direction acquisition means). The passing direction information is information indicating either a first direction from the area A to the area B or a second direction from the area B to the area A, and here, the movement from the area A to the area B. Let (first direction) be "OUT" and move from area B to area A (second direction) be "IN" (or vice versa).

In step S805, a process of identifying the person recognized in step S803 is performed. Although this process will be described with reference to FIG. 10, it is a process for identifying a recognized person (person identification means). By this process, unnecessary persons among the identified persons can be excluded from the number of people count.

Step S806 determines whether or not the person recognized in step S803 is the count exclusion target. If it is determined in step S806 that the target person is a count exclusion target, the process proceeds to step S808, the movement of the target person is stored as a count exclusion target (memory means) so that it is not reflected in the number of people count, and in step S809. move on. If it is determined in step S806 that the target person is not the count exclusion target, the process proceeds to step S807, the IN / OUT movement count is incremented, and the process proceeds to step S809.

In step S809, it is confirmed whether all the processes of the person recognized in step S803 have been completed. If not, the process returns to step S804 to perform the process for the next person, and the process for all the persons is completed. Goes to step S810.

In step S810, the contents processed in this flow are saved in the result table, and the processing is terminated. In addition, this process may be saved collectively in this step, or may be saved each time in each step of the flowchart.

Next, the result table (person table) saved in this flowchart will be described with reference to FIG.

FIG. 12 is a diagram showing an example of a person DB of an image processing system according to an embodiment of the present invention. The person DB 1200 is a result table in which the result of the number of people counting process described with reference to FIG. 9 is stored. In the person DB 1200, the user name of the user recognized by the image A and the image B and the type of the user are stored. Details will be described later.

The passing person DB 1210 is an example of the passing person DB, which is a result table stored in the line passing counting process of FIG.

In the passing person DB 1210, an image 1211, a time when the image was taken 1212, a passing direction 1213, a user 1214, and a counter 1215 are stored in association with each other.

Image 1211 is an image name, for example, an image file name. Images A to C correspond to images A601 to C603 of FIG. 6, respectively, and were taken at 9:00, 9:03, and 9:05, respectively.

The passing direction 1213 stores the direction detected in step S804, and the user 1214 stores the user specified in step S805.

The counter 1215 is an IN / OUT counter in step S807, and is a counter for the number of IN users and OUT users. For example, it can be seen that OUT was counted by 2 people and IN was counted by 1 person.

In addition, Mr. C in image C is excluded. This is due to the setting that "movement of the same user in the same direction within a predetermined time is excluded from the count". That is, the person exclusion unit 414 determines that the same user who moves in the same direction within a predetermined time is the count exclusion target, and the determination result is stored in the result table.

In addition, Mr. A in image C is excluded. This is due to the setting that "Mr. A is (excluded) by default" as shown in FIG. That is, the person exclusion unit 414 determines that the default user is the count exclusion target person, and the determination result is saved in the result table. In addition to this setting, the setting can be set in advance by the administrator.

FIG. 9 is a flowchart showing an example of the number of people counting process of the image processing system according to the embodiment of the present invention.

This process is a process of counting how many users are shown in a predetermined area (for example, 501), and targets an image stored in a server.

For example, five images are shown in both image A501 and image B502, but the two images can be compared to exclude the same user, exclude a predetermined user, or display only the number of new users. Can be processed. That is, not limited to the preset exclusion target person, the process of determining whether or not the person is the count exclusion target based on the image and excluding the person determined to be the count exclusion target from the number of people count. It can be carried out.

Further, in this process, since the images are compared, the number of set predetermined number of people is counted. In this embodiment, two images are the predetermined number of images A and B. ..

In step S901, the area information is read. The area information is, for example, the area 106 in FIG. 1, and corresponds to an image of the area preset and captured by the network camera 102.

In step S902, it is determined whether or not a person has been identified (recognized) in the area, and if so, the process proceeds to step S903 to perform a person identification process. The person identification process will be described with reference to FIG. 10, but the person identification process determines whether or not the recognized person is an exclusion target.

In step S904, it is determined whether or not the person recognized in step S902 is excluded, and if it is determined that the person is excluded, the process proceeds to step S906, and if it is determined that the person is not excluded. Goes to step S905.

In step S906, the person determined to be excluded is not reflected in the number of people count, and the process proceeds to step S907. In step S905, the number of persons recognized in step S902 is counted, and the process proceeds to step S907.

In step S907, it is determined whether or not the processing for all the images (here, two images) is completed. If the processing is not completed, the process returns to step S903, and if the processing is completed, the process proceeds to step S908. , Save the contents of the process in the result table and finish the process. In addition, this process may be saved collectively in this step, or may be saved each time in each step of the flowchart.

The result table saved in this flowchart will be described using the person DB 1200 of FIG. Image A is an image taken at 9:00, and image B is an image taken at 9:05.

The person DB 1200 manages the number of recognized users and whether or not the users are excluded for each image.

The user type 1201 and the user name are stored in the person DB 1200 corresponding to the image A. The person DB 1200 stores information on five users A to E recognized based on the image 501 of FIG. 5 as the user name 1202. In addition, the user type 1201 of user A, who is excluded by default, is set as "default". ..

In the person DB 1200 corresponding to the image B, information of five users A to C, F, and G is stored as the user name 1204. The user types 1203 of users F and G, who are users (new) that do not exist in the image A, are blank.

In addition to the user type 1204 of user A being set to "default", the user type 1204 of users A to C specified in image A is stored as "specified" (specified / A). There is. That is, the person exclusion unit 414 determines that the same user who is located on the same floor even after the lapse of a predetermined time is the count exclusion target, and saves the determination result in the result table. Here, it is assumed that the specified user and the default user are treated as exclusion targets.

FIG. 10 is a flowchart showing an example of face recognition processing of the image processing system according to the embodiment of the present invention.

This face recognition process corresponds to the person identification process of S805 of FIG. 8 and step S903 of FIG. In this embodiment, the person identification process by face recognition will be described, but any technique may be used as long as the person can be identified, not limited to the face recognition process.

In step S1001, the person image recognized by the image is received, and in step S1002, the characteristic points of the person are cut out and the individual is identified. The process of cutting out the characteristic points of this person and identifying the individual shall utilize the technology already proposed.

In step S1003, it is confirmed whether or not the specified person is registered in the result table (passing person DB1210 in the case of FIG. 8, person DB1200 in the case of FIG. 9), and if registered, in step S1006. If the procedure is not registered, the process proceeds to step S1004.

In step S1006, the person registered in the result table is returned as an excluded person (excluded target). In the case of the person DB 1200, the registered state is determined by whether or not the user types 1201 and 1203 are registered as exclusion targets. Specifically, in the case of image A, Mr. A is excluded because the user type is the default, and in the case of image B, the default Mr. A and the specified A to C are excluded persons (exclusion targets). It will be replied and the process will be completed.

Further, in the case of the passing person DB 1210, Mr. A, which is the default, and Mr. C (OUT) of the image C, which is the second movement (moving multiple times) in the same direction, are returned as excluded persons (excluded targets) and processed. Is finished.

In step S1004, the person determined not to be registered in step S1003 is registered in the result table (as an exclusion). For example, in the person DB 1200, Mr. A to C, who have not been registered based on the fact that they have been specified in the person DB 1200 corresponding to the image A, have already been specified in the image A in the person DB 1200 corresponding to the image B. Is registered.

Further, in the passing person DB 1210, Mr. C (OUT) of image C, which is a case where the same user is moving in the same direction and is not registered, and Mr. A (OUT) of image C, which is a default user. -Default) is registered as an exclusion target. Subsequently, in step S1005, the person registered in step S1004 is returned as a non-excluded person (non-excluded target), and the process ends.

FIG. 11 is a flowchart showing an example of browsing processing of the image processing system according to the embodiment of the present invention.

This process is a process of displaying and controlling the result of image processing on the user's terminal according to an instruction from the user.

In step S1101, the camera to be displayed is selected. For example, the serial number or name of the network camera 102 (A) is selected, and the image taken by the camera selected in step S1102 is requested. This image may be a still image format, a moving image format, or a LIVE moving image.

In step S1103, it is determined whether the requested image is the number of people count (FIG. 5) or the line passing number of people (FIG. 6). move on.

In step S1104, processing is performed so as to display the viewing screen of the number of people count shown in FIG. 13 as the number of people information. In step S1105, processing is performed so as to display the viewing screen of the passing number of people count shown in FIG. The details of the browsing screen will be described later with reference to FIGS. 13 and 14.

In step S1106, a process of synthesizing an image of an item to be displayed on the browsing screen is performed, and in step S1107, screen display control is performed so as to be displayed on the client screen, and the process is terminated.

FIG. 13 is a diagram showing an example of a viewing screen for counting the number of people according to the embodiment of the present invention.

Two examples of the number of people count viewing screen 1301 and the number of people count viewing screen 1310 will be described.

On the person count viewing screen 1301, the current image 1302, the recognized number of people (5 people) 1303, the default exclusion target (1 person) 1304, the personally identified exclusion target (3 people) 1305, and a new screen. The recognized persons and the number of people (2 persons) 1306 are displayed.

The number of people count viewing screen 1310 is another viewing screen example of 1301, and it is also possible to display the display contents including the display contents of 1301.

The count number of people excluding the exclusion target person, the default exclusion target number of people 1311, and the acquired image are displayed.

In addition, the counted person may be displayed by surrounding it with a frame such as a thick frame to improve visibility, and Mr. A, who is the default exclusion target, can be distinguished from other users by a different color or line. It may be displayed in.

In addition, the user name (for example, Mr. A or Mr. B) is synthesized and displayed for the user who can identify the individual in the person identification process, and nothing is displayed for the user who could not identify the individual. Is also valid.

FIG. 14 is a diagram showing an example of a viewing screen for counting the number of passing people according to the embodiment of the present invention.

Two examples of the passing number count viewing screen 1401 and the passing number counting viewing screen 1410 will be described.

On the passing person count viewing screen 1401, the number and direction 1402 of passing a predetermined line (position), the default exclusion target (1 person) 1403, and the excluded person and the number of people 1404 by passing twice are displayed.

The passing number count browsing screen 1410 is another browsing screen example of 1401, and it is also possible to display the display contents of 1401.

The number of passing people / direction / number of exclusions 1411 and the acquired image are displayed. In the embodiment, IN (1 person), OUT (2 people), and the number of exclusions (2 people) are used, but the number of exclusions includes both the exclusion targets of 1403 and 1404.

In the image, the person counted by passing the line may be displayed by surrounding it with a frame such as a thick frame to improve visibility, and Mr. A, who is the default exclusion target, may be displayed in a different color or line. It may be displayed so as to be distinguishable from other users. In addition, it is also possible to display the user 1412 who did not pass the line so that it can be distinguished from the person who passed the line without displaying the frame line or the like (indicated by a dotted line for convenience, but nothing needs to be displayed). Is.

In addition, the user name (for example, Mr. A or Mr. B) is synthesized and displayed for the user who can identify the individual in the person identification process, and nothing is displayed for the user who could not identify the individual. Is also valid.

Next, an image processing system according to another embodiment applicable to the case of performing the passing number counting process will be described. In the other embodiment, the person exclusion unit 414 repeatedly passes (straddles) a person who crosses the boundary line, that is, a person who wanders on the boundary line, that is, a predetermined position within a predetermined time (for example, about 5 minutes). It differs from the above embodiment in that it is determined whether or not it is a count exclusion target based on whether or not it is a person. Hereinafter, this determination is also referred to as "a wandering exclusion determination".

Hereinafter, the other embodiment will be described in detail with reference to FIG. FIG. 15 is a flowchart showing an example of the number of passing people counting process of the image processing system according to another embodiment.

This process is a passing number counting process for counting how many users have passed a predetermined boundary line (for example, position 604) set in a predetermined area (for example, the area reflected in the image 601) within a predetermined time. It is a process to exclude people who repeatedly pass on the boundary line from the number of people count. This processing is performed by the server 100 of the image processing system.

This process starts, for example, when a start instruction for starting this process is input by the user. Further, in this process, for example, when it is determined that all the persons have been analyzed in step S1504 described later, when an end instruction is input by the user, or when a preset processing time is completed, etc. finish.

First, in step S1501, the boundary line information and the wandering interval time are acquired. The boundary line information and the wandering time interval are acquired from, for example, setting information set in advance by a user or the like.

The boundary line information is the position information of the boundary line, and is indicated by, for example, pixel coordinates with respect to the upper left end of the image 601. The wandering interval time does not simply pass the boundary line in order to move the same person from the area A to the area B (or from the area B to the area A) at the timing interval when the same person passes the boundary line multiple times. It is a time considered to be wandering around on the boundary line, and is set to, for example, 5 minutes.

Subsequently, in step S1502, the image acquisition unit 411 acquires an image taken by the network camera 102. The image acquisition unit 411 acquires, for example, an image taken within a predetermined time.

Subsequently, in step S1503, the person recognition unit 413 recognizes all the persons and their positions in the image acquired in step S1503, and creates a list of all persons and their positions on the memory.

Subsequently, in step S1504, it is determined whether or not there is a person who has not been analyzed on the list created in step S1503. Here, the analysis means processing the steps after step S1505, which will be described later.

When it is determined that there is no person who has not been analyzed (step S1504; NO), that is, when all the persons have been analyzed, the process is terminated. In this case, the process may be returned to the process of step S1503 without ending the process, the image may be acquired again, and the processes after step S1504 may be repeated.

If it is determined that there is a person who has not been analyzed (step S1504; YES), the process proceeds to step S1505. That is, the processes after step S1505 are repeated until the analysis is completed for all the persons.

Subsequently, in step S1505, among the unprocessed persons on the list created in step S1503 that have not been analyzed, the person information to be analyzed next is acquired.

Subsequently, in step S1506, the person recognition unit 413 recognizes the same person as the person indicated by the person information to be analyzed next acquired in step S1505.

Subsequently, in step S1507, the person recognition unit 413 caches the position information of the person recognized as the same person in step S1506. Specifically, it is stored as the person position information DB 1600 as shown in FIG. FIG. 16 is a diagram showing an example of the person position information DB 1600.

As shown in FIG. 16, the person position information DB 1600 stores the time stamp 1601, the person ID 1602, the camera ID 1603, and the position information 1604 in association with each other.

The time stamp 1601 is a character string indicating the date and time when the image recognized by the person was taken by the network camera. In the example of FIG. 16, the date and time of the time stamp 1601 are stored so as to be arranged in chronological order from the top. For example, "2018.07.27 12:00:01:00" is stored as the time stamp 1601, and then "2018.07.27 12:00:00:00" is stored as the time stamp 1601. That is, the date and time when the time stamp 1601 is "2018.07.27 12:00:00:00" and the date and time when the time stamp 1601 is "2018.07.27 12:00:01:00" are chronologically mutually exclusive. There is a context. Hereinafter, among the dates and times in such a context, the time before the time stamp 1601 is "2018.07.27 12:00:00:00" is also simply referred to as "previous time", and the time stamp 1601 is "2018.07. The time after "27 12:00:01:00" is also simply called "after time".

Although not shown in the example of FIG. 16, in reality, data associated with the time stamp 1601 of a predetermined time including the date and time before the date and time of the time stamp 1601 shown is stored. There is.

The person ID 1602 is identification information indicating the person recognized in step S1506. The camera ID 1603 is identification information indicating the network camera 102 that captures the person recognized in step S1506. The position information 1604 is the position information of the person recognized in step S1506, and is shown in pixel coordinates with respect to, for example, the upper left end of the image 601. The position information 1604a is the position information indicated by the X coordinate, and the position information 1604b is the position information indicated by the Y coordinate.

With reference to FIG. 15 again, in the following step S1508, for the person recognized as the same person in step S1506, the movement position and the movement direction within a predetermined time such as one hour are calculated based on the person position information DB 1600.

The moving position is a position where the same person has moved within a predetermined time. The moving position is calculated as, for example, each position indicated by the same person ID 1602 and the position information 1604 associated with the time stamp 1601 within a predetermined time.

The moving direction is the direction in which the same person has moved within a predetermined time. The movement direction is calculated, for example, as a vector indicating a direction from the position in the previous time to the position in the later time based on each calculated position.

Subsequently, in step S1509, it is determined whether or not the same person has passed the boundary line based on the boundary line information acquired in step S1501 and the movement position and movement direction calculated in step S1508.

When it is determined that the same person does not cross the boundary line (step S1509; NO), the process returns to step S1504, and it is determined whether or not there is a person who has not been analyzed. If it is determined that the same person has passed the boundary line (step S1509; YES), the process proceeds to step S1510.

In step S1510, the passage information of the boundary line is cached for the person determined to have passed the boundary line in step S1509. Specifically, the passing information of the boundary line is stored as the boundary crossing information DB 1700 as shown in FIG. FIG. 17 is a diagram showing an example of the boundary crossing information DB 1700.

As shown in FIG. 17, in the boundary crossing information DB 1700, the time stamp 1701, the person ID 1702, the camera ID 1703, and the boundary crossing direction 1704 are stored in association with each other.

The time stamp 1701 is a character string indicating the time when the same person crosses the boundary line. The person ID 1702 is identification information indicating a person determined to have passed the boundary line in step S1509. The camera ID 1703 is identification information indicating a network camera 102 that captures a person determined to have passed the boundary line in step S1509.

The boundary crossing direction 1704 is information indicating a direction beyond the boundary line of the person determined to have passed the boundary line in step S1509, and is indicated by the above-mentioned "IN" or "OUT". The boundary crossing direction 1704 is calculated and stored, for example, based on the movement position and movement direction calculated in step S1508 and the boundary line information acquired in step S1501 (passing direction information acquisition means).

Subsequently, in step S1511, the boundary crossing information DB 1700 stored in step S1510 is referred to, and the same person searches for a plurality of information that has passed the boundary line within a predetermined time (for example, one hour).

Subsequently, in step S1512, it is determined whether or not the boundary line has been wandered around based on the information searched in step S1511. For example, within the wandering interval time (for example, 5 minutes) acquired in step S1501, the movement in the opposite direction (movement from "IN" to "OUT" or from "OUT" to "IN") is performed once. If there are any of the above, it is determined that the boundary line has been wandered around.

If it is determined in step S1512 that the boundary line has been wandered around (step S1512; YES), the process proceeds to step S1513. In step S1513, the person exclusion unit 414 determines that the person who has passed the boundary line is the count exclusion target, and the number counting unit 412 excludes the number of persons determined to be the count exclusion target from the count. After step S1513, the process returns to step S1504, and the determination as to whether or not there is a person who has not been analyzed is performed again.

In the present specification, excluding from the count includes not only the case where the count of the number of people is not incremented but also the case where the count once incremented is decremented.

If it is determined in step S1512 that the boundary line has not been wandered around (step S1512; NO), the process proceeds to step S1514, and the same process as the normal number counting process is performed. Specifically, first, in step S1514, it is determined whether or not the same person has moved a plurality of times in the same direction based on the information searched by Tep S1511.

If it is determined in step S1514 that the person has moved in the same direction a plurality of times (step S1514; YES), the process proceeds to step S1515, and the person exclusion unit 414 determines that the person who has passed the boundary line is the count exclusion target, and the number of people. The counting unit 412 excludes the number of persons determined to be excluded from the count from the count. After step S1514, the process returns to step S1504, and the determination as to whether or not there is a person who has not been analyzed is performed again.

If it is determined in step S1514 that the vehicle has not moved in the same direction a plurality of times (step S1514; NO), the process proceeds to step S1516, and the count of the number of people who have passed the boundary line is incremented for each IN / OUT passage direction. After step S1516, the process returns to step S1504, and the determination as to whether or not there is a person who has not been analyzed is performed again.

The result of the number count incremented in step S1516 is stored as, for example, the line counter DB1800 as shown in FIG. FIG. 18 is a diagram showing an example of the line counter DB1800.

As shown in FIG. 18, in the line counter DB1800, the time stamp 1801, the camera ID 1802, and the count number 1803 are stored in association with each other.

The time stamp 1801 is a character string indicating the latest date and time when the number of people count is incremented. The camera ID 1802 is identification information indicating a network camera 102 that captures a person whose number count is incremented. The count number 1803 is the number of people incremented for each camera ID and passing direction, and is the total number of people added from a certain starting point. The count number 1803a is the number of people who have passed in the "IN" direction, and the count number 1803b is the number of people who have passed in the "OUT" direction.

As shown in FIG. 18, in step S1506, the camera ID and the number of people counted for each passing direction are stored.

As described above, according to the present embodiment, after the same person moves in the first direction (for example, "IN") based on the passing direction in which the person has passed the boundary line, the second direction is within a predetermined wandering interval time. When moving to (for example, "OUT"), the same person is excluded from the passage count. Therefore, the number of people passing by is counted as a person who is not passing through the boundary line to move from the area A to the area B (or from the area B to the area A), but simply wanders around the boundary line. It can be removed as noise.

Next, an image processing system according to another embodiment applicable when counting the number of people in an image will be described. The other embodiment is different from the above embodiment in that the person exclusion unit 414 determines at least one of the long-term stayer and the short-time stayer in the predetermined area as the count exclusion target.

Hereinafter, the other embodiment will be described in detail with reference to FIG. FIG. 19 is a flowchart showing an example of counting processing of the number of people reflected in an image in the image processing system according to another embodiment.

In this process, in the number counting process for counting the number of people included in the image, at least one of a long-term stayer and a short-time stayer in a predetermined area (for example, a part of the area 601) is counted. It is a process to exclude from. This processing is performed by the server 100 of the image processing system.

This process starts, for example, when a start instruction for starting this process is input by the user. Further, in this process, for example, when it is determined that all the persons have been analyzed in step S1904 described later, when an end instruction is input by the user, or when a preset processing time is completed, etc. finish.

First, in step S1901, the position information of the stay area and the stay determination time are acquired. The position information of the predetermined area and the stay determination time are acquired from, for example, the setting information set in advance by the user or the like.

A stay area, for example, a predetermined area in a part of the area 601 and surrounded so as to exhibit a predetermined shape such as a rectangle or a circle. The position information of the stay area is shown, for example, in pixel coordinates with respect to the upper left edge of the image showing the area 601. The stay determination time is a time for determining whether the same person stays in the stay area for a long time or a short time. The stay determination time is, for example, a long-term stay determination time (for example, 1 hour) for determining whether or not it is a long time, and a short-term stay determination time (for example, 1 hour) for determining whether or not it is a short time. 10 minutes) and.

Since the subsequent processes of steps S1902 to S1907 are the same as the processes of steps S1502 to S1507 in FIG. 15 described above, the description thereof will be omitted.

In step S1908 following step S1907, is the same person recognized in step S1906 in the stay area based on the position information of the stay area acquired in step S1901 and the position information of the person cached in step S1907? Judge whether or not.

If it is determined in step S1908 that the same person is not in the stay area (step S1908; NO), the process returns to step S1904, and it is determined whether or not there is a person who has not been analyzed. If it is determined in step S1908 that the same person is in the stay area (step S1908; YES), the process proceeds to step S1909.

In step S1909, the staying time of the same person determined to be in the staying area in step S1908 is calculated. Specifically, based on the person position information DB 1600 stored in step S1907, the latest time zone associated with the same person determined to be in the stay area is searched, and the oldest time in the time zone is used. By calculating the difference from the latest time, the staying time of the same person is calculated.

Subsequently, in step S1910, it is determined whether or not the stay time calculated in step S1909 is equal to or longer than the long-term stay determination time acquired in step S1901. That is, it is determined whether or not the recognized same person is a long-term resident who has stayed in the stay area for a predetermined time (for example, 1 hour) or more.

If it is determined in step S1910 that the stay time is longer than the long stay determination time (step S1910; YES), the process proceeds to step S1911. In step S1911, the person exclusion unit 414 determines that the person determined to be a long-term stayer is the count exclusion target, and the number counting unit 412 excludes the number of persons determined to be the count exclusion target from the count. .. After step S1911, the process returns to step S1904, and it is determined again whether or not there is a person who has not been analyzed.

The person excluded from the count in step S1911 is stored as a count exclusion target due to a long stay at the timing of the count exclusion. Further, the number of people excluded from the count in step S1911 is incremented as the number of people excluded from the count due to the long stay at the timing of the count exclusion.

If it is determined in step S1910 that the stay time is not longer than the long stay determination time (step S1910; NO), the process proceeds to step S1912. In step S1912, it is determined whether or not the stay time calculated in step S1909 is equal to or longer than the short stay determination time acquired in step S1901. That is, it is determined whether or not the recognized same person is a short-term resident who has stayed in the stay area for less than a predetermined time (for example, 10 minutes).

If it is determined in step S1912 that the stay time is less than the short stay determination time (step S1912; YES), the process proceeds to step S1913. In step S1913, the person exclusion unit 414 determines that the person determined to be a short-term stayer is the count exclusion target, and the number counting unit 412 excludes the number of persons determined to be the count exclusion target from the count. .. After step S1913, the process returns to step S1904, and it is determined again whether or not there is a person who has not been analyzed.

The person excluded from the count in step S1913 is stored as a count exclusion target due to a short stay at the timing of the count exclusion. Further, the number of people excluded from the count in step S1913 is incremented as the number of people excluded from the count due to a short stay at the timing of the count exclusion.

If it is determined in step S1912 that the stay time is not less than the short stay determination time (step S1912; NO), the process proceeds to step S1914 and the number of people counter is incremented. That is, only persons whose stay time is longer than the short-term determination time and less than the long-term determination time can be counted. Hereinafter, the time that is longer than the short-time judgment time and less than the long-time judgment time is also referred to as "appropriate time", and the person whose stay time is appropriate time is also referred to as "appropriate time stayer". After step S1914, the process returns to step S1904, and the determination as to whether or not there is a person who has not been analyzed is performed again.

In the present embodiment, the result screen display control unit 415 displays, for example, the screen shown in FIG. 20 as a viewing screen for viewing the result of the number of people counting.

FIG. 20 is a diagram showing an example of a viewing screen for counting the number of people. As shown in FIG. 20, on the browsing screen, the counted number of people 2001 counted by excluding the count exclusion target and the count excluded number 2002 as the count exclusion target are displayed.

Further, among the number of people 2002 excluded from the count as the count exclusion target, the number 2003 excluded from the count as a long-term resident and the number 2004 excluded from the count as a short-term resident are displayed in an identifiable manner. ..

Further, among the persons staying in the stay area 2005 indicated by the alternate long and short dash line, the long-term stayer, the short-time stayer, and the appropriate stayer are displayed in an identifiable manner.

For example, a long-term stayer is displayed surrounded by a solid rectangular frame 2006, a short-time stayer is displayed surrounded by a broken line rectangular frame 2007, and a proper stayer is displayed without being surrounded by anything. Further, the staying time of each person in the staying area 2005 may be displayed.

As described above, according to the present embodiment, the long-term stayers and the short-term stayers in the stay area are excluded from the count. Therefore, only the appropriate-time stayers who are neither the long-term stayers nor the short-time stayers in the stay area are counted. can do. The appropriate time can be set to a user's desired time by appropriately setting and changing the long-term stay determination time and the short-term determination time. Therefore, it is possible to count the persons who are staying for the appropriate time desired by the user.

Further, according to the present embodiment, the long-term resident, the short-term resident, the appropriate resident and the number of the appropriate resident are displayed in an identifiable manner. Therefore, it is possible to easily grasp how long the person staying in the staying area has been staying in the staying area.

In the present embodiment, an example of excluding long-term stayers and short-term stayers in the stay area from the count has been described, but the present invention is not limited to this, and either the long-term stayers or the short-term stayers in the stay area are excluded. One may be excluded from the count. For example, when excluding only long-term residents from the count, the short-time determination time may be set to 0. Further, when excluding only short-term residents from the count, the long-term determination time may be set to a very large value.

Although the embodiments have been described above, the present invention can be implemented as, for example, a system, an apparatus, a method, a program, a storage medium, or the like, and specifically, the present invention is applied to a system composed of a plurality of devices. It may be applied to a device consisting of one device.

The present invention includes a software program that realizes the functions of the above-described embodiment, which is directly or remotely supplied to a system or an apparatus. The present invention also includes cases where the computer of the system or apparatus can also read and execute the supplied program code.

Therefore, in order to realize the functional processing of the present invention on a computer, the program code itself installed in the computer also realizes the present invention. That is, the present invention also includes the computer program itself for realizing the functional processing of the present invention.

In that case, as long as it has a program function, it may be in the form of an object code, a program executed by an interpreter, script data supplied to the OS, or the like.

Examples of the recording medium for supplying the program include a flexible disk, a hard disk, an optical disk, a magneto-optical disk, MO, a CD-ROM, a CD-R, a CD-RW, and the like. There are also magnetic tapes, non-volatile memory cards, ROMs, DVDs (DVD-ROM, DVD-R) and the like.

In addition, as a method of supplying the program, the browser of the client computer is used to connect to the homepage of the Internet. Then, it can also be supplied by downloading the computer program of the present invention itself or a compressed file including an automatic installation function to a recording medium such as a hard disk from the homepage.

It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from different homepages. That is, the present invention also includes a WWW server that causes a plurality of users to download a program file for realizing the functional processing of the present invention on a computer.

In addition, the program of the present invention is encrypted, stored in a storage medium such as a CD-ROM, and distributed to users, and the key information for decrypting the encryption is downloaded from the homepage to the user who clears the predetermined conditions. Let me. Then, by using the downloaded key information, it is also possible to execute an encrypted program and install it on a computer.

Further, the function of the above-described embodiment is realized by the computer executing the read program. In addition, based on the instruction of the program, the OS or the like running on the computer performs a part or all of the actual processing, and the function of the above-described embodiment can be realized by the processing.

Further, the program read from the recording medium is written to the memory provided in the function expansion board inserted in the computer or the function expansion unit connected to the computer. After that, based on the instruction of the program, the function expansion board, the CPU provided in the function expansion unit, or the like performs a part or all of the actual processing, and the function of the above-described embodiment is also realized by the processing.

It should be noted that the above-described embodiments are merely examples of embodiment of the present invention, and the technical scope of the present invention should not be construed in a limited manner by these. That is, the present invention can be implemented in various forms without departing from the technical idea or its main features.

100 server 101 client PC
102 network camera

Claims (12)

A counting means that counts the number of people who have passed a predetermined position based on the captured image of the person.
A person identification means for identifying a person who is the same person in the photographed image and has passed the predetermined position in a different direction within a predetermined time after passing the predetermined position.
An image processing system characterized by being equipped with. The image processing system according to claim 1, further comprising display control means for displaying a person specified by the person identification means in an identifiable manner. A counting means that counts the number of people who have passed a predetermined position based on the captured image of the person.
A person identification means for identifying a person who is the same person in the photographed image and has passed the predetermined position a plurality of times in the same direction.
A display control means for displaying and controlling the number of people counted by the counting means excluding the person specified by the person specifying means and the number of people specified by the person specifying means.
An image processing system characterized by being equipped with. The display control means is characterized in that a person whose number of people is counted excluding the person specified by the person identification means by the counting means and a person specified by the person identification means are displayed in an identifiable manner. The image processing system according to claim 2 or 3. The method according to any one of claims 1 to 4, wherein the counting means excludes the person specified by the person specifying means so that the same person is not counted in duplicate. The image processing system described. The image processing system according to any one of claims 1 to 5, further comprising a storage means for storing a person specified by the person specifying means. The direction in which the person has passed the predetermined position, the first direction from the first region to the second region divided by the predetermined position, and the second direction from the second region to the first region. The image processing system according to any one of claims 1 to 6, further comprising a passage direction acquisition means for acquiring any of the directions. The image processing system according to claim 7, wherein the counting means counts the number of people for each passing direction acquired by the passing direction acquisition means. Based on the captured image person is captured, a counting step of counting the number of persons passing through the position of Jo Tokoro,
A person identification step for identifying a person who is the same person in the photographed image and has passed the predetermined position in a different direction within a predetermined time after passing the predetermined position.
A method of controlling an image processing system, which comprises: Computer,
Based on the captured image person is captured, a counting means for counting the number of persons passing through the position of Jo Tokoro,
A program for identifying a person who is the same person in the photographed image and has passed the predetermined position in a different direction within a predetermined time after passing the predetermined position. .. A count step that counts the number of people who have passed a predetermined position based on the captured image of the person.
A person identification step for identifying a person who is the same person in the photographed image and has passed the predetermined position a plurality of times in the same direction.
A display control step for displaying and controlling the number of people counted by the count step excluding the person specified by the person identification step and the number of people specified by the person identification step.
A method of controlling an image processing system, which comprises: A counting means that counts the number of people who have passed a predetermined position based on a photographed image of a person on a computer.
A person identification means for identifying a person who is the same person in the photographed image and has passed the predetermined position a plurality of times in the same direction.
A program for functioning as a display control means for displaying and controlling the number of people counted by the counting means excluding the person specified by the person specifying means and the number of people specified by the person specifying means.

Images