JP7828615B2 - Location management system, location management method, and location management program - Google Patents

Description

This disclosure relates to a location management system, a location management method, and a location management program for managing the location of a person in a specific area.

Traditionally, in areas requiring security, it has been necessary to manage people's movements. Therefore, access control devices aimed at improving security have been considered (see, for example, Patent Document 1). The access control device described in this patent document detects the number of individuals attempting to enter or exit a security area requiring security measures through image analysis of captured images. Furthermore, it detects the number of individuals attempting to enter or exit the security area by reading a color barcode attached to each individual. If the number detected by image analysis and the color barcode match, entry or exit to the security area is permitted.

Furthermore, object tracking devices that track objects contained in moving images are being considered (see, for example, Patent Document 2). The object tracking device described in this patent document detects feature points in each frame by inputting each frame image of the moving image into a machine learning model capable of outputting the type of subject. The object is tracked by comparing the positions of the feature points in each frame.

Japanese Patent Publication No. 2017-173890 Japanese Patent Publication No. 2019-185210

However, the technology described in Patent Document 1 cannot manage the movement of people within a security area. Furthermore, the technology described in Patent Document 2 cannot identify the tracked object.

The location management system that solves the above problems comprises a camera and a control unit connected to a location information storage unit. The control unit identifies a person image region in the image captured by the camera, sequentially identifies the person's location according to the position of the person image region, and records it in the location information storage unit. If a code image is detected in the person image region, the control unit records the code corresponding to the code image in the location information storage unit, associating it with the location.

According to this invention, security can be improved by managing the user's location.

This is an explanatory diagram of the location management system according to the embodiment. This is an explanatory diagram of the hardware configuration of the embodiment. This is an explanatory diagram of the processing procedure of the embodiment. This is an explanatory diagram of the processing procedure of the embodiment. This is an explanatory diagram of the captured images of the embodiment. This is an explanatory diagram of the captured images of the embodiment. This is a diagram illustrating the processing procedure for the example changes.

Below, an embodiment of the location management system will be described with reference to Figures 1 to 6.
This embodiment describes a location management system, location management method, and location management program used to manage the location of workers (users) within a factory. In this embodiment, the system identifies the user by recognizing a code image held by the user and tracks the user's location within the managed area using image tracking technology. Chameleon Code (registered trademark) is used as the code image. Chameleon Code is a code image that displays a code by arranging multiple colors (cyan, magenta, yellow, and black) in a predetermined layout. This Chameleon Code can recognize multiple code images at once, even if they are included in a single captured image.
As shown in Figure 1, the location management system A1 consists of a camera C1, a management terminal 10, a management server 20, etc., all connected via a network.

(Description of hardware configuration)
Figure 2 illustrates the hardware configuration of the information processing device H10, which constitutes the management terminal 10 and the management server 20. The information processing device H10 includes a communication device H11, an input device H12, a display device H13, a storage device H14, and a processor H15. Note that this hardware configuration is just one example, and it can be implemented using other hardware.

Communication device H11 is an interface that establishes a communication path with other devices and performs data transmission and reception; for example, it could be a network interface or a wireless interface.

The input device H12 is a device that accepts input of various types of information, such as a mouse or keyboard. The display device H13 is a display that shows various types of information.
The storage device H14 is a storage device that stores data and various programs for executing various functions of the management terminal 10 and the management server 20. Examples of storage devices H14 include ROM, RAM, and hard disks.

The processor H15 controls the various processes in the management terminal 10 and the management server 20 using programs and data stored in the memory device H14. Examples of processor H15 include CPUs, GPUs, and MPUs. This processor H15 loads programs stored in ROM, etc., into RAM and executes various processes for each task.

The processor H15 is not limited to performing all of its operations through software processing. For example, the processor H15 may include dedicated hardware circuits (e.g., application-specific integrated circuits: ASICs) that perform hardware processing for at least some of the operations it performs. The processor includes a CPU/GPU and memory such as RAM and ROM, where memory stores program code or instructions configured to cause the CPU/GPU to execute the processing. Memory, or computer-readable media, includes any available media accessible by a general-purpose or dedicated computer.

(Explanation of location management system A1)
Camera C1 transmits a color video of a predetermined field of view within the managed area to the management server 20. Camera C1 is installed in the factory in a position that allows it to view the managed area from above (for example, on the ceiling). Camera C1 then generates an image of the managed area below (a predetermined shooting range) taken from above.

The management terminal 10 is a computer terminal used by an administrator to manage the location of users.
The management server 20 is a computer that manages the location of users. The management server 20 comprises a control unit 21, a user information storage unit 22, an area information storage unit 23, and a location information storage unit 24.

The control unit 21 performs the processes described later (image recognition, tracking, and code recognition). By executing the management program for these processes, the control unit 21 functions as the image recognition unit 211, the tracking unit 212, and the code recognition unit 213.

The image recognition unit 211 recognizes a person in the captured image and executes a process to generate a bounding box (region of the image containing the person). This image recognition unit 211 can be implemented, for example, using the object detection algorithm YOLO®.

The tracking unit 212 performs a process to track the movement of a person by sequentially identifying their location using the feature quantities of the person image region contained in each frame image of the video.
The code recognition unit 213 identifies the code image contained in the frame image and performs a process to obtain the color code by decoding the code image.

The user information storage unit 22 stores user management data for managing each user. This user management data is recorded when each user is registered. This user management data includes color code, user ID, name, and affiliation information.

A color code is an identifier used to identify a colored image.
The user ID is an identifier used to identify the user who carries the code image.
The name and affiliation are information related to this user's name and affiliation.

The area information storage unit 23 stores area management data for managing the area captured by camera C1. This area management data is recorded when the installation location of camera C1 is registered. This area management data includes the camera ID and data related to the shooting area.

The camera ID is an identifier used to identify each camera C1.
The shooting area consists of coordinates representing the region included in the image captured by this camera within the managed area. This shooting area allows for the identification of the coordinates where a subject (person) is located within the managed area.

The location information storage unit 24 stores location management data for managing the location of each user. This location management data is recorded when a user's location is identified. This location management data includes data related to the detection time and location for each feature quantity, person image ID, and user ID.

Feature quantities are information (such as numerical values or vectors) that represents the shape characteristics of the subject contained within the detected human image region.
A person image ID is an identifier that identifies the area of a person in an image.

The user ID is an identifier for a user identified by recognizing the code image contained within the area of the person's image.
The detection time is the date and time when the area containing the person in the image was detected.
The location information is used to identify the place where this person image area was detected. This location information can be represented, for example, by coordinates or an area identifier.

(Person tracking process)
Next, we will explain the person tracking process using Figure 3.

Here, the control unit 21 of the management server 20 executes the frame image acquisition process (step S11). Specifically, the image recognition unit 211 of the control unit 21 acquires a predetermined frame image (captured image) from the video captured by the camera C1.

Next, the control unit 21 of the management server 20 performs person recognition processing (step S12). Specifically, the image recognition unit 211 of the control unit 21 searches for a person in the frame image using object recognition processing.

Next, the control unit 21 of the management server 20 performs a determination process to determine whether a person is included (step S13). Specifically, the image recognition unit 211 of the control unit 21 determines that a person is included if it detects a person in the frame image.

If it is determined that no people are included (i.e., "NO" in step S13), the control unit 21 of the management server 20 returns to the frame image acquisition process (step S11). Here, the image recognition unit 211 of the control unit 21 acquires subsequent frame images from the video.

On the other hand, if it is determined that a person is present (if the answer is "YES" in step S13), the control unit 21 of the management server 20 performs a process to identify the person image area (step S14). Specifically, the image recognition unit 211 of the control unit 21 identifies the person image area in the frame image where a person has been recognized.

As shown in Figure 5, the frame image 500 contains multiple individuals P1 and P2. In this case, the individual image regions IP1 and IP2 are identified corresponding to each individual P1 and P2.

Next, the control unit 21 of the management server 20 performs a determination process to determine whether the feature quantities of the person image region are already registered (step S15). Specifically, the tracking unit 212 of the control unit 21 calculates the feature quantities of each person image region. Next, the tracking unit 212 checks whether feature quantities with a high similarity to the calculated feature quantities are recorded in the location information storage unit 24. Here, for example, if the feature quantities match by 80% or more, it is determined that the similarity is high.

If no highly similar features are recorded, and the features of the person image region are determined not to be already registered (i.e., "NO" in step S15), the control unit 21 of the management server 20 executes the process of assigning a new registration code (step S16). Specifically, the tracking unit 212 of the control unit 21 assigns a person image ID to the features of the person image region. Next, the tracking unit 212 records the location management data, including the features and person image ID, in the location information storage unit 24.

On the other hand, if a feature with a high degree of similarity is recorded and the feature in the person image region is determined to be already registered (if "YES" is answered in step S15), the control unit 21 of the management server 20 performs the process of identifying the already registered code (step S17). Specifically, the tracking unit 212 of the control unit 21 identifies the person image ID recorded in the location management data where the feature with a high degree of similarity is recorded in the location information storage unit 24.

Next, the control unit 21 of the management server 20 performs the process of recording the location of the person (step S18). Specifically, the tracking unit 212 of the control unit 21 uses the area information storage unit 23 to identify the location according to the position of the person's image area. The tracking unit 212 then records the location along with the detection time in the location management data.
Next, the control unit 21 of the management server 20 returns to the frame image acquisition process (step S11).

(Code recognition processing)
Next, we will explain the code recognition process using Figure 4. This process is performed when a frame image is acquired.

Here, the control unit 21 of the management server 20 performs the code image search process (step S21). Specifically, the code recognition unit 213 of the control unit 21 searches for code images in the frame image by pattern recognition of the layout pattern of the code images.

Next, the control unit 21 of the management server 20 performs a determination process to determine whether a code image is included (step S22). Specifically, the code recognition unit 213 of the control unit 21 determines that a code image is included if it detects the layout pattern of a code image in the frame image. It is also possible to identify multiple code images in the frame image.

If it is determined that no code image is present (i.e., "NO" in step S22), the control unit 21 of the management server 20 terminates the code recognition process. Then, if a subsequent frame image is acquired, the code recognition process is executed again.

On the other hand, if it is determined that a code image is included (if the answer is "YES" in step S22), the control unit 21 of the management server 20 performs code identification processing (step S23). Specifically, the code recognition unit 213 of the control unit 21 decodes the code image and identifies the color code.

Next, the control unit 21 of the management server 20 performs a determination process to determine whether it corresponds to a person (step S24). Specifically, the code recognition unit 213 of the control unit 21 compares the detection position of the code image with the arrangement of the person image area. If the detection position of the code image is included within the person image area, it determines that it corresponds to a person.

If the detected location of the code image is outside the area of the person image and it is determined that it does not correspond to a person (i.e., "NO" in step S24), the control unit 21 of the management server 20 terminates the code recognition process.

On the other hand, if the detected location of the code image is within the person image area and it is determined to correspond to a person (if the answer is "YES" in step S24), the control unit 21 of the management server 20 performs user ID identification processing (step S25). Specifically, the code recognition unit 213 of the control unit 21 uses the user information storage unit 22 to identify the user ID corresponding to the color code.

As shown in Figure 6, in frame image 501, code images CC1 and CC2 are detected. In this case, code image CC1 within the person image area IP1 of person P1 and code image CC2 within the person image area IP2 of person P2 are associated with each other. On the other hand, code image CC3, which is outside the person image area, is not used to identify the user ID because it is outside the person image area.

Next, the control unit 21 of the management server 20 performs the process of recording the user ID for the person image (step S26). Specifically, the code recognition unit 213 of the control unit 21 records the user ID in the location management data where the person image ID of the person image area is recorded.

(Location verification process)
When an administrator needs to check a user's location, they access the management server 20 using the management terminal 10. In this case, the control unit 21 of the management server 20 places the movement trajectories of each user, linked in chronological order from the location information storage unit 24, onto a map of the managed area. The control unit 21 then outputs the map with the movement trajectories to the management terminal 10.

(Operation of this embodiment)
By using both object recognition and code recognition, a single captured image can be used to track a person identified through object recognition processing, and this person can also be identified using a code image.

(Effects of this embodiment)
(1) In this embodiment, the control unit 21 of the management server 20 performs frame image acquisition processing (step S11) and person recognition processing (step S12). This makes it possible to confirm the presence of a person in the captured image.

(2) In this embodiment, if it is determined that a person is included (if the answer is "YES" in step S13), the control unit 21 of the management server 20 performs a process to identify the area of the person's image (step S14). This makes it possible to identify the area in the frame image where the person is visible.

(3) In this embodiment, if it is determined that the feature quantities of the person image area are not already registered (i.e., "NO" in step S15), the control unit 21 of the management server 20 executes the process of assigning a new registration code (step S16). This allows a person image ID to be assigned to newly photographed people.

(4) In this embodiment, if the feature quantities in the person image region are determined to be already registered (if "YES" is answered in step S15), the control unit 21 of the management server 20 executes a process to identify the already registered code (step S17). This makes it possible to track person images with common feature quantities.

(5) In this embodiment, the control unit 21 of the management server 20 performs the process of recording the location of the person (step S18). This makes it possible to identify the person's location using the captured image.

(6) In this embodiment, the control unit 21 of the management server 20 performs a code image search process (step S21). This allows the code image to be identified in the captured image.

(7) In this embodiment, the control unit 21 of the management server 20 performs code identification processing (step S23) and determination processing (step S24) to determine whether it corresponds to a person. This allows the code image to be associated with the person recognized by the object recognition processing.

(8) In this embodiment, if it is determined that a person is represented (if "YES" is answered in step S24), the control unit 21 of the management server 20 executes a user ID identification process (step S25). This makes it possible to manage the location of users. Normally, in factories, workers wear the same work clothes, making it difficult to identify individuals. It is also possible to assign a unique code to each individual and track them, but this requires equipment. For example, if tracking is done using color codes, cameras must be installed at a distance where the color codes can be recognized, requiring many cameras. Also, if wireless technology such as RFID is used, many antennas must be installed. In this invention, since the code image is associated with a person and the person is tracked, location management can be performed with relatively simple equipment.

This embodiment can be implemented with the following modifications. This embodiment and the following modifications can be combined with each other to the extent that they do not contradict each other technically.
In the above embodiment, the image recognition unit 211 is implemented by the object detection algorithm YOLO. The person tracking process is not limited to the use of YOLO.
In the above embodiment, Chameleon Code is used as the code image. However, it is not limited to Chameleon Code as long as it is a code image that can be recognized by a camera.

• In the above embodiment, a camera is provided for each area. Alternatively, multiple cameras may be placed at different locations within a single area. In this case, one person is filmed by multiple cameras.

As shown in Figure 7, cameras C11 and C12, which photograph the same area, are installed in different positions.
Then, the control unit 21 of the management server 20 executes the process of acquiring person features for each camera (step S31). Specifically, the image recognition unit 211 of the control unit 21 captures people P3 and P4 with each of the cameras C11 and C12. In this case, camera C11 identifies the person image regions IP31 and IP41 of people P3 and P4. Camera C12 also identifies the person image regions IP32 and IP42 of people P3 and P4. Then, image features are calculated from the person image regions IP31, IP41, IP32, and IP42. In this case, image features and a user ID are associated with each person P3 and P4.

Next, the control unit 21 of the management server 20 performs a process to identify person features with high similarity (step S32). Specifically, the tracking unit 212 of the control unit 21 identifies location management data 241 and 242, respectively, whose person image regions captured by cameras C11 and C12 have similar feature quantities. Then, based on the similarity of the respective image feature quantities, it performs tracking of the person image region using the location management data 241 and 242.

This allows for complementary tracking by utilizing human features captured by multiple cameras. For example, if the features of the human image region in one camera's image become dissimilar, making tracking impossible, tracking can continue using the image from the other camera. In this case, if one camera detects a human image region with similar features and the system returns to a state where tracking is possible using the captured image, tracking becomes possible again using all cameras.

A1…Location management system, C1…Camera, 10…Management terminal, 20…Management server, 21…Control unit, 211…Image recognition unit, 212…Tracking unit, 213…Coat recognition unit, 22…User information storage unit, 23…Area information storage unit, 24…Location information storage unit.

Claims (5)

A location management system comprising a camera and a control unit connected to a location information storage unit,
The control unit,
In the image captured by the aforementioned camera, the region of the image in which a person was recognized is identified.
The feature quantities of the subject included in the person image region are calculated, and if the similarity between the feature quantities and the previously registered feature quantities recorded in the location information storage unit is greater than or equal to a predetermined threshold, the previously registered person image ID is associated with the person image region, and if the similarity is less than the predetermined threshold, a new person image ID is assigned.
Regardless of whether a code image is detected, a first process sequentially identifies the location of the person according to the position of the person image region and records it in the location information storage unit.
A location management system characterized by performing a second process in which, when a code image is detected in the person image area , a code corresponding to the code image is recorded in the location information storage unit in association with the time series of the location associated with the person image ID . The location management system according to claim 1, characterized in that the control unit connects the location data in chronological order to identify the movement trajectory. The control unit,
Multiple cameras recognize the same person and identify the respective image regions of the person.
The location management system according to claim 1 or 2, characterized in that when the code image is detected in the person image region of at least one of the images captured by the camera, a code corresponding to the code image is associated with each of the person image regions. A method for managing locations using a location management system that includes a camera and a control unit connected to a location information storage unit,
The control unit,
In the image captured by the aforementioned camera, the region of the image in which a person was recognized is identified.
The feature quantities of the subject included in the person image region are calculated, and if the similarity between the feature quantities and the previously registered feature quantities recorded in the location information storage unit is greater than or equal to a predetermined threshold, the previously registered person image ID is associated with the person image region, and if the similarity is less than the predetermined threshold, a new person image ID is assigned.
Regardless of whether a code image is detected, a first process sequentially identifies the location of the person according to the position of the person image region and records it in the location information storage unit.
A location management method characterized by performing a second process in which, when a code image is detected in the person image area , a code corresponding to the code image is recorded in the location information storage unit in association with the time series of the location associated with the person image ID . A program for managing locations using a location management system that includes a camera and a control unit connected to a location information storage unit,
The control unit,
In the image captured by the aforementioned camera, the region of the image in which a person was recognized is identified.
The feature quantities of the subject included in the person image region are calculated, and if the similarity between the feature quantities and the previously registered feature quantities recorded in the location information storage unit is greater than or equal to a predetermined threshold, the previously registered person image ID is associated with the person image region, and if the similarity is less than the predetermined threshold, a new person image ID is assigned.
Regardless of whether a code image is detected, a first process sequentially identifies the location of the person according to the position of the person image region and records it in the location information storage unit.
A location management program characterized by being configured as a means to perform a second process in which, when a code image is detected in the person image area, a code corresponding to the code image is recorded in the location information storage unit in association with the time series of the location associated with the person image ID .