JP7638857B2 - Surveillance system - Google Patents

Description

The present invention relates to a surveillance system that displays an image of a specified surveillance area captured by an imaging device on a display device.

In recent years, human detection and object recognition technologies using deep learning and other techniques have been developed. These technologies make it possible to automatically recognize a person's physical characteristics, such as gender and age, whether they are wearing a mask, and whether they are using care products such as a wheelchair or a white cane.

In order to utilize such person detection and object recognition technologies, surveillance cameras are being installed to ensure safety and security in facility operations. On the other hand, consideration must be given to the privacy of users in vaccination sites for infectious diseases, which have been spreading in recent years, as well as in changing rooms and restrooms in hospitals, commercial facilities, and airport facilities, or in their vicinity. For this reason, there are some locations where surveillance cameras cannot be installed, even if the purpose is to ensure safety and security in facility operations or to improve customer satisfaction. Therefore, in the past, anonymized video that takes privacy into consideration was created after object recognition and human behavior analysis, and this was used for video surveillance.

The following are examples of prior art in the technical field related to the present invention. For example, Patent Document 1 discloses an invention that protects an individual's privacy by reducing or modifying personal privacy information while maintaining a state in which it is possible to determine the identity of a person across multiple cameras.

JP 2006-287287 A

There were concerns that surveillance footage that overly anonymizes characteristics such as vaccination status and whether or not a mask is worn, as well as personal and physical characteristics, would make it difficult to properly monitor situations such as the "Three Cs" (closed spaces, crowded places, and close contact) as part of infectious disease prevention measures. In addition, the vaccination status of facility personnel is synonymous with indicating the cleanliness of the facility itself and whether it is a safe and secure environment, so for facility operators, disclosing the vaccination status of facility employees also had the advantage of improving customer satisfaction. However, displaying footage that does not take into account the wishes of facility employees not to get vaccinated was problematic from the perspective of protecting the personal information of unvaccinated people.

The present invention was made in consideration of the above-mentioned conventional circumstances, and aims to provide a surveillance system that can monitor unvaccinated individuals using images that respect their privacy.

In order to achieve the above object, a surveillance system according to one aspect of the present invention is configured as follows. That is, the surveillance system according to the present invention includes an imaging device that captures an image of a specified surveillance area, an image processing device that determines the need for anonymization of each person included in an image captured by the imaging device and performs an anonymization process to process image portions of people determined to require anonymization, and a display device that displays images after processing by the image processing device, and is characterized in that the image processing device checks the vaccination status of people included in the image by referring to person data including the vaccination status of people registered in advance, and determines that people who cannot be confirmed to have been vaccinated are people who require anonymization.

Here, in the surveillance system according to the present invention, the person data is associated with a facial image of a person registered in advance or facial feature data including the features thereof, and the image processing device can be configured to identify a person included in an image captured by the imaging device through face recognition processing using the facial feature data, and to determine the need for anonymization based on the person data for the identified person.

In addition, in the surveillance system according to the present invention, the person data is associated with a worn item image corresponding to the affiliation of the person registered in advance or wearable item feature data including the feature amount thereof, and the image processing device can be configured to identify the person included in the image captured by the imaging device through a wearable item recognition process using the wearable item feature data, and to determine the need for anonymization based on the person data for the identified person.

In addition, in the surveillance system according to the present invention, the image processing device can be configured to include people who are not the subject of personal data among people included in an image captured by the imaging device as people who need to be anonymized.

The monitoring system according to the present invention may also be configured to display an alarm on the display device when the number of people who need to be anonymized within a specified range is equal to or greater than a specified value.

The monitoring system according to the present invention may also be configured to display an alarm on the display device if a state that satisfies the above conditions continues for a predetermined period of time or longer.

The present invention provides a surveillance system that can monitor unvaccinated individuals using images that respect their privacy.

1 is a diagram illustrating an example of the configuration of a monitoring system according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an example of a flowchart of a process performed by the monitoring system of FIG. 1 . FIG. 2 is a diagram showing a display example of a passenger electronic signage display device. FIG. 13 is a diagram showing a display example of a facility operator display device. FIG. 13 is a diagram showing a display example of a facility operator display device. FIG. 13 is a diagram showing a display example of a facility operator display device.

One embodiment of the present invention will be described with reference to the drawings. FIG. 1 shows an example of the configuration of a surveillance system according to one embodiment of the present invention. The surveillance system in FIG. 1 includes an imaging device 10, an image processing device 20, a passenger electronic signage display device 30, a facility operator display device 40, and an image recording device 50. Each of these devices is connected to a network N. The surveillance system in this example is intended for application to facilities such as train stations and airports, but can also be applied to a variety of other facilities.

The imaging device 10 has a camera configuration and is placed in various locations. For example, it is placed as a surveillance camera to capture a specified surveillance area. The imaging device 10 can be a camera configuration that obtains information by focusing incident light on an imaging element via a lens and an aperture. Examples of imaging elements here include a CCD (Charge-Coupled Device) image sensor and a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The imaging device 10 captures images at, for example, 3 frames per second (3 fps) or more, and the information is sent to the image processing device 20 and the image recording device 50. Multiple imaging devices 10 can be installed depending on the shape, size, and other conditions of the surveillance area.

The image processing device 20 has a control unit 21, a user interface unit 22, an image analysis unit 23, an anonymized image generation unit 24, and an alarm generation unit 25.

The control unit 21 can exchange information with other devices through the network N, and controls the user interface unit 22, image analysis unit 23, anonymized image generation unit 24, and alarm generation unit 25. When a user inputs or operates the passenger electronic signage display device 30 or the facility operator display device 40, the user interface unit 22 performs processing such as accepting the input or operation information. The image analysis unit 23 analyzes the image captured by the imaging device 10, and performs processing to determine the necessity of anonymization for each person according to the vaccination status of each person included in the image. The anonymized video generation unit 24 performs processing to apply processing to the image portion of the image captured by the imaging device 10 for a person determined to require anonymization. The alarm generation unit 25 performs processing to generate an alarm according to the congestion status of unvaccinated people.

The passenger electronic sign display device 30 and the facility operator display device 40 are display devices that can display the contents processed by the image processing device 10. The passenger electronic sign display device 30 is a display device provided to users of facilities such as stations and airports, and is installed in any location that the facility users can visit. The facility operator display device 40 is a display device provided to those involved with the facility (operators and employees), and is installed in a location that can only be used by those involved with the facility, such as a monitoring room. These display devices 30, 40 display using, for example, a liquid crystal display (LCD), an organic electroluminescence (OEL) display, a touch panel, or the like. The display devices 30, 40 may also be equipped with operation buttons. The number of display devices 30, 40 can be installed as needed.

The image recording device 50 is a device that records images captured by the imaging device 10. The image recording device 50 may also record images or results after processing by the image processing device 10. As the image recording device 50, for example, a hard disk drive (HDD), a solid state drive (SSD), a digital data storage (DDS), or other suitable method can be applied as necessary.

The network N is a line capable of data communication that connects each device. The network N can be any type of line, including a dedicated line, an intranet, or an IP network such as the Internet.

Figure 2 shows an example of a flowchart of processing by the monitoring system of this example. Here, it is assumed that person data in which facility personnel are registered in advance is stored in a database (not shown) accessible from the image processing device 20. In addition to the person's name, this person data includes data indicating vaccination status. In addition, various feature data expressing the characteristics of the person are associated with the person data. Examples of feature data include physical feature data indicating the person's age, sex, whether or not they wear glasses, whether or not they wear hearing aids, whether or not they can walk, etc., facial feature data required for face recognition processing to identify a person, and wearable item feature data required for wearable item recognition processing to identify the department (category) to which the person belongs. The facial feature data includes, for example, a person's facial image or its feature values. The wearable item feature data includes, for example, images or feature values of hats, uniforms, badges, armbands, belongings, etc. according to the department to which the person belongs.

The image processing device 20 analyzes the images captured by the imaging device 10 using the image analysis unit 23. Specifically, the image analysis unit 23 first detects people included in each frame of the image captured by the imaging device 10 (step S11). Next, the image analysis unit 23 extracts features related to each person detected from the image captured by the imaging device 10 (step S12). In this example, facial features and features of worn items are extracted as features related to the people.

Next, the image analysis unit 23 performs an article recognition process to compare the features of the person's articles of clothing detected from the image with the article characteristic data, and determines whether the person is a facility-related person who belongs to the facility (step S13). For example, if article characteristic data is found that matches the features of the articles of clothing extracted from the image to a predetermined value or higher, the person is determined to be a facility-related person. On the other hand, if no article characteristic data that meets the above requirements is found, the person is determined to be not a facility-related person (i.e., a facility user).

Next, the image analysis unit 23 performs face recognition processing on the person determined to be a facility-related person to match the facial features extracted from the image with the facial feature data to identify the person, and determines the vaccination status based on the person data associated with the corresponding facial feature data (step S14). For example, if facial feature data is found that matches the facial features extracted from the image to a predetermined value or more, it determines whether the person has been vaccinated or not by referring to the person data associated with the facial features. On the other hand, if no facial feature data that meets the above requirements is found, it determines that the vaccination status is unknown. Furthermore, the image analysis unit 23 determines that the vaccination status is unknown for people determined not to be facility-related people (i.e., facility users) without performing face recognition processing.

Next, the image analysis unit 23 performs a process of recording the results of the judgments made in steps S13 and S14, i.e., the judgment results of whether the person is a facility-related person or not and the judgment results of the vaccination status, in memory (step S15). The above process is repeatedly performed for all people included in the image captured by the imaging device 10 (step S16).

Next, the anonymized image generating unit 24 processes the image of a person who has not been confirmed as having been vaccinated by the analysis in the image analyzing unit 23, that is, a person who has not been vaccinated or whose vaccination status is unknown, as a person who needs to be anonymized, and performs a process of anonymizing the image of the person by processing the image portion of the person (step S17). For example, in anonymizing a face, the face image or face part image of the target person is compared (for example, feature comparison) with multiple types of abstract face images or abstract face part images registered in advance, and replaced and synthesized with the abstract face image or abstract face part image with the highest similarity. In a similar manner, the body and clothes other than the face may be anonymized. Alternatively, one or more avatar images may be prepared in association with age and gender, and the target person may be replaced and synthesized with an avatar image according to the age and gender of the target person.

Next, the alarm generation unit 25 determines whether an alarm is necessary depending on the degree of congestion of people who need to be anonymized (i.e. people who have not been vaccinated or whose vaccination status is unknown), and issues an alarm if it is determined that an alarm is necessary (step S18). For example, it determines that an alarm is necessary when the condition that the number of people who need to be anonymized within a specified range is equal to or greater than a specified value is satisfied. Alternatively, it determines that an alarm is necessary when a state that satisfies the above condition continues for a specified period of time or more. In addition, the behavior of people who need to be anonymized may be analyzed by detecting skeletal structures, etc., and used for the above determination.

The anonymized image generated by the anonymized image generation unit 24 is transmitted to the passenger electronic signage display device 30 and the facility operator display device 40 and displayed on their screens. In addition, when an alarm is generated by the alarm generation unit 25, the data is transmitted to the passenger electronic signage display device 30 and the facility operator display device 40 and displayed on their screens.

Figure 3 shows an example of the display of the passenger electronic signage display device 30. The screen of the passenger electronic signage display device 30 in this example has a display section 31 showing the vaccination status at required locations in the facility, a display section 32 showing a map of the facility, and a display section 33 showing an anonymized image that has been anonymized from an image captured by one of the cameras (imaging device 10) in the facility.

For example, the display unit 31 displays a pie chart showing the ratio of vaccinated and unvaccinated facility personnel in the vicinity of each of the three gates A, B, and C installed within the facility. This ratio can be calculated by using the image processing device 10 to analyze images captured by an imaging device 10 installed near the area and identify vaccinated and unvaccinated facility personnel (including those with unknown vaccination status).

The cameras to be displayed by the display unit 33 may be automatically switched at regular intervals, for example. In addition, the facility user may be able to select the camera to be displayed by the display unit 33 by operating the passenger electronic signage display device 30. For example, when the facility user touches the map displayed on the display unit 32, anonymized images may be obtained for the cameras installed in the location within the facility that corresponds to the touched position, and displayed on the display unit 33.

Figures 4, 5 and 6 show examples of the display of the facility operator display device 40. The screen of the facility operator display device 40 in this example has a display section 41 showing a map of the facility and the vaccination status at required locations in the facility, a display section 42 showing an image or anonymized image captured by any camera (imaging device 10) in the facility, an operation section 43 that accepts an operation from facility personnel to select a camera to be displayed by the display section 42, a display section 44 showing the occurrence status of an alarm, and an operation section 45 that accepts an operation from facility personnel to check the details of an alarm.

As shown in FIG. 4, the facility operator's display device 40 normally uses the area occupying most of the screen as a display section 41 for a map of the facility and the vaccination status at required locations in the facility, and when a camera selection operation is received at the operation section 43, it switches to a display section 42 for an image captured by that camera (imaging device 10) or an anonymized image, as shown in FIG. 5 or FIG. 6. FIG. 5 is an example of a format in which an image captured by the camera is displayed directly on the display section 42, and FIG. 6 is an example of a format in which an image captured by the camera is anonymized and then displayed on the display section 42. Which display format is adopted may be determined in advance, or may be dynamically switched depending on the authority of the user of the facility operator's display device 40, etc.

The display unit 44 displays the occurrence of alarms in the facility in chronological order. Furthermore, when the operation unit 45 is operated with any alarm selected on the display unit 44, details of the selected alarm are further displayed. At this time, an image or an anonymized image at the time of the alarm may be read from the image recording device 50 and displayed on the display unit 42.

As described above, the surveillance system of this example includes an imaging device 10 that captures an image of a specified surveillance area, an image processing device 20 that performs an anonymization process that determines the need for anonymization of each person included in the image captured by the imaging device 10 and processes the image portions of the people determined to require anonymization, and a passenger electronic signage display device 30 and a facility operator display device 40 that display the images after processing by the image processing device 20. The image processing device 20 checks the vaccination status of people included in the image by referring to person data including the vaccination status of people registered in advance, and determines that people who cannot be confirmed to have been vaccinated (i.e. people who have not been vaccinated or whose vaccination status is unknown) are people who require anonymization.

Therefore, according to the monitoring system of this example, the passenger electronic signage display device 30 can display the vaccination status to facility users, so that if the vaccination status of facility personnel is low, the possibility of viral infection from facility personnel is low, and facility users can be informed that the facility is safe and secure. On the other hand, for people who have not been vaccinated or whose vaccination status is unknown, they can be anonymized to respect the individual's privacy and intentions of not being vaccinated, making it possible to monitor using images that respect the privacy of unvaccinated people.

In addition, in the monitoring system of this example, when the image processing device 20 is able to identify the affiliation (i.e., that the person is a facility-related person) of a person included in an image captured by the imaging device 10 through a wearing item recognition process using the wearing item feature data, the image processing device 20 is configured to identify the person through a face recognition process using the face data, and to determine the need for anonymization based on the person data for the identified person. With this configuration, it is possible to omit the face recognition process for people who are found to not be facility-related through the wearing item recognition process, thereby making it possible to efficiently determine the vaccination status.

In this example, anonymization processing is performed to process the image portions of unvaccinated individuals, but image portions of vaccinated individuals may also be processed. That is, for example, facial images of facility personnel without infectious disease control products such as masks or face guards are captured and stored, and the facial images of vaccinated individuals are replaced with facial images without infectious disease control products. This makes it easier for facility personnel to identify people in the images when performing surveillance work.

In addition, in this example, all facility users are anonymized as people with unknown vaccination status, but facility users who use nursing care items such as wheelchairs or white canes can be excluded from the anonymization process for the purpose of providing support to the users. This makes it easier to identify people who need support, allowing facility personnel to quickly go to those people.

Although the embodiments of the present invention have been described above, these embodiments are merely illustrative and do not limit the technical scope of the present invention. The present invention can take various other embodiments, and various modifications such as omissions and substitutions can be made without departing from the gist of the present invention. These embodiments and their modifications are included in the scope and gist of the invention described in this specification, etc., and are included in the scope of the invention described in the claims and their equivalents.

The present invention can be provided not only as the devices described above or as systems composed of these devices, but also as methods executed by these devices, programs for implementing the functions of these devices using a processor, and storage media for storing such programs in a computer-readable format.

The present invention can be used in a surveillance system that displays an image of a specified surveillance area captured by an imaging device on a display device.

10: Imaging device, 20: Image processing device, 21: Control unit, 22: User interface unit, 23: Image analysis unit, 24: Anonymous image generation unit, 25: Alarm generation unit, 30: Passenger electronic sign display device, 40: Facility operator display device, 50: Image recording device

Claims (4)

An imaging device that captures an image of a predetermined monitoring area;
an image processing device that performs an anonymization process by referring to personal data including the vaccination status of pre-registered persons, confirming the vaccination status of each person included in the image captured by the imaging device, determining that a person who has not been confirmed to have been vaccinated is a person who needs to be anonymized, and processing the image portion of that person;
a display device for displaying an image after processing by the image processing device,
The person data is associated with face feature data including a face image or a feature amount of the pre-registered person, and wearing item feature data including a wearing item image or a feature amount of the wearing item according to an affiliation of the pre-registered person,
The surveillance system is characterized in that, when the image processing device can identify the affiliation of a person included in an image captured by the imaging device through wearing item recognition processing using the wearing item feature data, the image processing device identifies the person through face recognition processing using the facial feature data, and determines the need for anonymization based on the personal data of the identified person . 2. The monitoring system according to claim 1 ,
The surveillance system is characterized in that the image processing device includes, among people included in an image captured by the imaging device, people who are not the subject of the personal data, as people who need to be anonymized. 3. The monitoring system according to claim 1 ,
A monitoring system characterized in that, when a condition that the number of people who need to be anonymized within a predetermined range is equal to or greater than a predetermined value is satisfied, an alarm is displayed on the display device. 4. The monitoring system according to claim 3 ,
a monitoring system that displays an alarm on the display device when the state satisfying the condition continues for a predetermined period of time or longer.

Images