JP7564064B2 - Video monitoring system and video monitoring method - Google Patents

Description

This disclosure relates to a video surveillance system and a video surveillance method.

In security and surveillance work, video surveillance systems are used that display surveillance footage transmitted from cameras installed at surveillance locations, and it is important for security personnel to detect specific events, which are specific surveillance events, by checking the surveillance footage. Specific events include, for example, incidents such as crimes or accidents, and events that require response by security, police, or emergency services.

To detect specific events, supervisors such as monitors and security guards must determine whether there is anything unusual about the appearance and behavior of people, objects, and other subjects captured on surveillance footage, taking into account factors such as location and time. For this reason, the experience or intuition of the supervisor is required to detect specific events, which can be difficult to verbalize and communicate.

In response to this, Patent Document 1 discloses a video surveillance system in which a user registers the likelihood of an incident occurring as attribute information for past surveillance footage that shows an incident. This video surveillance system detects incidents from new surveillance footage based on the registered attribute information.

JP 2019-198000 A

The technology described in Patent Document 1 is based on the premise that the user who reviews the surveillance footage is able to register appropriate attribute information related to the incident, so if there is no user with specialized knowledge of surveillance work, there is a problem that incidents cannot be detected accurately.

The objective of this disclosure is to provide a video surveillance system and a video surveillance method that are capable of detecting specific events without the need for a user with specialized knowledge.

A video surveillance system according to one aspect of the present disclosure includes a storage unit that stores specific event data extracted from a plurality of feature data indicating the features of a subject captured in each of a plurality of first video data in response to a search query related to the features for searching for desired first video data, an analysis unit that generates analysis result data indicating the features of a subject captured in second video data, and a detection unit that detects the presence or absence of a specific event, which is a specific surveillance event captured in the second video data, based on the analysis result data and the specific event data.

The present invention makes it possible to detect specific events even without a user with specialized knowledge.

1 is a diagram illustrating an example of a video monitoring system according to a first embodiment of the present disclosure. FIG. 13 is a diagram illustrating an example of a search query table. FIG. 13 is a diagram illustrating an example of a frame-by-frame table. FIG. 13 is a diagram illustrating an example of a tracking ID table. 11 is a flowchart illustrating an example of a video search process. 11 is a flowchart illustrating an example of an adjustment process. FIG. 2 is a diagram illustrating another example of the video monitoring system according to the first embodiment of the present disclosure. 11 is a flowchart illustrating an example of a specific event detection process. 11 is a flowchart illustrating an example of a specific event determination process. 11 is a diagram for explaining an example of a specific event determination process. FIG. FIG. 2 is a diagram illustrating another example of the video monitoring system according to the first embodiment of the present disclosure. FIG. 11 is a diagram illustrating an example of a video monitoring system according to a second embodiment of the present disclosure. FIG. 11 is a diagram illustrating another example of a video monitoring system according to the second embodiment of the present disclosure. 13 is a flowchart illustrating another example of the specific event detection process.

Embodiments of the present disclosure will be described below with reference to the drawings.

The video surveillance system of this embodiment described below is a system that extracts and accumulates specific event data, which is data related to a specific event that is a specific surveillance event, from feature data that indicates the characteristics of a specific subject captured in the video data, and detects a specific event captured in the video data of the surveillance target based on the accumulated specific event data. Specific events are, for example, incidents such as crimes or accidents, events that require response by security, police, or emergency services, or events that are a sign of the occurrence of such incidents or events.

Figure 1 is a diagram showing a configuration for accumulating specific event data in a video surveillance system according to a first embodiment of the present disclosure. In Figure 1, the video surveillance system includes a video distribution device 1, a video processing device 2, an FDB (Feature Database) server 3, a user terminal 4, a video search device 5, and a specific event DB server 6. Each of the devices 1 to 6 is connected to each other so that they can communicate with each other, for example, via a communication network (not shown).

Video distribution device 1 is a device that distributes video data. In this embodiment, video distribution device 1 is a camera that acquires and distributes video data showing a specified monitoring location, but it may also be a recorder or a VMS (Video Management System). In this embodiment, the video data is moving image data consisting of multiple frames. Furthermore, video distribution device 1 distributes only the video data, or distributes the video data with meta information about the video data added to it. The meta information is, for example, a camera ID, which is identification information that identifies the video distribution device 1, and an acquisition date and time, which is the date and time when the video data was acquired.

The video processing device 2 is a device that performs a predetermined video analysis process on the video data distributed by the video distribution device 1. Specifically, the video processing device 2 has a video input unit 21, a video analysis unit 22, and an FDB registration unit 23.

The video input unit 21 accepts video data distributed by the video distribution device 1. In the example of FIG. 1, the video data is the first video data for generating specific event data.

The video analysis unit 22 performs a predetermined video analysis process on the video data received by the video input unit 21. The video analysis process includes a feature estimation process that estimates (acquires) the features of a predetermined subject that appears in the video data. In this embodiment, the predetermined subject is a person, but it may also be an animal or object other than a person.

The feature estimation process includes a person detection process that detects a person who is a specified subject, an attribute estimation process that estimates the attributes of the detected person detected in the person detection process, and a behavior estimation process that estimates the behavior of the detected person. In this case, the attributes and behavior of the detected person are the features. The attributes are external attributes (attributes determined from the appearance), such as age (age category), sex, and color of clothes. The behaviors are, for example, running and looking around (looking around). Note that the video analysis process includes processes for performing feature estimation processes, such as background recognition process that recognizes the background of the video data and skeleton estimation process that estimates the skeleton of the detected person. In addition, the feature estimation process is performed for each frame of the video data, and includes a tracking process that identifies the common person when a person is captured in multiple frames. In addition, the feature estimation process may include a process of calculating the image feature amount of an area in which a person is captured in a frame of the video data as a person image feature amount.

The FDB registration unit 23 registers in the FDB server 3 the video data received by the video input unit 21 and feature data indicating the results of the video analysis process performed by the video analysis unit 22 on the video data. The feature data is, for example, data that associates, for each frame of video data, a camera ID, an acquisition date and time, and feature information indicating the features of the person appearing in that frame. If meta-information is not attached to the video data, the FDB registration unit 23 may generate the camera ID and acquisition date and time. The feature information has multiple estimated items corresponding to each type of feature. The estimated items include behavior items that indicate features related to a person's behavior, attribute items that indicate features related to a person's attributes, and image feature items that indicate person image features. There may be multiple behavior items and attribute items.

The item value of an action item indicates whether or not an action corresponding to the action item has occurred. The item value of an attribute item indicates the attribute corresponding to the attribute item. If the attribute is an age category, the item value is, for example, "0" for age 15 or younger, "1" for ages 15-45, "2" for ages 45-60, and "3" for ages 60 or older.

The FDB server 3 is a server that has an FDB 31, which is a storage unit that stores video data and feature data.

The user terminal 4 is a terminal device operated by a user who uses the video surveillance system. The user terminal 4 has an input unit 41 and a display unit 42. The input unit 41 accepts various information such as a video search query, which is a search query (search condition) for searching desired video data from the user, and transmits it to the video search device 5. The display unit 42 accepts and displays various information such as a search result, which is a response to the video search query. The video search query includes search conditions related to, for example, a camera ID, acquisition date and time, and feature information.

The video search device 5 is a device that searches for video data from the FDB 31 based on a video search query from the user terminal 4. The video search device 5 has a video search unit 51 and a specific event DB registration unit 52.

The video search unit 51 searches the FDB 31 for video data that matches the video search query from the user terminal 4, and transmits the search results to the user terminal 4. Specifically, the video search unit 51 searches for video data that shows a tracked person who has characteristics that match the search conditions of the video search query, based on the characteristic data stored in the FDB 31.

The specific event DB registration unit 52 is a registration unit that extracts feature data corresponding to a video search query from the feature data stored in the FDB 31 of the FDB server 3 as specific event data related to a specific event and registers it in the specific event DB server 6. Specifically, the specific event DB registration unit 52 registers feature data corresponding to searched video data, which is video data searched by the video search unit 51 in response to a video search query, in the specific event DB server 6 as specific event data. The specific event data includes a frame-specific table that aggregates feature data for each frame, and a tracking ID-specific table that aggregates feature data for each tracking target person. The tracking ID-specific table is generated from the frame-specific table. The specific event DB registration unit 52 may also register a video search query in the specific event DB server 6. In this case, the specific event DB registration unit 52 may register, for example, in the form of a search query table that collects video search queries, or in the form of adding to the frame-specific table, or in both forms.

The specific event DB server 6 is a server having a specific event DB 61, which is a storage unit that stores video search queries and specific event data.

Figure 2 is a diagram showing an example of a search query table. The search query table 200 shown in Figure 2 is table information having each video search query as a record, and includes fields 201 to 204.

Field 201 stores the camera ID of the video distribution device 1 that acquired the video data. Field 202 stores the time period when the video data was acquired. Field 203 is provided for each action item of the characteristic information, and stores action characteristic designation information indicating whether or not the action item is designated as a search target. In the example of FIG. 2, the action characteristic designation information indicates "1" if an action item is designated, and indicates "0" if an action item is not designated. Field 204 is provided for each attribute item of the characteristic information, and stores attribute characteristic designation information indicating a designated value that designates one of the item values that the attribute item can take. Note that a video search query does not need to have values for all of fields 201 to 204.

Figure 3 shows an example of a frame-specific table included in feature event data. The frame-specific table 300 shown in Figure 3 is table information in which feature data is a record, and includes fields 301 to 308.

Field 301 stores the camera ID of the video distribution device 1 that acquired the searched video data. Field 302 stores a frame ID that identifies a frame of the searched video data. Field 303 stores the time when the frame was acquired. Field 304 stores a person ID that identifies a person to be tracked among the person IDs that identify each person appearing in the frame. Field 305 stores a tracking ID that identifies a person to be tracked appearing in the frame. Note that a person ID is an ID that identifies a person in a frame, and a tracking ID is an ID that is assigned to the same person between frames.

Field 306 is provided for each action item, and stores the item value and certainty of the action item for the tracked person appearing in the frame. Field 307 is provided for each attribute item, and stores the item value and certainty of the attribute item for the tracked person appearing in the frame. In this embodiment, fields 306 and 307 store the item value and certainty in the form of a numeric string represented as "item value, certainty" such as "1, 0.9". Field 308 stores the person image feature amount for the frame.

When a video search query is assigned to a frame-specific table, for example, a field for storing the video search query is added to the frame-specific table 300. For example, the field stores the items specified in the video search query (e.g., exercise items A and B, age category, etc.).

Figure 4 shows an example of a tracking ID table included in the characteristic event data. The tracking ID table 400 shown in Figure 4 includes fields 401 to 406.

Field 401 stores the camera ID of the video distribution device 1 that acquired video data having frames showing the tracked person. Field 402 stores the acquisition time of each frame showing the tracked person. Field 403 stores the tracking ID that identifies the tracked person. Field 404 is provided for each action item, and stores an aggregated value that aggregates the item values of the action items for the tracked person. Field 405 is provided for each attribute item, and stores an aggregated value that aggregates the item values of the attribute items for the tracked person. Field 406 stores an aggregated value that aggregates the person image features of each frame showing the tracked person.

Figure 5 is a flowchart illustrating an example of a video search process in a video surveillance system.

First, the input unit 41 of the user terminal 4 accepts a video search query from a user and transmits the video search query to the video search device 5. When the video search unit 51 of the video search device 5 receives the video search query from the user terminal 4, it searches the FDB server 3 for video data showing a tracked person who has characteristics matching the video search query (step S101). Then, the specific event DB registration unit 52 registers the video search query in the specific event DB 61 of the specific event DB server 6 as a record in the search query table (step S102).

The video search unit 51 transmits a list of searched video data, which is the searched video data, to the user terminal 4 as a search result. The display unit 42 of the user terminal 4 receives the search results and displays the search results (step S103). Note that the search results may include thumbnail images of the searched video data, etc.

The input unit 41 receives a selection request from the user to select one of the searched video data displayed in the search results, and transmits the selection request to the video search device 5. The video search device 5 transmits selected video data, which is the searched video data selected by the selection request from the user terminal 4, to the user terminal 4. The display unit 42 of the user terminal 4 receives the selected video data and displays (plays) the selected video data (step S104).

The specific event DB registration unit 52 of the video search device 5 identifies the person who most accurately matches the video search query from the selected video data as the person to be tracked (step S105). The specific event DB registration unit 52 extracts a tracking ID that identifies the target person from the feature data corresponding to the selected video data stored in the FDB 31 (step S106).

The specific event DB registration unit 52 extracts feature data including the extracted tracking ID (feature data corresponding to each of all frames showing the person to be tracked) from the FDB 31, and registers the feature data as a frame-specific table in the specific event DB 61 of the specific event DB server 6 (step S107).

Then, the specific event DB registration unit 52 generates a tracking ID table based on the frame table, registers it in the specific event DB 61 of the specific event DB server 6 (step S108), and ends the search process.

The item values of each estimated item in the tracking ID table are determined, for example, by one of the following three methods, methods 1 to 3.

In the first method, the specific event DB registration unit 52 sets, for each estimated item, the item value that appears most frequently among the item values in the frame-specific table whose confidence level is equal to or greater than the judgment threshold, as the item value in the tracking ID-specific table.

In the second method, the specific event DB registration unit 52 identifies the item value with the highest frequency of occurrence in the frame-specific table for each estimation item as the most frequent estimation value, and for behavior items, the most frequent estimation value is multiplied by the frequency of occurrence of the most frequent estimation value as the item value in the tracking ID-specific table, and for attribute items, the most frequent estimation value is the item value in the tracking ID-specific table. Here, the frequency of occurrence is the ratio of the number of frames corresponding to the estimation value to the total number of frames, which is the total number of frames in which the target person of the tracking ID appears. For example, if the total number of frames is 1000, the number of frames with age category "1" is 100, and the number of frames with age category "2" is 300, the frequency of occurrence of age category "1" is 100/1000 = 0.1, and the frequency of occurrence of age category "2" is 300/1000 = 0.3.

In the third method, the specific event DB registration unit 52 calculates, for each estimated item, a vector value that connects item values corresponding to each frame in the frame-specific table as an item value in the tracking ID-specific table. At this time, the specific event DB registration unit 52 may normalize the vector value to a vector value with a predetermined number of dimensions, for example by thinning out the elements of the vector value.

In the configuration and operation related to the search process described above, the specific event DB registration unit 52 registered feature data corresponding to the video search query as specific event data, but the feature data to be registered as specific event data may be specified directly by the user via the user terminal 4. In this case, the specific event DB registration unit 52 may provide the user terminal 4 with an interface for specifying feature data.

In addition, in the video search query, only the item value of the estimated item is specified, but the video search query may further include not only the item value of the estimated item but also a search threshold, which is a threshold for the certainty of the item value. In this case, for example, fields 203 and 204 of the search query table shown in FIG. 2 store a numeric string expressed as "specified value, search threshold" such as "1, 0.8". In addition, when the action item corresponding to action A (running) is "1, 0.8", the video search unit 51 searches for video data whose certainty is 0.8 or more and is determined to be "1". The search threshold may be adjusted by the specific event DB registration unit 52.

Figure 6 is a flowchart for explaining an example of an adjustment process for adjusting a search threshold. The adjustment process is executed when the video search unit 51 of the video search device 5 receives a video search query from the user terminal 4. Alternatively, the adjustment process may be executed when the video search unit 51 of the video search device 5 receives an adjustment instruction based on a user operation from the user terminal 4.

In the adjustment process, the specific event DB registration unit 52 first refers to the frame-specific table registered in the specific event DB 61 of the specific event DB server 6 (step S201). Then, for each estimated item included in the video search query, the specific event DB registration unit 52 obtains the confidence level of each item value of the estimated item from the frame-specific table (step S202).

The specific event DB registration unit 52 calculates the minimum value as a statistical value of the confidence of each item value for each estimated item (step S203). The statistical value is not limited to the minimum value, and may be, for example, the average value or a value obtained by subtracting the standard deviation from the average value.

The specific event DB registration unit 52 sends a request screen to the user terminal 4, requesting the user to select whether or not to adjust the search threshold. The display unit 42 of the user terminal 4 receives the request screen and displays the request screen (step S204). After that, when the input unit 41 receives an adjustment request to adjust the search threshold, it sends the adjustment request to the video search device 5. When the specific event DB registration unit 52 of the video search device 5 receives the adjustment request, it sets the search threshold of each item value to the minimum value of the above-mentioned confidence level for each estimated item (step S205), and ends the adjustment process. Note that the specific event DB registration unit 52 may automatically set the search threshold of the item value to the minimum value of the above-mentioned confidence level without displaying the request screen.

Figure 7 shows a configuration for detecting a specific event in a video surveillance system. The video surveillance system shown in Figure 7 differs from the video surveillance system shown in Figure 1 in that it has a specific event search device 7 instead of a video search device 5. However, the video surveillance system may be equipped with both the video search device 5 and the specific event search device 7, or the video search device 5 and the specific event search device 7 may be configured as the same physical device.

In the example of FIG. 7, the video data received by the video input unit 21 of the video processing device 2 from the video distribution device 1 is second video data to be detected to detect whether or not a specific event is captured, and the video analysis unit 22 performs video analysis processing on the second video data and transmits the processing result to the specific event search device 7 as analysis result data. The video analysis processing is the same as the processing described in FIG. 1 etc., and the analysis result data may be data similar to the feature data.

The specific event search device 7 is a detection device that detects whether a specific event is captured in video data. The specific event search device 7 has a specific event search unit 71 and an integrated display unit 72.

The specific event search unit 71 detects whether or not a specific event is captured in the video data based on the analysis result data, which is the result of the video analysis process performed on the video data by the video analysis unit 22 of the video processing device 2, and the specific event data stored in the specific event DB 61 of the specific event DB server 6.

When a specific event is detected by the specific event search unit 71, the integrated display unit 72 integrates alert information indicating the specific event with the video data and transmits the integrated information to the user terminal 4, thereby displaying the alert information and the video data on the display unit 42 of the user terminal 4. The alert information is integrated with the video data as, for example, pop-up information.

The specific event data stored in the specific event DB 61 may be added based on, for example, feature data and specific event data acquired by another video surveillance system and stored on the cloud, in addition to the data accumulated in the video surveillance system shown in FIG. 1. The video surveillance system may also upload feature data and specific event data to the cloud.

Figure 8 is a flowchart illustrating an example of a specific event detection process that detects a specific event using a video surveillance system.

First, when the video input unit 21 of the video processing device 2 receives video data (step S301), the video analysis unit 22 performs video analysis processing on the video data and transmits the analysis result data, which is the result of the video analysis processing, and the video data to the specific event search device 7 (step S302).

The specific event search unit 71 of the specific event search device 7 generates an event search query, which is a search query for searching for specific event data, based on the analysis result data from the video processing device 2. The specific event search unit 71 searches for specific event data stored in the specific event DB 61 based on the event search query (step S303).

The specific event search unit 71 analyzes the search results and determines whether a specific event is captured in the video data (step S304).

If the specific event is not captured, the integrated display unit 72 transmits the video data to the user terminal 4. The display unit 42 of the user terminal 4 receives the video data, displays the video data (step S305), and ends the process.

On the other hand, if a specific event is captured, the integrated display unit 72 generates alert information and transmits the video data and the alert information to the user terminal 4. The display unit 42 of the user terminal 4 receives the video data and the alert information, displays the video data and the alert information (step S306), and ends the process.

Figure 9 is a flowchart for explaining in more detail the specific event determination process, which is the process of steps S303 to S306 in Figure 8. Figure 10 is a diagram for explaining an example of the specific event determination process.

First, the specific event search unit 71 acquires analysis result data from the video processing device 2 (step S401). The analysis result data includes not only the processing result corresponding to the latest frame in the video data, but also the processing result linked to the tracking ID of each person appearing in the latest frame.

The specific event search unit 71 converts the analysis result data into data by tracking ID that aggregates, for each tracking ID, feature data related to the tracking target person of that tracking ID (step S402). The conversion method is the same as the method for converting a table by frame into a table by tracking ID. In FIG. 10, analysis result data 500 is shown as an example of analysis result data by tracking ID.

The specific event search unit 71 generates an event search query using as a search condition a vector consisting of the item values of the action item (item 501 in FIG. 10), which is the first item, from among the analysis result data by tracking ID, and searches the table by tracking ID stored in the specific event DB 61 (step S403). Here, the specific event search unit 71 calculates the similarity between the event search query and each record of the tracking table (table 550 in FIG. 10) in the action item, and sorts each record in descending order from the highest similarity. For example, the reciprocal of the distance between feature quantities such as Euclidean distance can be used as the similarity. In addition, when there are multiple action items, the specific event search unit 71 may calculate a statistical value such as the sum or average value of the similarity of each action item as the similarity between the event search query and each record of the tracking table.

The specific event search unit 71 removes records whose similarity is less than a predetermined similarity from the trace-based table, and extracts data whose similarity is equal to or greater than the predetermined similarity (step S404). If the number of extracted data is equal to or less than a set value, the specific event search unit 71 may extract data whose similarity is the highest from the trace-based table. The set value is, for example, 0.

The specific event search unit 71 analyzes the distribution of item values for each of the second items other than the action items (attribute items, camera ID, acquisition time, etc.) and calculates an evaluation value that evaluates the bias (localization) of the distribution of the item values (step S405). The evaluation value is, for example, a standard deviation.

The specific event search unit 71 extracts items whose evaluation value is equal to or greater than a predetermined evaluation value as biased items (step S406).

The specific event search unit 71 compares the most frequent item value in the tracking table for the biased item with the item value in the event search query (step S407).

If the item values match, the specific event search unit 71 determines that a specific event is captured in the video data and generates alert information (step S408). In this embodiment, if the item values match for all biased items, the specific event search unit 71 generates a warning alert as alert information indicating that there is a high probability that a specific event is captured, and if the item values match for some of the biased items, it generates a caution alert as alert information indicating that a specific event may be captured.

The specific event search unit 71 may store in the specific event DB 61 an association between the alert information and a specific event ID that identifies the video data determined to include a specific event. If the item values do not match for all bias items, the specific event search unit 71 may store in the specific event DB 61 low alert information, indicating that the specific event is unlikely to be included, in association with the specific event ID. A caution alert may be used instead of low alert information.

Figure 11 shows a configuration for displaying a list of specific events in a video surveillance system. The video surveillance system shown in Figure 11 differs from the video surveillance system shown in Figure 7 in that the specific event search device 7 includes a content generation unit 73 instead of a specific event search unit 71 and an integrated display unit 72. However, the specific event search device 7 may also include a specific event search unit 71, an integrated display unit 72, and a content generation unit 73.

The content generation unit 73 generates a list of specific event IDs stored in the specific event DB 61 as a specific event list, and displays an event selection screen on the display unit 42 for selecting one of the specific event IDs. Thereafter, when the input unit 41 selects one of the specific event IDs, the content generation unit 73 acquires video data corresponding to the selected specific event ID from the FDB 31 or the like, and transmits it to the user terminal 4, thereby displaying it on the display unit 42. The content generation unit 73 may also acquire related video data related to the video data, for example, from the FDB 31, the specific event DB 61, or the video distribution device 1 (in the case of a recorder), and display it on the display unit 42. The related video data is, for example, video data showing the same person as the person shown in the video data.

The specific event DB 61 may store the content (video data and related video data) generated by the content generation unit 73. This content may also be uploaded to the cloud or the like.

As described above, according to this embodiment, the specific event DB 61 stores specific event data extracted from a plurality of feature data indicating the features of the subject appearing in each of a plurality of first video data in response to a search query regarding features for searching for desired first video data. The video analysis unit 22 generates analysis result data indicating the features of the subject appearing in the second video data. The specific event search unit 71 detects whether or not a specific event, which is a specific surveillance event, is shown in the second video data based on the analysis result data and the specific event data. Therefore, whether or not a specific event is shown is detected based on past search results in which a user searched for a specific event that may be shown, so that it is possible to detect a specific event even without a user with specialized knowledge.

In addition, in this embodiment, the specific event search unit 71 compares the features indicated in the analysis result data with the features indicated in the subject-specific data (each record in the tracking ID-specific table), and detects whether or not a specific event is captured based on the comparison result. Therefore, since a specific event is detected based on the features of images previously searched, it becomes possible to more accurately detect whether or not a specific event is captured.

In addition, in this embodiment, the specific event search unit 71 extracts subject-specific data that indicates a feature in the first feature item that has a similarity to the feature indicated in the analysis result data that is equal to or greater than a predetermined similarity, and compares the feature indicated in the analysis result data with the feature indicated in each of the extracted subject-specific data. It is preferable that the first feature item is an action item. In this case, it becomes possible to more accurately detect whether or not a specific event is captured.

In addition, in this embodiment, if the evaluation value for evaluating the bias of each feature of the multiple subject-specific data in the second feature item is equal to or greater than a predetermined value, the specific event search unit 71 compares the most frequent value of the value of that item with the value of that item in the analysis result data. Then, if the values of each item match, the specific event search unit 71 detects that a specific event is captured. It is preferable that the first item includes an attribute item. In this case, it becomes possible to detect with greater accuracy whether or not a specific event is captured.

In addition, in this embodiment, the specific event DB registration unit 52 extracts specific event data from the feature data in response to the video search query and registers the specific event data in the specific event DB 61. This makes it possible to update the specific event data, making it possible to more accurately detect whether or not a specific event is captured.

In addition, in this embodiment, the specific event DB registration unit 52 extracts feature data selected by the user from among feature data that matches the video search query as specific event data. Therefore, whether or not a specific event is captured is detected based on past search results in which the user performed searches that were deemed highly likely to contain a specific event, making it possible to more accurately detect whether or not a specific event is captured.

In addition, in this embodiment, if a specific event is captured, the integrated display unit 72 displays alert information indicating the specific event and the second video data. This makes it possible to properly notify the user that a specific event is captured.

Second Embodiment
In this embodiment, a video surveillance system in which a specific event is identified using machine learning will be described.

Figure 12 is a diagram showing the configuration for performing machine learning in the video surveillance system of this embodiment. The video surveillance system shown in Figure 12 is different from the video surveillance system shown in Figure 1 in that it further includes a machine learning device 8. Note that the machine learning device 8 may be configured as the same device physically as the video search device 5.

The machine learning device 8 constructs a specific event model, which is a machine learning model for detecting a specific event, based on the specific event data stored in the specific event DB 61 of the specific event DB server 6. The machine learning device 8 has a specific event learning unit 81 and a specific event model unit 82.

The specific event learning unit 81 uses the specific event data stored in the specific event DB 61 of the specific event DB server 6 as learning data to construct a specific event model that classifies the analysis result data of the video analysis unit 22 according to the presence or absence of a specific event. For example, the specific event learning unit 81 constructs a specific event model by performing multi-label machine learning using the camera ID and acquisition time of the specific event data as data, and the item values (estimated classes) and confidence levels of the action items and attribute items as teacher labels. The machine learning method is not particularly limited, but may be, for example, multiple regression analysis.

The specific event model unit 82 holds the specific event model constructed by the specific event learning unit 81.

Figure 13 is a diagram showing a configuration for detecting a specific event using a machine learning model in the video surveillance system of this embodiment. The video surveillance system shown in Figure 13 differs from the video surveillance system shown in Figure 12 in that it has a specific event search device 7 and a machine learning device 8 instead of the video search device 5. The specific event search device 7 differs from the specific event search device 7 shown in Figure 7 in that it has a specific event estimation unit 74 instead of the specific event search unit 71. Note that in Figure 13, only the specific event model unit 82 of the machine learning device 8 is shown.

The specific event estimation unit 74 is a detection unit that uses the specific event model set in the specific event model unit 82 to detect whether or not a specific event is captured in the video data based on the analysis result data and the specific event data.

Figure 14 is a flowchart illustrating an example of a specific event detection process for detecting a specific event by the video surveillance system of this embodiment.

First, when the video input unit 21 of the video processing device 2 receives video data (step S501), the video analysis unit 22 performs video analysis processing on the video data and transmits the analysis result data, which is the result of the video analysis processing, and the video data to the specific event search device 7 (step S502).

The specific event estimation unit 74 of the specific event search device 7 inputs the analysis result data from the video processing device 2 into the specific event model held in the specific event model unit 82 of the machine learning device 8 (step S503). The specific event estimation unit 74 checks the output of the specific event model and determines whether or not a specific event is captured in the video data (step S504).

If the specific event is not captured, the integrated display unit 72 transmits the video data to the user terminal 4. The display unit 42 of the user terminal 4 receives the video data, displays the video data (step S505), and ends the process.

On the other hand, if a specific event is captured, the integrated display unit 72 generates alert information and transmits the video data and the alert information to the user terminal 4. The display unit 42 of the user terminal 4 receives the video data and the alert information, displays the video data and the alert information (step S506), and ends the process.

The user who has checked the alert information may determine whether or not a specific event is captured in the video data, and if the specific event is not captured, may input this to the input unit 41. In this case, the specific event estimation unit 74 may update the specific event data stored in the specific event DB 61 accordingly. In this case, the learning data used by the specific event learning unit 81 for machine learning is updated, making it possible to update the specific event model.

Each of the devices 1 to 8 described above may be configured, for example, by a computer system equipped with a processor (computer) and memory (both not shown). In this case, the components and functions of each of the devices 1 to 8 are realized, for example, by the processor reading a computer program and executing the read computer program. The computer program can be recorded on a computer-readable recording medium such as memory. Furthermore, each of the devices 1 to 8 may have an input/output device for inputting and outputting information, and an auxiliary storage device for storing information, as necessary.

The above-described embodiments of the present disclosure are illustrative examples of the present disclosure, and are not intended to limit the scope of the present disclosure to only those embodiments. Those skilled in the art may implement the present disclosure in various other forms without departing from the scope of the present disclosure.

1: Video distribution device 2: Video processing device 3: FDB server 4: User terminal 5: Video search device 6: Specific event DB server 7: Specific event search device 8: Machine learning device 21: Video input unit 22: Video analysis unit 23: FDB registration unit 31: FDB 41: Input unit 42: Display unit 51: Video search unit 52: Specific event DB registration unit 61: Specific event DB 71: Specific event search unit 72: Integrated display unit 73: Content generation unit 74: Specific event estimation unit 81: Specific event learning unit 82: Specific event model unit

Claims (11)

a storage unit that stores specific event data extracted from a plurality of feature data indicating features of a subject captured in each of a plurality of first video image data in response to a search query related to the features for searching for a desired one of the first video image data;
an analysis unit that generates analysis result data indicating characteristics of a subject captured in the second video data;
a detection unit that detects whether or not a specific event that is a specific monitoring event is captured in the second video data based on the analysis result data and the specific event data ,
the specific event data includes, for each subject having the characteristic matching the search query, a plurality of subject-specific data summarizing characteristics of a subject identical to the subject in the first video data;
The detection unit compares the features indicated in the analysis result data with the features indicated in each of the plurality of subject-specific data, and detects whether or not the specific event is captured based on the comparison result;
A video surveillance system in which the detection unit extracts the subject-specific data that indicates, in a first item of the features, a feature whose similarity to the feature indicated in the analysis result data is equal to or greater than a predetermined similarity, and compares the feature indicated in the analysis result data with the feature indicated in each of the extracted subject-specific data . The video surveillance system according to claim 1 , wherein the first item indicates a characteristic related to the behavior of the subject. a storage unit that stores specific event data extracted from a plurality of feature data indicating features of a subject captured in each of a plurality of first video image data in response to a search query related to the features for searching for a desired one of the first video image data;
an analysis unit that generates analysis result data indicating characteristics of a subject captured in the second video data;
a detection unit that detects whether or not a specific event that is a specific monitoring event is captured in the second video data based on the analysis result data and the specific event data,
the specific event data includes, for each subject having the characteristic matching the search query, a plurality of subject-specific data aggregating characteristics of a subject identical to the subject in the first video data;
The detection unit compares the features indicated in the analysis result data with the features indicated in each of the plurality of subject-specific data, and detects whether or not the specific event is captured based on the comparison result;
In a video surveillance system, when an evaluation value evaluating the bias of each feature of the plurality of subject-specific data in a second item of the features is equal to or greater than a predetermined value, the detection unit compares the most frequent value among the values of the item with the value of the item of the analysis result data. The video surveillance system according to claim 3 , wherein the detection unit detects that the specific event is captured in the video when values of the respective items match. The video surveillance system according to claim 3 , wherein the second item indicates a feature related to an attribute of the subject. A learning unit that uses the specific event data as learning data to construct a specific event model that classifies the analysis result data according to the presence or absence of the specific event,
The video surveillance system according to claim 1 , wherein the detection unit detects whether or not the specific event is captured by using the specific event model. a storage unit that stores specific event data extracted from a plurality of feature data indicating features of a subject captured in each of a plurality of first video image data in response to a search query related to the features for searching for a desired one of the first video image data;
an analysis unit that generates analysis result data indicating characteristics of a subject captured in the second video data;
a detection unit that detects whether or not a specific event that is a specific monitoring event is captured in the second video data based on the analysis result data and the specific event data;
A registration unit that extracts the specific event data from the feature data in response to the search query and registers the specific event data in the storage unit. The video surveillance system of claim 1, further comprising an integrated display unit that displays alert information indicating the specific event and the second video data when the specific event is captured. A video monitoring method using a video monitoring system, comprising:
storing specific event data extracted from a plurality of feature data indicating features of a subject appearing in each of a plurality of first video image data in response to a search query relating to the features for searching for a desired one of the first video image data;
generating analysis result data indicative of characteristics of a subject captured in the second image data;
Detecting whether or not a specific event, which is a specific monitoring event, is captured in the second video data based on the analysis result data and the specific event data ;
the specific event data includes, for each subject having the characteristic matching the search query, a plurality of subject-specific data summarizing characteristics of a subject identical to the subject in the first video data;
In the detection, a feature indicated by the analysis result data is compared with a feature indicated by each of the plurality of subject-specific data, and based on the comparison result, it is detected whether or not the specific event is captured;
In the detection, the subject-specific data that indicates a feature in a first item of the feature having a similarity to the feature indicated in the analysis result data that is equal to or greater than a predetermined similarity is extracted, and the feature indicated in the analysis result data is compared with the feature indicated in each of the extracted subject-specific data . A video monitoring method using a video monitoring system, comprising:
storing specific event data extracted from a plurality of feature data indicating features of a subject appearing in each of a plurality of first video image data in response to a search query related to the features for searching for a desired one of the first video image data;
generating analysis result data indicative of characteristics of a subject captured in the second image data;
Detecting whether or not a specific event, which is a specific monitoring event, is captured in the second video data based on the analysis result data and the specific event data;
the specific event data includes, for each subject having the characteristic matching the search query, a plurality of subject-specific data aggregating characteristics of a subject identical to the subject in the first video data;
In the detection, a feature indicated by the analysis result data is compared with a feature indicated by each of the plurality of subject-specific data, and based on the comparison result, it is detected whether or not the specific event is captured;
In the detection, when an evaluation value evaluating the bias of each feature of the plurality of subject-specific data in a second item of the features is equal to or greater than a predetermined value, the most frequent value among the values of the item is compared with the value of the item of the analysis result data, in this video surveillance method. A video monitoring method using a video monitoring system, comprising:
storing specific event data extracted from a plurality of feature data indicating features of a subject appearing in each of a plurality of first video image data in response to a search query relating to the features for searching for a desired one of the first video image data;
generating analysis result data indicative of characteristics of a subject captured in the second image data;
Detecting whether or not a specific event, which is a specific monitoring event, is captured in the second video data based on the analysis result data and the specific event data;
The specific event data is extracted from the feature data in response to the search query and registered.

Images