JP7637381B2 - Video image analysis support device, video image analysis support system, and program - Google Patents

Description

The present invention relates to a video image analysis support device, a video image analysis support system, and a program.

In order to monitor whether any unauthorized activity is taking place in a certain space, surveillance cameras that capture the space may be installed. The images captured by the surveillance cameras are recorded. In this way, when unauthorized activity is suspected, an administrator can play back the recorded video data to confirm whether unauthorized activity has occurred.

Patent Document 1 discloses a technology that determines the state from images captured by a surveillance camera, monitors whether any fraudulent activity is occurring, and makes an announcement according to the state, as well as a technology that allows fraudulent activity to be confirmed by playing back recorded video data.

JP 2010-124186 A

However, video data captured by a surveillance camera often has a long recording time, and visually checking the video data to identify events such as fraudulent activity places a heavy burden on the administrator.
An object of the present invention is to provide a video analysis support device, a video analysis support system, and a program that can assist managers in easily checking whether operations are being performed correctly from surveillance camera footage.

According to a first aspect of the present invention, a video analysis support device includes an acquisition unit that acquires video data captured by an imaging device that captures an image of a space in which a task is performed, a setting unit that accepts settings of business rules indicating detection conditions that define a state in which the task is performed correctly or a state in which the task is not performed correctly in a frame of the video data, a determination unit that determines whether or not a frame that satisfies the detection conditions is present in the video data, and a generation unit that generates index data that indicates a frame that is defined by the detection conditions.

According to a second aspect of the present invention, a video analysis support system includes a video analysis support device according to the above aspect, and a playback unit that plays back the video data based on the index data.

According to a third aspect of the present invention, the program causes a computer to execute the steps of acquiring video data captured by an imaging device that captures an image of a space in which a task is performed, accepting settings of task rules indicating detection conditions that define a state in which the task has been performed correctly or an improper state in a frame of the video data, determining whether or not a frame that satisfies the detection conditions exists in the video data, and generating index data that indicates a frame that meets the detection conditions.

According to at least one of the above aspects, it is possible to assist managers in easily checking from surveillance camera footage whether operations are being performed correctly.

1 is a configuration diagram of a video analysis support system according to a first embodiment. FIG. 2 is a schematic block diagram showing a configuration of a video server according to the first embodiment. FIG. 4 is a diagram showing a configuration of an analysis result file according to the first embodiment. FIG. 2 is a schematic block diagram showing a configuration of a client terminal according to the first embodiment. 4 is a flowchart showing a method for setting a business rule by the video analysis support system according to the first embodiment. FIG. 13 is a diagram illustrating an example of an analysis target selection screen according to the first embodiment. FIG. 11 is a diagram illustrating an example of a business rule setting screen according to the first embodiment. 5 is a flowchart showing an analysis processing method performed by the video server according to the first embodiment. 4 is a flowchart showing a video data playback method performed by the video analysis support system according to the first embodiment. FIG. 4 is a diagram showing an example of a video selection screen according to the first embodiment. FIG. 4 is a diagram showing an example of a playback screen according to the first embodiment; FIG. 1 is a schematic block diagram illustrating a configuration of a computer according to at least one embodiment.

First Embodiment
Configuration of video analysis support system
Hereinafter, the embodiments will be described in detail with reference to the drawings.
1 is a configuration diagram of a video analysis support system 1 according to a first embodiment. The video analysis support system 1 supports an administrator in monitoring for the presence or absence of fraudulent activities. Specifically, the video analysis support system 1 assigns indexes to points in video data captured by a surveillance camera 11 that may indicate fraudulent activities, and plays the video data based on the indexes. This allows the administrator to focus on monitoring points in the video data that may indicate fraudulent activities.

The video analysis support system 1 comprises a surveillance camera 11, a video server 13, and a client terminal 15. The surveillance camera 11 is installed to capture images of the space where business is conducted, i.e., the space to be monitored. For example, the surveillance camera 11 is installed in a bank branch. The video server 13 stores the videos captured by the surveillance camera 11. The video server 13 analyzes the video data and generates index data that indicates points in the video data that may be potential fraudulent activities. The client terminal 15 plays the video data based on the index data generated by the video server 13. The video server 13 and the client terminal 15 are installed in the bank's headquarters. The video server 13 is an example of a video analysis support device.

FIG. 2 is a schematic block diagram showing the configuration of the video server 13 according to the first embodiment.
The video server 13 includes an acquisition unit 131 , a video storage unit 132 , a detection unit 133 , a setting unit 134 , a setting storage unit 135 , a selection unit 136 , a determination unit 137 , a generation unit 138 , and an output unit 139 .

The acquisition unit 131 acquires moving image data from the surveillance camera 11. The acquisition unit 131 records the acquired moving image data in the moving image storage unit 132.
The video storage unit 132 stores the video data acquired by the acquisition unit 131 in association with an analysis result file indicating an analysis result of the video data. FIG. 3 is a diagram showing the configuration of the analysis result file according to the first embodiment. As shown in FIG. 3, the analysis result file includes video data information, detection result data, index data, and version data. The video data information is metadata of the video data to be analyzed. The video data information includes the format, file name, file path, file size, frame rate, and number of frames of the video data. The detection result data indicates information of an object detected from the video data. The detection result data includes the frame number where the object was detected, the type of the detected object, the reliability of the detection, the coordinates of the area where the object was detected (the upper left coordinate position of the area), and the size (height and width) of the area where the object was detected. The index data indicates an index of a frame where a process according to a business rule was detected. The index data includes the frame number where a process according to a business rule was detected, a message related to the detected process, a recording type indicating whether the index was added manually (M: Manual) or automatically (A: Automatic), and a warning level. The warning level is indicated by three types: "Level 0" indicating that a harmless process was detected, "Level 1" indicating that a suspected fraudulent process was detected, and "Level 2" indicating that a fraudulent process was detected. The version data is data indicating the version of the analysis result file. The version data is updated every time a value is added, changed, or deleted in the analysis result file.

The detection unit 133 detects an object from the video data acquired by the acquisition unit 131, and outputs detection result data indicating the frame in which the object was detected, the type of the detected object, the area in which the object was detected, and the reliability of the detection. Specifically, the detection unit 133 inputs each frame of the video data to a trained object detection model, and performs object detection processing for each frame. The object detection model is a model trained to output the coordinates of a bounding box containing an object included in the image and the type of object included in each bounding box when an image is input. The object detection model is trained using a learning dataset consisting of input samples, which are frames of video data, and output samples, which are combinations of bounding boxes surrounding objects included in the input samples and object types. The type of object to be detected is selected in advance in consideration of the business content and is likely to be a detection target. The detection result data indicating the detection result of the object is recorded in the video storage unit 132 in association with the video data of the detection target.

The setting unit 134 accepts the setting of business rules indicating the start and end conditions of a business and the detection conditions of fraudulent activity for each space and business to be monitored. The start and detection conditions are represented by the type of object to be detected and the detection state of the object. The detection state is selected from whether or not the object has been detected, whether or not the state in which the object has been detected continues, and whether or not the state in which the object has not been detected continues. The detection state may include the specification of an area to be detected in a frame of video data. The end condition may be represented by the detection state of the object, or may be represented by the elapsed time since the start condition was satisfied. The business rules set by the setting unit 134 are recorded in the setting storage unit 135 in association with the business rule name.

The selection unit 136 accepts, for each piece of video data stored in the video storage unit 132, a selection of a business rule to be used for monitoring the video data.
The determination unit 137 determines whether or not each frame of the video data satisfies the detection condition of the business rule based on the analysis result of the video data by the detection unit 133 and the business rule associated with the video data. The determination unit 137 identifies the frame number of the frame that satisfies the detection condition.

The generating unit 138 generates index data that associates the frame number identified by the determining unit 137 with the detection condition detected in the frame indicated by the frame number.
The output unit 139 outputs the analysis result file stored in the video storage unit 132 to the client terminal 15 .

FIG. 4 is a schematic block diagram showing the configuration of the client terminal 15 according to the first embodiment.
The client terminal 15 includes a display control unit 151 , a request unit 152 , a playback unit 153 , and an editing unit 154 .
The display control unit 151 controls the display on the display based on information received from the video server 13 .
The request unit 152 transmits a request based on the user's input to the video server 13 .
The playback unit 153 plays back the video data acquired from the video server 13. The playback unit 153 accepts an index designation based on the index data from the user, thereby designating a playback position in the video data.
The editing unit 154 edits the index data of the analysis result file stored in the video server 13. Specifically, the editing unit 154 adds an index indicating the playback position of the video data being played back by the playback unit 153 to the index data.

How to set up business rules
5 is a flowchart showing a method for setting business rules by the video analysis support system 1 according to the first embodiment. The administrator operates the client terminal 15 to access the video server 13. The request unit 152 of the client terminal 15 transmits a request for rule information to the video server 13 (step S1). When the output unit 139 of the video server 13 receives the request for rule information from the client terminal 15, it transmits to the client terminal 15 the file name of the video data stored in the video storage unit 132, the analysis result file, and the business rule stored in the setting storage unit 135 (step S2).

The display control unit 151 of the client terminal 15 generates an analysis target selection screen based on the file name of the video data received from the video server 13 and the analysis result file, and displays it on the display (step S3). FIG. 6 is a diagram showing an example of the analysis target selection screen according to the first embodiment. The analysis target selection screen displays a video list i01 for selecting an analysis target from multiple video data, and a setting start button i02 for proceeding to the setting of business rules for the selected video data. The video list i01 includes a check box i011 for accepting a selection of whether or not to select the video data as an analysis target, a mark i012 indicating the analysis result, and a file name i013. The mark i012 indicates whether or not the analysis of the associated video data has been completed, and if completed, whether or not fraudulent activity or a behavior requiring confirmation has been detected. Note that if both fraudulent activity and a behavior requiring confirmation have been detected, the mark i012 is displayed in a manner indicating that fraudulent activity has been detected.

The request unit 152 of the client terminal 15 determines whether the setting start button i02 has been pressed by the user (step S4). If the setting start button i02 has not been pressed (step S4: NO), the request unit 152 returns the process to step S4 and waits for the setting start button i02 to be pressed. On the other hand, if the setting start button i02 has been pressed (step S4: YES), the check box i011 associated with the video data to be analyzed is turned ON on the analysis target selection screen, and when the button i02 is pressed, the display control unit 151 of the client terminal 15 generates a business rule setting screen based on the file name and business rule of the video data received from the video server 13 in step S2, and displays it on the display (step S5). FIG. 7 is a diagram showing an example of a business rule setting screen according to the first embodiment. The business rule setting screen includes a video list i11, a rule list i12, a rule display pane i13, an add rule button i14, and an analysis start button i15.

The video list i11 displays the file name of the video data related to the selected check box i011. The video list i11 is configured to allow selection of one or more files from among a plurality of video data. The rule list i12 displays the names of business rules received from the video server 13. A check box is associated with each business rule name, and a selection is accepted as to whether or not to set the business rule to the video data selected in the video list i11. When a business rule in the rule list i12 is clicked, the contents of the business rule are displayed in the rule display pane i13.
The add rule button i14 accepts the addition of a business rule when pressed. The start analysis button i15, when pressed, executes an analysis based on the associated business rule for each video data to be analyzed.

The request unit 152 of the client terminal 15 determines whether the user has pressed the Add Rule button i14 (step S6). If the Add Rule button i14 has been pressed (step S6: YES), the display control unit 151 displays a rule creation screen on the display (step S7). The rule creation screen includes a form for accepting input of the rule name, start condition, end condition, detection condition, and detection accuracy.

The request unit 152 accepts settings of the type of object to be detected from video data and the detection state as the start condition of the business rule. The type of object is selected from a list of types detectable by the object detection model used by the detection unit 133 of the video server 13. The detection state is selected from either a change from an undetected state to a detected state of the object, or a change from a detected state to an undetected state of the object. In addition, the request unit 152 is capable of accepting settings of a frame area as an option for the detection state.
The request unit 152 receives a setting of the detection state of the object or the elapsed time from the point in time when the start condition is satisfied as the end condition of the business rule.

The request unit 152 accepts settings of the type of object to be detected from the video data and the detection state as the detection condition of the business rule. The detection state related to the detection condition is selected from any one of the following: the object has changed from an undetected state to a detected state, the object has changed from a detected state to an undetected state, the object has not changed from an undetected state to a detected state until the end condition is satisfied, and the object has not changed from a detected state to an undetected state until the end condition is satisfied. In addition, the request unit 152 can accept settings of a frame area as an option for the detection state.
The request unit 152 accepts an integer greater than 0 and equal to or less than 100 as the detection accuracy of an object in a business rule. The detection accuracy may be expressed by a numerical value between 0 and 1.0. The detection accuracy may be a numerical value input directly or input using a slider. In this case, the input range of the slider may be set to, for example, 70 to 100, allowing the specification of a finer numerical value for the detection accuracy to be accepted.

When the request unit 152 accepts input of a business rule from a user, it transmits the input business rule to the video server 13 (step S8). When the setting unit 134 of the video server 13 receives the business rule from the client terminal 15, it associates the received business rule with the business rule name and records it in the setting storage unit 135 (step S9). The video analysis support system 1 returns the process to step S2 and transmits information including the added business rule to the client terminal 15.

If the Add Rule button i14 has not been pressed on the business rule setting screen (step S6: NO), the request unit 152 determines whether the Start Analysis button i15 has been pressed by the user (step S10). If the Start Analysis button i15 has not been pressed (step S10: NO), the request unit 152 returns the process to step S6 and monitors whether a button has been pressed.

When the analysis start button i15 is pressed (step S10: YES), the request unit 152 sends a video analysis request to the video server 13 (step S11). The analysis request includes information indicating the relationship of each video displayed in the video list i11 to the business rule indicated by the check box that is ON in the rule list i12 for that video. Upon receiving the analysis request, the selection unit 136 of the video server 13 accepts the selection of the business rule to be applied to each video (step S12). The video server 13 analyzes each video stored in the video storage unit 132 according to the selected business rule (step S13).

FIG. 8 is a flowchart showing an analysis processing method performed by the video server 13 according to the first embodiment.
When the video server 13 starts the analysis based on the business rules, it selects each video data recorded in the video storage unit 132 (step S31), and executes the following processes from step S32 to step S39.

The detection unit 133 of the video server 13 determines whether or not the video storage unit 132 contains detection result data associated with the video data selected in step S31 (step S32). If there is no detection result (step S32: NO), the detection unit 133 uses the object detection model to detect an object from each frame of the video data selected in step S31 (step S33). The detection unit 133 records the detection result data indicating the detection result in the video storage unit 132 in association with the video data of the detection target (step S34). The detection result data includes, for each frame, the frame number, the type of object detected in the frame, the coordinates of the bounding box surrounding the object, and the reliability, which are associated with each other. The generation unit 138 initializes the index data associated with the video data selected in step S31 (step S35). At this time, the generation unit 138 deletes indexes (indexes with record type A) that were added through analysis based on business rules from among the index data, but does not delete indexes (indexes with record type M) that were added manually by the administrator.

Next, the determination unit 137 selects each of the business rules selected in step S10 as the business rule to be applied to the video data selected in step S31 (step S36), and executes the processing in steps S37 to S39 below.

Based on the analysis result file of the video data selected in step S31, the determination unit 137 identifies a frame (start frame) that satisfies the start condition indicated by the business rule selected in step S36 and a frame (end frame) that satisfies the end condition indicated by the business rule selected in step S36 (step S37). Between the identified start frame and end frame, the determination unit 137 identifies a frame that satisfies the detection condition indicated by the business rule selected in step S36 and a frame that no longer satisfies the detection condition (step S38). Note that, if the determination results of whether or not the start condition, end condition, or detection condition are satisfied are different for a predetermined number of frames or less (for example, one frame) among consecutive frames, the determination unit 137 regards the frame as a detection miss or a false detection, and aligns the determination result of the corresponding frame with the determination results of the other frames. For example, if the detection condition is satisfied in 19 frames out of 20 consecutive frames and the detection condition is not satisfied in only one frame, the determination unit 137 treats the one frame as satisfying the detection condition. Note that the number of frames that are regarded as a detection miss or a false detection can be changed by an administrator's operation.

For example, when the detection condition indicates that an object has been detected, the determination unit 137 identifies the first frame that indicates that an object has been detected as the frame that satisfies the detection condition, whereas when the detection condition indicates that an object has been detected, and there is no frame in which an object has been detected between the start frame and the end frame, the determination unit 137 identifies the end frame as the frame in which the detection condition is no longer satisfied.
When the detection condition indicates that the object is no longer detected, the determination unit 137 identifies the first frame that indicates a state in which the object is not detected as a frame that satisfies the detection condition.On the other hand, when the detection condition indicates that the object is no longer detected, and there is no frame in which the object is not detected between the start frame and the end frame, the determination unit 137 identifies the end frame as a frame in which the detection state is no longer satisfied.
When the detection condition indicates that the state in which the object is detected continues, the determination unit 137 identifies the first frame in which the object is not detected as the frame in which the detection condition is no longer satisfied, whereas when the detection condition indicates that the state in which the object is detected continues and there is no frame in which the object is not detected between the start frame and the end frame, the determination unit 137 identifies the end frame as the frame in which the detection state is satisfied.
When the detection condition indicates that the state where the object is not detected continues, the determination unit 137 identifies the first frame showing the state where the object is detected as the frame where the detection condition is no longer satisfied. On the other hand, when the detection condition indicates that the state where the object is not detected continues and there is no frame where the object is detected between the start frame and the end frame, the determination unit 137 identifies the end frame as the frame where the detection condition is satisfied.

The generation unit 138 adds indices representing the frames identified in steps S37 and S38 to the index data initialized in step S35 (step S39). As shown in FIG. 3, each index indicates a frame number, a message representing the corresponding state, a record type, and a warning level. For example, in a frame that satisfies a detection condition indicating fraudulent activity, the generation unit 138 sets the warning level to level 1 if the reliability of the detected object is lower than a reference value, and sets the warning level to level 2 if the reliability is equal to or higher than the reference value. The reference value may be, for example, a value obtained by adding a predetermined value to the detection accuracy set in the business rules.

This allows the video server 13 to generate index data for each video data, recording the indexes of frames that correspond to the business rule selected in step S12.

"Method for playing video data"
9 is a flowchart showing a method for playing video data by the video analysis support system 1 according to the first embodiment. The administrator operates the client terminal 15 to access the video server 13. The request unit 152 of the client terminal 15 transmits a request for video information to the video server 13 (step S51). When the output unit 139 of the video server 13 receives the request for video information from the client terminal 15, it transmits to the client terminal 15 the file name of the video data stored in the video storage unit 132 and index data associated with the video data (step S52).

The display control unit 151 of the client terminal 15 generates a video selection screen based on the file name and index data of the video data received from the video server 13, and displays it on the display (step S53). FIG. 10 is a diagram showing an example of the video selection screen according to the first embodiment. On the video selection screen, a file name i31 and a mark i32 indicating an analysis result are displayed for each video data. The video selection screen is configured to allow one of a plurality of video data to be selected. When one of the file names i31 displayed on the video selection screen is selected, a playback screen of the selected video data is displayed. The mark i32 indicates whether the analysis of the associated video data has been completed, and if completed, whether fraudulent activity has been detected. In other words, when there is no index data received from the video server 13, the display control unit 151 attaches a mark i32 indicating that the analysis of the video data has not been completed. If index data received from the video server 13 exists and the analysis result file contains one or more pieces of warning level 1 or 2, the display control unit 151 attaches a mark i32 indicating that fraudulent activity has been detected. If index data received from the video server 13 exists and all warning levels are 0, the display control unit 151 attaches a mark i32 indicating that fraudulent activity has not been detected.

In addition, in other embodiments, the display control unit 151 may display the mark i32 indicating whether or not there is fraudulent activity separately for cases where there is obvious fraud (when there is something with a warning level of 2) and cases where there is suspicion and needs to be checked (when there is no thing with a warning level of 2 and there is something with a warning level of 1).

When the request unit 152 of the client terminal 15 accepts the selection of one piece of video data from the user, it sends a request for the selected video data to the video server 13 (step S54). The acquisition unit 131 of the video server 13 acquires the video data from the video storage unit 132 and sends it to the client terminal 15 (step S55).

The playback unit 153 generates a playback screen for the video data and displays it on the display (step S56). FIG. 9 is a diagram showing an example of the playback screen according to the first embodiment. On the playback screen, a playback pane i51 for the video data, an index pane i52, a control pane i53, and an index edit button i54 are displayed.

The playback pane i51 displays the video data distributed from the video server 13. When an object is detected in the displayed frame, the playback pane i51 displays a bounding box surrounding the object, and the type and reliability of the object. The bounding box can be displayed/hidden by the administrator. The index pane i52 displays a list of indexes indicated by index data associated with the video data displayed in the playback pane i51. In the index pane i52, the index is represented by a combination of a playback position and a message. In addition, in the index pane i52, the index is represented by a different color depending on the warning level. This allows the administrator to easily recognize a part of the video data where there is a high possibility of fraudulent activity or a part where fraudulent activity is suspected. The control pane i53 displays buttons for controlling the playback of the video data displayed in the playback pane i51. Specifically, the control pane i53 displays a seek bar indicating the playback position, a back skip button for moving the playback position back a predetermined time, a stop button for stopping playback, a play button for starting playback, a skip button for moving the playback position forward a predetermined time, and the like. Pressing the index edit button i54 edits the index data.

The playback unit 153 receives the video data selected in step S54 from the video server 13 and plays it in the playback pane i51 (step S57). The playback unit 153 determines whether or not one of the indexes displayed in the index pane i52 has been pressed (step S58). If an index has been pressed (step S58: YES), the playback unit 153 moves the playback position of the video data being played to the playback position associated with the pressed index (step S59). This allows the administrator to easily check noteworthy parts of the video data.

If the index has not been pressed (step S58: NO), the editing unit 154 determines whether the index edit button i54 has been pressed (step S60). If the index edit button i54 has been pressed (step S60: YES), the playback unit 153 stops the playback of the video data displayed in the playback pane i51 at the current playback position. When the editing unit 154 accepts a message input from the administrator, it sends an index addition request including the current playback position and the input message to the video server 13 (step S61). The generation unit 138 of the video server 13 updates the index data based on the received index addition request (step S62). This allows the administrator to supplement information such as omissions in the analysis by the video server 13.

Furthermore, when the index edit button i54 is pressed in a frame where an index already exists, the editing unit 154 accepts a change or deletion of the contents of that index. For example, the administrator may visually check a frame in which the warning level is 1, i.e., indicating suspected fraudulent activity, and when the administrator determines whether the activity detected in that frame is fraudulent or not, edit the index using the following procedure. The administrator presses the index to be edited, moves the playback position, and then presses the index edit button i54. This causes the editing unit 154 to accept editing of the pressed index. If fraudulent activity is confirmed, the administrator changes the warning level of that index to 2, and if it is confirmed that there is no fraudulent activity, the administrator changes the warning level of that index to 0.

The request unit 152 determines whether the playback screen has been closed (step S63). If the playback screen has not been closed (step S63: NO), the process returns to step S55 and playback of the video data continues. On the other hand, if the playback screen has been closed (step S63: YES), the video analysis support system 1 ends the playback process of the video data.

This allows the administrator to efficiently monitor operations by playing back video data using the index data. In addition, the administrator can set indexes for points that require attention while playing back video data, which can be used as a reference when viewing the video data later.

《Learning function from verified data》
The administrator can determine whether a frame indicated by an index with a warning level of 1 indicates fraudulent activity, and rewrite the warning level based on the determination result. The video server 13 may change the business rule based on the warning level rewritten by the administrator. For example, the video server 13 may change the start condition, end condition, detection condition, and detection accuracy based on the business rule. For example, in a business rule in which the detection accuracy is determined to be 95% or more as warning level 2, the detection accuracy is determined to be 90% or more but less than 95% as warning level 1, and the detection accuracy is determined to be less than 90% as warning level 0, if the frequency of rewriting from warning level 1 to warning level 2 is high, the video server 13 may change the threshold value that distinguishes between warning level 1 and warning level 2 to a value lower than 90%, for example. Also, for example, in a business rule in which the end condition is 3 seconds after the expiration of the start condition, if the detection condition is related to fraudulent activity and the frequency of rewriting from warning level 1 to warning level 2 is high, the time related to the end condition may be extended. Also, for example, if the object related to the detection condition is not related to fraudulent activity and the frequency of rewriting from warning level 1 to warning level 2 is high, the time related to the termination condition may be shortened.

<<Example of business rule settings>>
Here, an example of a business rule used in analyzing video data will be described.
For example, a business rule regarding cabinet operation can be set as a business rule. If the business rule regarding cabinet operation specifies that the cabinet should not be left open, the following business rule can be set:
The start condition of the business rule related to cabinet operation is that an object of the type "open cabinet" is detected. The end condition of the business rule related to cabinet operation is that an object of the type "open cabinet" is no longer detected. The detection condition of the business rule related to cabinet operation is that an object of the type "person" is no longer detected.
This allows the video server 13 to monitor whether or not the worker has left the cabinet open during cabinet operation.

For example, a business rule regarding document transfer can be set as a business rule. If the business rule regarding document transfer specifies that documents submitted by customers should be received and stored, that documents submitted to customers should be confirmed to be stored, and that lost documents should be discovered, the following business rule can be set.
The start condition of the business rule related to the transfer of documents is that two objects of the type "person" and an object of the type "document" are detected. The end condition of the business rule related to the transfer of documents is that the two objects of the type "person" are no longer detected. The detection condition of the business rule related to the transfer of documents is that the object of the type "document" continues to be not detected, and that the object of the type "document" is no longer detected in a position other than the area where it should be stored. Note that while the type of object in this business rule is "document," other types of objects, such as "banknote,""coin," and "card," may be set in other business rules. In addition, other business rules may specify the type of document more precisely than "document" (contract, identification document, pamphlet, etc.).
This allows the video image server 13 to monitor whether the document has been properly handed over or not.

For example, a business rule regarding entry and exit into a vault can be set as a business rule. If the business rule regarding entry and exit into a vault specifies that two or more people are to enter and exit the vault, the following business rule can be set.
The start condition of the business rule related to entry into the vault is that an object of the type "person entering" has been detected. An "entering person" is, for example, a person facing towards the inside of the vault. The end condition of the business rule related to entry into the vault is that a certain amount of time has elapsed since the start condition has been satisfied. The detection condition of the business rule related to entry into the vault is that a state in which an object of the type "person entering", different from the object of the start condition, has not been detected continues.
The start condition of the business rule related to leaving the vault is that an object of the type "person leaving the vault" has been detected. A "person leaving the vault" is, for example, a person facing away from the vault. The end condition of the business rule related to leaving the vault is that a certain amount of time has elapsed since the start condition has been satisfied. The detection condition of the business rule related to leaving the vault is that a state in which an object of the type "person leaving the vault", different from the object of the start condition, has not been detected continues.
This enables the video server 13 to monitor whether or not two or more people are entering or leaving the safe room.
In addition, in the business rule for entering and leaving the vault, the detection condition may include that at least one of the "persons" is an object of the type "administrator." In other words, the detection condition for the business rule for entering and leaving the vault may include a state in which an object of the type "administrator" is not detected for a long period of time.

For example, a business rule for operating a cash register can be set as a business rule. If it is determined that a general worker and a manager are to work together to access the inside of the cash register, the following business rule can be set as the business rule for operating the cash register.
The start condition for a business rule relating to the operation of a teller machine is that an object of the type "person" is detected near the teller machine. The end condition for a business rule relating to the operation of a teller machine is that an object of the type "person" is no longer detected near the teller machine. The detection condition for a business rule relating to the operation of a teller machine is that a state in which an object of the type "person" different from the object of the start condition is not detected continues. The detection condition for a business rule relating to the operation of a teller machine may be that a state in which an object of the type "manager" is not detected continues.
This allows the video server 13 to monitor whether the cash dispenser operation is being performed in the presence of a manager.
The above business rules can be applied to devices such as cash management devices, valuables management devices, and key management devices in addition to teller machines.

For example, a business rule can be set regarding the posting of discarded documents in a document processing device such as a shredder or document dissolving machine. If it is determined that the task of posting discarded documents in a document processing device is to be performed by a general worker and a manager, the following business rule can be set as the business rule regarding the posting of discarded documents.
The start condition of the business rule relating to the deposit of documents to be discarded is that an object of the type "person" is detected near the document processing device. The end condition of the business rule relating to the deposit of documents to be discarded is that an object of the type "person" is no longer detected near the document processing device. The detection condition of the business rule relating to the deposit of documents to be discarded is that a state in which an object of the type "person" different from the object of the start condition is not detected continues. The detection condition of the business rule relating to the deposit of documents to be discarded may also be that a state in which an object of the type "administrator" is not detected continues.
This allows the video image server 13 to monitor whether or not the disposal document is posted in the presence of an administrator.

Furthermore, if it is stipulated that documents to be discarded be inspected before being placed in the document processing device, the following business rules can be set.
In this case, the start condition of the business rule related to the deposit of discarded documents is that an object of the type "person" is detected near the document processing device. The end condition of the business rule related to the deposit of discarded documents is that an object of the type "person" is no longer detected near the document processing device. The detection condition of the business rule related to the deposit of discarded documents is that the state in which "documents" are detected continues for a certain period of time before the object of the start condition, that is, before the "documents" are deposited in the document processing device, the object of the type "documents" is no longer detected in the position where the objects of the type "documents" should be stored.
This allows the video image server 13 to monitor whether or not a "person" checks a document before depositing it for disposal.

For example, a business rule regarding postmarks on documents can be set as a business rule. When a business rule for documents is performed in which a postmark is affixed to a specific document as proof of receipt of the document, the following business rule can be set.
The start condition for a business rule related to postmarks on documents is that an object of the type "person" and an object of the type "document" are detected. The end condition for a business rule related to postmarks on documents is that objects of the type "person" and objects of the type "document" are no longer detected. The detection condition for a business rule related to postmarks on documents is that an object of the type "postmark" is detected superimposed on "document", which is the object of the start condition.
This allows the video image server 13 to monitor whether or not a postmark has been placed on a document.

For example, as the business rules, it is possible to set business rules regarding handover at a security center, a nurse center, etc. As the business rules for handover, when performing a task of handing over specific documents together with a roll call, the following business rules can be set.
The start condition of the business rule related to handover is that two objects of the type "person" and an object of the type "document" are detected. The end condition of the business rule related to handover is that the two objects of the type "person" are no longer detected. The detection conditions of the business rule related to handover are that a state in which an object of the type "person pointing" is not detected continues, and that an object of the type "document" is no longer detected in a position that is not an area where it should be stored.
This allows the video server 13 to monitor whether the handover has been performed appropriately.

<Action and Effects>
Thus, according to the first embodiment, the video analysis support system 1 detects frames indicating a state in which a task was not performed correctly from video data captured by the surveillance camera 11 based on the task rules, and generates index data indicating the frames. This allows the manager to check the video data based on the index data. In other words, the video analysis support system 1 can support the manager to easily check whether the task is being performed correctly from the video captured by the surveillance camera 11.

Other Embodiments
Although one embodiment has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes are possible. That is, in other embodiments, the order of the above-mentioned processes may be changed as appropriate. Also, some of the processes may be executed in parallel.
The video server 13 according to the above-described embodiment may be configured as a single computer, or the configuration of the video server 13 may be divided and arranged among a plurality of computers, and the plurality of computers may cooperate with each other to function as the video server 13. In another embodiment, the video server 13 and the client terminal 15 may be configured as a single computer.

The video server 13 according to the embodiment described above sets the detection accuracy for determining whether or not a detection has occurred for each business rule, but this is not limited to this. For example, in other embodiments, the detection accuracy may be set for each type of object, or the detection accuracy may be a fixed value.

The video server 13 according to the embodiment described above records the detection result data in the setting storage unit 135 when detection is performed by the detection unit 133, and skips analysis of video data for which detection result data has already been generated. As a result, if the detection accuracy setting in the business rules is changed after detection by the detection unit 133, the time required for generating an analysis result file can be shortened by using the detection result data that has already been generated. On the other hand, other embodiments are not limited to this. For example, the video server 13 according to other embodiments may perform detection by the detection unit 133 every time an analysis request is received. When the detection unit 133 performs detection each time in this manner, management of the analysis result file in the video server 13 can be facilitated.

The start condition and end condition of the business rule according to the embodiment described above are determined based on the detection result by the detection unit 133, but are not limited to this. For example, the start condition and end condition of the business rule according to other embodiments may be determined based on log data of a device installed in the monitored space. For example, the start condition may be that a predetermined type of cash dispenser operation has been performed on a cash dispenser installed in the monitored space, and a frame that satisfies the start condition may be identified from the log data of the cash dispenser.

The detection conditions for the business rules in the above-described embodiments specify fraud, i.e., a state in which the business is not performed correctly, but are not limited to this. For example, the detection conditions in other embodiments may specify a state in which the business is performed correctly.

Computer Configuration
FIG. 10 is a schematic block diagram illustrating a computer configuration according to at least one embodiment.
The computer 90 comprises a processor 91 , a main memory 93 , a storage 95 , and an interface 97 .
The above-mentioned video server 13 and client terminal 15 are each implemented in a computer 90. The operations of each of the above-mentioned processing units are stored in the form of a program in a storage 95. The processor 91 reads the program from the storage 95, loads it in the main memory 93, and executes the above-mentioned processing in accordance with the program. The processor 91 also secures storage areas in the main memory 93 corresponding to each of the above-mentioned storage units in accordance with the program. Examples of the processor 91 include a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), and a microprocessor.

The program may be for realizing some of the functions to be performed by the computer 90. For example, the program may be for realizing the functions by combining with other programs already stored in the storage or by combining with other programs implemented in other devices. In other embodiments, the computer 90 may include a custom LSI (Large Scale Integrated Circuit) such as a PLD (Programmable Logic Device) in addition to or instead of the above configuration. Examples of PLDs include PAL (Programmable Array Logic), GAL (Generic Array Logic), CPLD (Complex Programmable Logic Device), and FPGA (Field Programmable Gate Array). In this case, some or all of the functions realized by the processor 91 may be realized by the integrated circuit. Such an integrated circuit is also included as an example of a processor.

Examples of storage 95 include a magnetic disk, a magneto-optical disk, an optical disk, and a semiconductor memory. Storage 95 may be an internal medium directly connected to the bus of computer 90, or an external medium connected to computer 90 via interface 97 or a communication line. In addition, when this program is distributed to computer 90 via a communication line, computer 90 that receives the program may expand the program into main memory 93 and execute the above-mentioned processing. In at least one embodiment, storage 95 is a non-transitory tangible storage medium.

The program may be for realizing part of the above-mentioned functions. Furthermore, the program may be a so-called differential file (differential program) that realizes the above-mentioned functions in combination with another program already stored in storage 95.

1...Video analysis support system 11...Surveillance camera 13...Video server 131...Acquisition unit 132...Video storage unit 133...Detection unit 134...Setting unit 135...Settings storage unit 136...Selection unit 137...Determination unit 138...Generation unit 139...Output unit 15...Client terminal 151...Display control unit 152...Request unit 153...Playback unit 154...Editing unit 90...Computer 91...Processor 93...Main memory 95...Storage 97...Interface

Claims (10)

An acquisition unit that acquires video data captured by an imaging device that captures an image of a space in which a task is performed;
a setting unit that receives a setting of a business rule indicating a detection condition that specifies a state in which the business is correctly executed or a state in which the business is not correctly executed in the frame of the video data;
a determination unit that determines whether or not a frame that satisfies the detection condition is present from the video data;
a generating unit that generates index data in which an index indicating a frame defined in the detection condition is recorded,
When the setting unit receives a setting of a second business rule for video data that has already been analyzed according to a first business rule, the generation unit adds an index indicating a frame that satisfies the detection condition related to the second business rule to the initialized index data;
the generation unit initializes the index data by deleting indexes that have been added based on the analysis according to the business rules from the index data and leaving indexes that have been added manually.
Video image analysis support device. An acquisition unit that acquires video data captured by an imaging device that captures an image of a space in which a task is performed;
a setting unit that receives a setting of a business rule indicating a detection condition that specifies a state in which the business is correctly executed or a state in which the business is not correctly executed in the frame of the video data;
a determination unit that determines whether or not a frame that satisfies the detection condition is present from the video data;
a generating unit for generating index data in which an index indicating a frame defined in the detection condition is recorded;
Equipped with
When the setting unit receives a setting of a second business rule for video data that has already been analyzed according to a first business rule, the generation unit adds an index indicating a frame that satisfies the detection condition related to the second business rule to the initialized index data;
The determination unit, when detection results of a predetermined number or less of consecutive frames differ, aligns the determination result of the relevant frame with the determination results of the other frames . a selection unit that selects a business rule to be applied to the acquired video data from a plurality of business rules set for each business;
The video analysis support device according to claim 1 , wherein the determination unit determines whether or not a frame that satisfies the detection condition is present in the video data based on the selected business rule. a detection unit that detects an object from a plurality of frames constituting the video data,
The detection condition defines a detection state of the object;
The video analysis support device according to claim 1 , wherein the determination unit determines whether or not a frame that satisfies the detection condition is present based on a result of detection of the object from the video data. The detection unit identifies a type of the detected object for a frame in which the object is detected,
The detection condition specifies a type of the object and a detection state of the object.
The video analysis support device according to claim 4 . The detection unit specifies a detection accuracy of the object,
the business rule includes a threshold value of the detection accuracy for determining a detection state of the object,
The determination unit determines whether or not a frame that satisfies the detection condition exists by comparing the detection accuracy included in the detection result with the threshold value defined in the business rule.
The video analysis support device according to claim 4 or 5 . A video analysis support device according to any one of claims 1 to 6 ,
a playback unit that plays back the video data based on the index data;
a display control unit that displays a selection screen for selecting video data to be played;
Equipped with
A mark representing information on an analysis result of the video data is added to the selection screen in association with the video data. The generating unit of the video analysis support device assigns a warning level in association with the index of the index data,
an adding unit that adds an index indicating a frame related to a playback position of the video data to the index data;
an editing unit that edits the warning level recorded in the index data;
The video analysis support system according to claim 7 . On the computer,
A step of acquiring moving image data captured by an imaging device that captures an image of a space where a business is performed;
receiving a setting of a business rule indicating a detection condition that specifies a state in which the business is correctly executed or a state in which the business is not correctly executed in the frame of the video data;
determining whether or not a frame that satisfies the detection condition is present from the video data;
generating index data in which an index indicating a frame defined in the detection condition is recorded;
adding an index indicating a frame that satisfies the detection condition related to the second business rule to the initialized index data when a setting of a second business rule is accepted for video data that has already been analyzed according to a first business rule;
Run the command ,
The initialization of the index data is performed by deleting indexes that have been added based on the analysis according to the business rules from among the index data, and leaving indexes that have been added manually.
program. On the computer,
A step of acquiring moving image data captured by an imaging device that captures an image of a space where a business is performed;
receiving a setting of a business rule indicating a detection condition that specifies a state in which the business is correctly executed or a state in which the business is not correctly executed in the frame of the video data;
determining whether or not a frame that satisfies the detection condition is present from the video data;
generating index data in which an index indicating a frame defined in the detection condition is recorded;
adding an index indicating a frame that satisfies the detection condition related to the second business rule to the initialized index data when a setting of a second business rule is accepted for video data that has already been analyzed according to a first business rule;
Run the command,
In the step of determining whether or not a frame that satisfies the detection condition exists from the video image data, if the detection results of a predetermined number or less of consecutive frames are different, the determination result of the corresponding frame is made uniform with the determination results of the other frames.
Program.

Images