JP4944818B2 - Search device and search method - Google Patents

Description

  The present invention relates to a search apparatus and a search method for searching for and displaying information representing the state of an object represented by a case frame.

  2. Description of the Related Art In recent years, a technique for analyzing a captured image, recognizing an object displayed in the image, determining characteristics of the object, and displaying the image on the monitoring apparatus is known in a monitoring apparatus including a monitoring camera or the like. ing.

  For example, in Patent Document 1, an image captured by a monitoring camera is analyzed, and a feature amount is obtained from a person (for example, the face of a householder or an acquaintance) or clothes (for example, a uniform of a courier) displayed on the image. If the obtained feature quantity is similar to the feature quantity of a person or clothes that has been categorized and stored in advance by occupation or family relationship, it is determined that the person is not a suspicious person, and the result is displayed on the image monitoring device. Techniques for making them disclosed are disclosed.

JP 2001-016579 A

  However, in the technique described in Patent Document 1, whether or not a person is a suspicious person by simply comparing a feature amount such as a face image of a person registered in advance with a feature amount of a face image captured by a surveillance camera or the like. The user simply determines whether or not and displays the result, and the user captures and stores an image that is captured and stored using features such as actions and states performed by an object such as a person displayed in the captured image. There is a problem that a desired image cannot be retrieved and displayed from among the images, and it is not easy to identify an object or a person (for example, identification of a suspicious person or the like).

  The present invention has been made in view of the above, and provides a search device and a search method that can easily search and display an image of an image of the object using a key that is a feature of the object. The purpose is to do.

  In order to solve the above-described problems and achieve the object, the invention according to claim 1 is directed to an image information storage unit that stores image information of an object imaged by the imaging unit, and an object operation by analyzing the image information. Information indicating a motion state of the extracted object, including a verb indicating a type of motion of the object and a case element which is a noun related to the motion of the object, and the verb and the case State information storage means for storing state information represented by a case frame indicating a semantic relationship with an element in association with the image information stored by the image information storage means, and stored in the image information storage means Input receiving means for receiving input of key information for searching for the image information, and the state information storage means based on the key information received by the input receiving means. The status information acquisition means searches for and acquires the status information including the key information from the status information, and the status information acquisition means searches the image information stored in the image information storage means. Display means for displaying the image information acquisition means for acquiring the image information corresponding to the status information, the status information acquired by the status information acquisition means, and the image information acquired by the image information acquisition means And comprising.

  The invention according to claim 2 is the search device according to claim 1, further comprising face feature information storage means for storing a person name and face feature information of the person name in association with each other, wherein the input receiving means Receives an input of the person name as the case element as the key information, and the state information acquisition means determines whether or not the person name is included in the key information received by the input reception means. When the state information acquisition unit determines that the person name is included in the key information and the person name is included in the state information, the image information acquisition unit stores the face feature storage. The face feature information corresponding to the person name stored by the means is acquired, and stored in the image information storage means in association with the state information including face feature information similar to the acquired face feature information. image Obtaining a distribution, and wherein.

  According to a third aspect of the present invention, in the search device according to the second aspect, the input receiving unit further receives an input of a human face image, and the image information acquiring unit is included in the key information. If the face feature information corresponding to the person name is not stored in the face feature information storage means, the input accepting means analyzes the face image accepted for input, extracts face feature information, and extracts The image information stored in the image information storage unit is acquired in association with the state information including face feature information similar to the face feature information.

  According to a fourth aspect of the present invention, in the search device according to any one of the first to third aspects, the state information displayed by the case frame includes a case element, and a feature of the object. The operation information including the attribute information indicating, the type of the motion of the object, the position information indicating the place where the motion is performed, the motion start time when the motion is started, and the motion end time when the motion is finished, Used when the operation target information indicating the target of the operation, the starting point of the operation of the object with respect to the operation target information, the end point of the end of the operation, and the operation of the object with respect to the operation target information And means information indicating a thing.

  According to a fifth aspect of the present invention, in the search device according to the fourth aspect, the image information includes a start time at which imaging of the object is started and an end time at which imaging of the object is ended. The state information acquisition means sets the start time and the end time included in the image information to the operation start time and the operation end time included in the state information, and the image information acquisition means The image information corresponding to the state information in which the operation start time and the operation end time set by the state information acquisition unit are set from among the image information stored in the image information storage unit It is characterized by acquiring.

  The invention according to claim 6 is the search device according to claim 4 or 5, further comprising a timer for measuring an elapsed time from the start time when the monitoring area was imaged, and acquiring the image information The means determines whether the elapsed time counted by the timer has passed a predetermined time from the start time, and the elapsed time from the start time has passed a predetermined time. If it is determined that the image information is stored, the image information corresponding to the state information including the start time is deleted from the image information storage unit.

  The invention according to claim 7 is the search device according to any one of claims 4 to 6, wherein the attribute information indicating the feature of the object includes the body shape of the person and the appearance of the person. The input accepting means further accepts an input of the figure of the person or / and the appearance of the person, and the status information acquiring means is selected from the status information stored in the status information storage means. The input receiving means searches for and obtains the state information including the figure of the person who accepted the input and / or the appearance of the person, and the image information obtaining means stores the image stored in the image information storage means The image information is acquired from information based on the figure of the person and / or the appearance of the person included in the state information acquired by the state information acquisition unit. To.

  Further, the invention according to claim 8 is the search device according to any one of claims 1 to 7, wherein among the state information displayed by the case frame, the type of movement of the object, and A suspicious person information storage unit that stores a verb representing a motion type, a case element that is a noun related to the motion of the object, and a value of the case element, and the input reception unit receives an input Suspicious person information determining means for determining whether or not the key information includes the value of the case element stored in the suspicious person information storing means, and the status information acquiring means includes the suspicious person When the information determination means determines that the value of the case element is included in the key information, the state information stored in the state information storage means includes the value of the case element. Search and obtain status information It features a.

  According to a ninth aspect of the present invention, there is provided image information storage means for storing image information of an object picked up by the image pickup means, analysis of the image information to extract an operation state of the object, and operation state of the extracted object A case frame including a verb representing the type of action of the object and a case element that is a noun related to the action of the object, and indicating a semantic relationship between the verb and the case element In a search device comprising: state information storage means for storing the state information represented in association with the image information stored in the image information storage means; and display means for displaying the state information and the image information An input receiving step for receiving input of key information for searching for the image information stored in the image information storage means, and a status information acquisition means, which is a search method to be performed State information obtained by searching the state information including the key information from among the state information stored in the state information storage unit based on the key information received by the input receiving unit An acquisition step and an image information acquisition unit that acquires the image information corresponding to the state information searched by the state information acquisition unit from the image information stored in the image information storage unit And a step.

  According to the invention of claim 1, the image information storage means stores the image information of the object imaged by the imaging means, and the state information storage means analyzes the image information and extracts the operation state of the object, Information indicating the extracted motion state of the object, including a verb indicating the type of motion of the object and a case element that is a noun related to the motion of the object, between the verb and the case element State information represented by a case frame indicating a semantic relationship is stored in association with the image information stored in the image information storage unit, and an input reception unit stores the image information stored in the image information storage unit. The input of key information for searching is received, and the state information acquisition means is based on the key information received by the input reception means, from among the state information stored in the state information storage means, The status information including the key information is retrieved and acquired, and the image information acquisition means adds the status information searched by the status information acquisition means from the image information stored in the image information storage means. The corresponding image information is acquired, and the display unit displays the state information acquired by the state information acquisition unit and the image information acquired by the image information acquisition unit. Thus, there is an effect that a desired image is retrieved and displayed, and a desired object can be easily specified.

  According to a second aspect of the present invention, in the search device according to the first aspect, the face feature information storage unit stores the person name and the face feature information of the person name in association with each other, and the input acceptance The means accepts input of a person name as the case element as the key information, and the state information acquisition means determines whether or not the person name is included in the key information accepted by the input acceptance means. If the state information acquisition unit determines that the key name includes the person name and the state information includes the person name, the facial information is determined. The face feature information corresponding to the person name stored in the storage unit is acquired and stored in the image information storage unit in association with the state information including face feature information similar to the acquired face feature information. Image information Because obtaining an effect that it is possible to easily identify the person, such as a desired suspicious person.

  According to a third aspect of the present invention, in the search device according to the second aspect, the input receiving unit further receives an input of a human face image, and the image information acquiring unit is included in the key information. If the face feature information corresponding to the person name is not stored in the face feature information storage means, the input accepting means analyzes the face image accepted for input and extracts face feature information, Since the image information stored in the image information storage means is acquired in association with the state information including face feature information similar to the extracted face feature information, a person such as a desired suspicious person is further specified. There is an effect that it can be easily performed.

  According to a fourth aspect of the present invention, in the search device according to any one of the first to third aspects, the state information displayed by the case frame includes a case element and a feature of the object. An operation time including the attribute information indicating the type of motion of the object, the position information indicating the place where the motion was performed, the motion start time at which the motion was started, and the motion end time at which the motion was terminated, Used when the operation target information indicating the target of the operation, the starting point at which the object starts to operate with respect to the operation target information, the end point at which the operation ends, and when the object operates with respect to the operation target information. And means information indicating the things that have been present, so that it is possible to easily and finely identify a person such as an object or a suspicious person.

  According to the invention of claim 5, in the search device according to claim 4, the image information includes a start time at which imaging of the object is started, and an end time at which imaging of the object is ended. The state information acquisition means sets the start time and the end time included in the image information as the operation start time and the operation end time included in the state information, and the image information acquisition means Is the image corresponding to the state information in which the operation start time and the operation end time set by the state information acquisition unit are set from among the image information stored in the image information storage unit Since the information is acquired, there is an effect that the object or person can be easily identified even when the action of the person such as the object or the suspicious person changes.

  According to a sixth aspect of the present invention, in the search device according to the fourth or fifth aspect, the timer measures the elapsed time from the start time when the monitoring area was imaged, and the image information acquisition unit Determines whether the elapsed time measured by the timer has passed a predetermined time from the start time, and the elapsed time from the start time has passed the predetermined time If it is determined that the image information corresponding to the state information including the start time is deleted from the image information storage means, it is possible to reduce the storage capacity and efficiently specify a desired object. The effect that it can be performed automatically is produced.

  According to a seventh aspect of the present invention, in the search device according to any one of the fourth to sixth aspects, the attribute information indicating the feature of the object includes the figure of the person and the appearance of the person. The input receiving means further receives input of the figure of the person or / and the appearance of the person, and the status information acquisition means is selected from the status information stored in the status information storage means. The input receiving means searches for and obtains the state information including the figure of the person who received the input and / or the appearance of the person, and the image information acquisition means is stored in the image information storage means Since the image information is acquired from the image information based on the figure of the person or / and the appearance of the person included in the state information acquired by the state information acquisition unit, An effect that a particular person of the suspicious person, etc. Nozomu can be more easily performed.

  Moreover, according to the invention concerning Claim 8, in the search device of any one of Claims 1-7, a suspicious person information storage means is the said object among the status information displayed by the said case frame. The action type of the object, the verb representing the action type of the object, the case element which is a noun related to the action of the object, and the value of the case element are stored in association with each other. Determining whether or not the key information received by the input receiving means includes the value of the case element stored in the suspicious person information storing means, and the status information acquiring means includes the suspicious person When the information determination means determines that the value of the case element is included in the key information, the state information stored in the state information storage means includes the value of the case element. Because it retrieves and retrieves status information, An effect that a particular person of the suspicious person, etc. Nozomu can be further efficiently performed.

  Exemplary embodiments of a search apparatus and a search method according to the present invention will be explained below in detail with reference to the accompanying drawings.

(Embodiment 1)
FIG. 1 is a block diagram illustrating a functional configuration of the security device 100 according to the first embodiment. As shown in FIG. 1, the security device 100 includes a sensor 110, a monitoring camera 120, and a main body 130, and is connected to the monitoring center 200 via a network 300. The main body unit 130 includes a detection information receiving unit 131, an image processing unit 132, a security state switching unit 133, an operation panel control unit 134, a language information acquisition unit 135, an image information acquisition unit 136, and a transmission / reception unit. 137, a security state storage unit 138, a language structure construction unit 139, a storage unit 140, a timer 170, and an operation panel 180. The storage unit 140 further includes a first storage unit 150 that stores the video database 151, a second storage unit 160 that stores the language information database 161, and a third storage unit 190 that stores the face feature database 191. The operation panel 180 includes an input receiving unit 181, a language information display unit 182, and a video display unit 183.

  The sensor 110 includes a detection device that detects an abnormality in the detection region. For example, an infrared sensor that detects a person based on a change in the amount of received infrared light, or a block that detects a person by blocking infrared light or the like. It is composed of various sensors for detecting abnormalities in a detection region such as a sensor, a presence sensor for detecting a person by disturbance of electromagnetic waves, and a magnet sensor for detecting opening / closing of a door by a magnet. When the sensor 110 detects an abnormality in the detection area, the sensor 110 transmits a detection signal indicating that the abnormality has been detected to the detection information receiving unit 131 described later. Note that the detection region is a range in which the sensor 110 can detect an abnormality.

  The monitoring camera 120 is an imaging device that is installed near the entrance / exit of a residence, which is a monitoring area, or near the entrance / exit to a site, and images the monitoring area. Specifically, the monitoring camera 120 images the areas a1 and a2 included in the monitoring area as shown in FIG. 2, and stores the captured images in the video database 151 stored in the first storage unit 150 described later. FIG. 3 is a diagram illustrating a state in which the monitoring camera 120 images the area a1.

  Returning to FIG. 1, the detection information receiving unit 131 receives detection information transmitted when an abnormality is detected by the sensor 110. The detection information is a detection signal sent from the detection unit 111. When a plurality of sensors 110 are installed, the detection information includes a detection signal, a sensor ID for identifying the sensor, and the like.

  The image processing unit 132 analyzes an image captured by the monitoring camera 120 stored in the video database 151 described later, and extracts an image or video of the operation state of the object shown in the captured image. For example, when a person is reflected in the captured image, the image processing unit 132 identifies the person by detecting a skin area or the like, and performs the action of the identified person (for example, walking toward the entrance). Image) and video are extracted.

  As will be described later, the security state switching unit 133 is controlled by the operation panel control unit 134 in response to the switching signal of the security state that the operation panel 180 has received an input, and the security state to be described later according to the controlled switching signal. Store in the storage unit 138.

  The language information acquisition unit 135 determines whether or not a search key received by the input receiving unit 181 (to be described later) is a keyword, a sentence, or the like, and determines that the search key is a keyword, a sentence, or the like. Is an action name indicating the type of action (behavior) included in the state information stored in the language information database 161 stored in the second storage unit 160 described later, or a case element (deep case) that is a noun related to the action. It is determined whether or not it is stored, and a deep case frame including a keyword, a sentence, or the like that has been accepted as a search key is displayed on the operation panel 180 described later.

  In addition, the language information acquisition unit 135 determines whether or not a person name is included in a keyword or a sentence that the input receiving unit 181 described later receives an input.

  Further, as will be described later, the language information acquisition unit 135 displays the imaging start time and the imaging end time included in the images and videos stored in the video database 151 as the deep case frames stored in the language information database 161 described later. It is set to a time case which is one of (described later).

  Note that the language information acquisition unit 135 described above searches for information such as a main case and an attribute case stored as a deep case frame (described later) according to a keyword, a sentence, a face image, or the like received by the input receiving unit 181. The method is such that when the entered keyword or the like completely matches the content stored as a deep case frame (described later) such as the main case and the attribute case, or if they partially match, the content is similar. In some cases, it is possible to search. Here, a search method in the case where there is similarity can be searched by a conventional technique such as a search method by performing principal component analysis of the content.

  The image information acquisition unit 136 includes images and videos captured by the monitoring camera 120 stored in the video database 151 stored in the first storage unit 150 described later, and the deep case frames determined by the language information acquisition unit 135 described above. An image or video stored in the video database 151 including the deep case frame ID is acquired and displayed on the video display unit 183 described later.

  Further, as will be described later, the image information acquisition unit 136 acquires face feature information similar to the face feature information corresponding to the person name stored in the face feature information database 190. Then, a deep case frame (described later) including face feature information similar to the acquired face feature information is specified from a language information database 161 described later, and stored in the video database 151 corresponding to the specified deep case frame. Get images and videos. Note that the face feature information includes the characteristics (for example, “the nose is high”, “the eyes are double”, “the eyebrows are dark”, etc.) of each element constituting the face of the person (eg, the nose, eyes, eyebrows, etc.). It is what you represent.

  Furthermore, when the face feature information corresponding to the person name that the input receiving unit 181 has received is not stored in the face feature information database 190, the image receiving unit 181 receives an input from the input receiving unit 181 described later. The received face image of the person is analyzed to extract face feature information of the person. Then, an image or video stored in the video database 151 corresponding to a deep case frame including facial feature information similar to the extracted facial feature information is acquired.

  Further, when the image information acquisition unit 136 sets the imaging start time and the imaging end time in the time case of the deep case frame from the language information acquisition unit 135, the image and video from the imaging start time to the imaging end time are displayed. Extracted from the images and videos stored in the video database 151 and displayed on the video display unit 183 to be described later.

  Further, the image information acquisition unit 136 has a predetermined elapsed time stored in advance in a storage medium (not shown) such as a memory, since the time since the image or video was stored in the video database 151 measured by the timer 170 described later. Delete images and videos that have passed.

  The transmission / reception unit 137 transmits / receives various kinds of information such as detection information transmitted from the sensor 110 to the monitoring center 200 via the network 300. Specifically, when the transmission / reception unit 137 receives detection information or the like from the sensor 110 and the security mode is stored in the current security state stored in the security state storage unit 138, the transmission / reception unit 137 enters the monitoring area. Alarm information indicating that an abnormality has occurred is transmitted to the monitoring center 200.

  The security state storage unit 138 is a storage medium such as an HDD (hard disk drive) or a memory that stores either the security mode or the security release mode which is the current security state of the security device 100.

  Here, the security mode is a state in which alarm information for notifying an abnormality in the monitoring area is transmitted to the monitoring center 200 connected via the network 300 when the sensor 110 detects the abnormality and acquires the detection information. The security release mode refers to a state in which alarm information notifying the abnormality is not transmitted to the monitoring center 200 even when the detection information is acquired.

  The language structure constructing unit 139 is represented by a deep case frame described later from images, videos, keywords, sentences, and the like of the motion states of objects extracted from the images and videos stored in the video database 151 by the image processing unit 132 described above. Build state information.

  The storage unit 140 includes an HDD that stores a first storage unit 150 that stores the video database 151, a second storage unit 160 that stores the language information database 161, and a third storage unit 190 that stores the face feature database 191. (Hard disk drive) and storage media such as a memory.

  The video database 151 stores an image or video captured by the monitoring camera 120 together with the time when the image was captured (for example, imaging start time, imaging end time) and a deep case frame ID described later of the deep case frame.

  The language information database 161 stores state information representing the motion state of an object in a deep case frame structure in association with images and videos stored in the video database 151. The deep case frame (hereinafter, also simply referred to as case frame) is a language structure that expresses the semantic relationship of a noun to a verb as a deep case with a verb as a reference. Specifically, the images and videos stored in the video database 151 and the language information database 161 are associated with each other by a deep case frame ID described later.

  FIG. 4 shows a data structure of state information stored in the language information database 161, and FIG. 5 shows an example of the data structure. As shown in FIG. 4, the state information includes a case frame ID for identifying a case frame, an action name indicating the type of action (action), and a case element (deep case) that is a noun related to the action. Yes. The deep case includes a main case, an attribute case, a location case, a time case, a source case, a target case, a target case, a tool case, and an action name on which the main case operates.

  The main character represents a subject that causes an action, and includes, for example, the name of a person. The attribute case refers to information on the main case. For example, the attribute case includes height information, body shape, upper body color, lower body color, head color, facial information such as appearance, and gait information.

  Face information and gait information are further divided into detailed elements. For example, the face information includes elements such as facial features (high nose, double eyes, dark eyebrows, etc.) representing facial features, line-of-sight direction, presence / absence of mask, presence / absence of sunglasses, gender, and age. The gait information includes elements such as stride length, posture, leg length, and walking speed. In the face feature database 191 to be described later, face features and names of persons having the face features are stored in association with each other.

  A place case represents a place where an action or a state occurs. For example, the place case includes a head position, a body position, a hand position, a face position, a leg position, and the like.

  The time case represents the time at which an action or condition occurs. For example, the time case represents a time representing the current time, an operation start time representing the time at which the operation was started, an operation end time representing the time at which the operation was completed, and a date.

  The source case represents the starting point of movement or state change. For example, the source case includes location information indicating the position of the starting point where the leading character moves or moves, and attribute information indicating the attribute of the leading character at the starting point.

  The target case represents an end point such as movement or state change. For example, the target case includes location information indicating the position of the end point where the main character moves or moves, and attribute information indicating the attribute of the main character at the end point.

  A target case represents an object that is the target of an action or state. The target case includes a target ID that is unique identification information and attribute information that is unique information.

  The tool scale represents an auxiliary means of movement. The tool case includes a tool ID that is unique identification information and attribute information that is unique information.

  In addition, the deep case which can be utilized is not restricted to these, For example, all other deep cases, such as the experience person's case showing the experience person of a mental event, can be used.

  FIG. 5 is a diagram illustrating a specific example of state information stored in the language information database 161. FIG. 5 shows an example of state information generated when the person A (main character) detects the action of “squatting”. In this case, specific values such as the height and body shape of the person A are set in the attribute case. In the place case and the time case, specific values relating to the position and time when the person A squats down are set.

  As will be described later, in the search processing performed by the language information acquisition unit 135 and the image information acquisition unit 136, the keywords received by the input reception unit 181, which will be described later, are included in the language information database 161 stored in advance. It is determined whether or not a sentence is included, and images and videos stored in association with these keywords and sentences are acquired and displayed on a language information display unit 182 and a video information display unit 183 described later.

  Returning to FIG. 1, the face feature database 191 stores the name of a person and the features of the person's face (for example, nose, eyes, eyebrows, etc.) (for example, “high nose”, “double eyes”, “eyebrows”). Are stored in association with each other.

  The timer 170 measures the time from the time when the image is taken by the monitoring camera 120.

  The operation panel 180 includes a liquid crystal panel such as an LCD (Liquid Crystal Display), and inputs the security state of either the security mode or the security release mode described above by pressing an operation button displayed on the liquid crystal panel. The operation panel control unit 134 performs control such as output of display information to the operation panel 180 and acquisition of input information such as a search key input from the operation panel 180.

  The input receiving unit 181 receives an input of a keyword, a sentence, or a person's face image for a user to search for and acquire an image or video stored in the video database 151. FIG. 6 is a diagram illustrating an example of the operation panel 180 including the input receiving unit 181.

  As shown in FIG. 6, the input receiving unit 181 displays a case element (deep case) configured as the above-described deep case frame, a personal name, or an action name that is one of the deep cases in a pull-down menu. It is also possible to select and search from a window or the like, or to specify a keyword or sentence that is not displayed in these pull-down menus. For example, an image or video of a celebrity is stored in advance in the video database 151, and a deep case frame is constructed, so that the name of the celebrity is designated as a search key, and the case elements constituting the celebrity deep case frame are specified. Searching for a deep case frame that includes a face case information that is similar to the face feature of the face image of the person whose input was accepted, or searching for a person such as a searcher or a suspicious person It is also possible to search.

  The video display unit 183 displays the image and video acquired by the image information acquisition unit 136 described above. Specifically, an image or video is displayed in the upper left area on the operation panel 180 shown in FIG.

  The language information display unit 182 displays the contents of the case elements (deep case) that constitute the deep case frame including the keyword or sentence that the input receiving unit 181 has received. The user presses a button or the like displayed on the input receiving unit 181 to display an image or video on the video display unit 183 or summarize the contents of the case element (deep case) on the language information display unit 182. It is also possible to display.

  As described above, when the image information acquisition unit 136 determines that a predetermined elapsed time has elapsed and deletes the image or video, the video display is performed even when these buttons are pressed. No image or video is displayed on the part 183.

  The monitoring center 200 receives a report from the security device 100 that has detected an abnormality in the monitoring area, and instructs the standby security guard to go to the monitoring area (not shown) where the abnormality has been detected. Report to relevant agencies such as the police and fire departments.

  The network 300 is a communication line such as the Internet or a telephone line that mediates communication between the security device 100 and the monitoring center 200.

  Then, each process performed with the security apparatus 100 concerning Embodiment 1 comprised as mentioned above is demonstrated.

  FIG. 7 is a flowchart showing a processing procedure until the user searches for an image or video stored in the security device 100 by inputting a keyword, text, or the like and acquires the result in the first embodiment. . In the following description, the video database 151 stores images and videos previously captured by the monitoring camera 120, and the language information database 161 stores deep case frames stored in association with the images and videos. It shall be.

  First, when the input receiving unit 181 receives an input of a keyword, text, or the like for searching for an image or video from the user (step S701), the language information acquiring unit 135 determines that the search key that has received the input is a keyword. It is determined whether or not the language is a language such as text (step S702).

  When the language information acquisition unit 135 determines that the search key is a keyword, a sentence, etc., is a language (step S702; Yes), the language structure construction unit 139 converts the language, such as the keyword, the sentence, etc., into a deep case. The state information represented by the frame is constructed (step S703), then the case frame stored in the language information database 161 is read (step S704), and the input is accepted in step S701 from the read case frames. It is determined whether or not there is a case frame including a language such as a keyword or a sentence that has been entered (step S705).

  Thereafter, the language information acquisition unit 135 further determines whether or not the input receiving unit 181 further receives an input of a language such as a keyword or a sentence (step S706), and receives an input of the language such as the keyword or the sentence. If it is determined that it is present (step S706; Yes), the process returns to step S701, and the processes from step S701 to S705 are repeated.

  On the other hand, if the language information acquisition unit 135 determines that the input of a language such as a keyword or a sentence is not accepted (step S706; No), the searched deep case frame including the keyword or the sentence is displayed on the operation panel 180. Then, the imaging start time and the imaging end time stored in the video database 151 are stored in the searched time frame of the deep case frame (steps S713 and S714).

  Thereafter, the image information acquisition unit 136 acquires, from the video database 151, an image or video including the time between the imaging start time and the imaging end time set in the time case in steps S 713 and S 714. It is displayed on the display unit 183 (step S715).

  Then, the image information acquisition unit 136 deletes an image or video that has passed a predetermined time since the image or video was stored in the video database 151 measured by the timer 170 (step S716).

  On the other hand, if the language information acquisition unit 136 determines in step S702 that the keyword or sentence is not a language (step S702; No), it further determines whether or not the keyword or sentence is a personal name (step S707). ), When it is determined that the keyword or sentence is not a person's name (step S707; No), the process returns to step S701.

  On the other hand, when the language information acquisition unit 136 determines that the keyword, sentence, or the like is a person name (step S707; Yes), the image information acquisition unit 136 corresponds to the person name as a keyword in the face feature database 191. The facial feature is searched (step S708), and it is determined whether or not the facial feature corresponding to the person name is registered (step S709).

  If the image information acquisition unit 136 determines that the face feature corresponding to the person name is registered (step S709; Yes), the face information is sent to the language structure construction unit 139, and the process returns to step S703. The search process shown in step S704 and thereafter is performed.

  On the other hand, when it is determined that the face feature corresponding to the person name is not registered (step S709; No), the image information acquisition unit 136 further determines whether the input reception unit 181 has received an input of a face image. (Step S710). When the input receiving unit 181 determines that the input of the face image is not received (step S710; No), the process returns to step S701.

  On the other hand, when the input reception unit 181 determines that the input of the face image has been received (step S710; Yes), the face information is extracted from the face image, and the face information is sent to the language structure construction unit 139 (step S711). Returning to step S703, the search processing after step 704 is performed. Thus, when the processing from step S701 to S716 ends, all processing in the present embodiment ends.

  As described above, in the present embodiment, the video database 151 stores the image and video of the object captured by the monitoring camera 120, and the language information database 161 extracts the operation state of the object by analyzing the image and video. Information indicating the motion state of the extracted object, including a verb representing the type of motion of the object and a case element that is a noun related to the motion of the object, and a semantic relationship between the verb and the case element A deep case frame represented by a case frame indicating the image is stored in association with an image or video stored in the video database 151, and the input receiving unit 181 searches for the image or video stored in the video database 151. The language information acquisition unit 135 receives an input of a keyword, a sentence, and the like, and the language information acquisition unit 135 receives a language information database based on the keyword, the sentence, etc. that the input reception unit 181 receives an input. The deep case frames including keywords, sentences, etc. are searched and acquired from the deep case frames stored in the image 161, and the image information acquisition unit 136 includes the images and videos stored in the video database 151. The language information acquisition unit 135 acquires an image or video corresponding to the searched deep case frame, and the language information display unit 182 acquires the deep case frame acquired by the language information acquisition unit 135 and the image information acquisition unit 136. The displayed image and video are displayed, so that a desired image can be searched from the captured images and displayed, and a desired object can be easily identified.

  Further, in the present embodiment, the face feature database 191 stores the person name and the face feature information of the person name in association with each other, and the input receiving unit 181 stores the person name that is a case element as a keyword or a sentence. The language information acquisition unit 135 receives an input, determines whether or not a person name is included in a keyword or a sentence that the input reception unit 181 receives an input, and the image information acquisition unit 136 performs a language information acquisition unit 135. If it is determined that the person name is included in the keyword or sentence and the person name is included in the deep case frame, face feature information corresponding to the person name stored in the face feature database 191 is acquired, Since an image or video stored in the video database 151 is acquired in association with a deep case frame including facial feature information similar to the acquired facial feature information, a desired person such as a suspicious person It is possible to perform certain easily.

  In the present embodiment, the input receiving unit 181 further receives input of a person's face image, and the image information acquiring unit 136 stores face feature information corresponding to the person name included in the keyword in the face feature database 191. If not stored, the input receiving unit 181 analyzes the face image that the input has been received to extract face feature information, and associates it with a deep case frame that includes face feature information similar to the extracted face feature information. Since the images and videos stored in the video database 151 are acquired, it is possible to more easily identify a desired person such as a suspicious person.

  In the present embodiment, the deep case frame includes case elements, which are attribute cases indicating the characteristics of the object, types of movement of the object, place cases indicating the place where the movement is performed, and movement. A time case that includes an operation start time that has started and an operation end time that has ended the operation, a target case that indicates the target of the operation, a source case that the object has started to operate with respect to the target case, and a target case that has ended the operation And a tool case indicating an object used when the object operates with respect to the target case, it is possible to easily and finely identify a person such as an object or a suspicious person.

  In the present embodiment, the video database 151 includes a start time at which imaging of an object is started and an end time at which imaging of the object is completed, and the language information acquisition unit 135 is included in an image or video. The start time and the end time are set as the operation start time and the operation end time included in the deep case frame, and the image information acquisition unit 136 uses the language information from the images and videos stored in the video database 151. Since an image or video corresponding to the deep case frame in which the operation start time and the operation end time set by the acquisition unit 135 are set is acquired, even when the action of a person such as an object or a suspicious person changes The object or person can be easily identified.

  In the present embodiment, the timer 170 measures the elapsed time from the start time when the monitoring area was imaged, and the video database 151 has a predetermined elapsed time from the start time. If it is determined whether the elapsed time from the start time has passed a predetermined time, an image or video corresponding to the deep case frame including the start time is displayed. Since the data is deleted from the database 151, the storage capacity can be reduced and a desired object can be identified efficiently.

  In the present embodiment, the attribute case indicating the feature of the object includes the figure of the person and the appearance of the person, and the input receiving unit 181 further inputs the figure of the person and / or the figure of the person. The language information acquisition unit 135 receives and searches the deep case frames including the body type and / or the appearance of the person that the input reception unit 181 has received from the deep case frames stored in the language information database 161. The image information acquisition unit 136 acquires the body shape of the person included in the deep case frame acquired by the language information acquisition unit 135 from the images and videos stored in the video database 151 and / or Since an image or video is specified based on the appearance and the specified image or video is acquired, it is possible to more easily specify a person such as a desired suspicious person.

(Embodiment 2)
In the first embodiment described above, an image stored in advance can be stored by specifying a search key such as a keyword including a person name or a sentence from among the deep case frames stored in association with the image or video. We decided to acquire the video. However, when acquiring an image of a person with a certain characteristic (for example, a person such as a suspicious person), it may be possible to search for an image or a video more efficiently if the characteristic is registered in advance. is there. Therefore, description will be given of a case in which information representing the characteristics of such a person (hereinafter referred to as suspicious person information) is stored in advance, and an image or video is retrieved and acquired using the stored suspicious person information. To do.

  FIG. 8 is a block diagram illustrating a configuration of a security device 800 according to the second embodiment. The security device 800 according to Embodiment 2 is implemented in that the main body 830 of the security device further includes a suspicious person information determination unit 860 and a fourth storage unit 850 that stores a suspicious person information database 851. This is different from the main body 130 in the first embodiment. In the following description, the same components as those in the first embodiment are denoted by the same reference numerals, and the description thereof is omitted.

  The suspicious person information database 851 stores an action name, a deep case, and a specific value of the deep case among the deep cases included in the deep case frame in association with each other as a rule of suspicious person information (hardware). Wear disk drive) and memory.

  FIG. 9 is a diagram illustrating an example of the suspicious person information database 851. The shaded portion in FIG. 9 is for a suspicious person or a suspicious behavior when the action name is “Korokoro” and the body position of the place character is “in front of the entrance”, that is, when the action is in front of the entrance. An example stored as is shown.

  The suspicious person information determination unit 860 searches the deep case frame including the information stored as the suspicious person information with reference to the suspicious person information database 851 described above, and the keyword or text input from the input receiving unit 181 or the like. It is determined whether or not information stored as suspicious person information, that is, suspicious behavior is included.

  When the suspicious person information determination unit 860 determines that the suspicious behavior stored as the suspicious person information is included in the keyword, the sentence, or the like, the language information display unit displays the deep case frame including the suspicious action. 182.

  It should be noted that action names and types of deep cases stored in the suspicious person information database 851 are stored in advance by the user, and these contents can be changed according to the season and time zone, or the surveillance camera 120 can capture images. It is also possible to switch whether or not to perform the process by the suspicious person information determination unit 860 according to the authentication result by authenticating the person shown in the image or video.

  FIG. 10 is a flowchart showing a processing procedure performed by the security device 800. In each process shown in FIG. 10, as described above, a suspicious person information database 851 as shown in FIG. 9 is stored in advance by the user, and an instruction for making a suspicious person determination by the user from the input receiving unit 181 or the like. Shall be made.

  First, the suspicious person information determination unit 860 reads the stored suspicious person information with reference to the suspicious person information database 851 (step S1001). Then, using the read suspicious person information as a key, a deep case frame including the suspicious person information is searched from the language information database 161 (step S1002).

  Then, it is determined whether or not there is a deep case frame including suspicious person information in the language information database 161 (step S1003). When it is determined that there is no deep case frame including suspicious person information (step S1003; No) ), Proceed to the next step without doing anything.

  On the other hand, when it is determined that there is a deep case frame including suspicious person information (step S1003; Yes), the deep case frame including the suspicious person information is displayed on the language information display unit 182 (step S1004). After that, it is determined whether or not all the suspicious person information rules stored in the suspicious person information database 851 are referred to (step S1005), and when it is determined that all the suspicious person information rules are not referred to ( Step S1005; No), it returns to Step S1001, and the subsequent processing is repeated. On the other hand, when it is determined that all the suspicious person information rules have been referenced (step S1005; Yes), all the processes for searching and acquiring the suspicious person information are completed.

  In the above-described embodiment, the process for searching for and acquiring suspicious person information is described as being separate from the process in the first embodiment. For example, steps S703 to S706 in the first embodiment are described. By incorporating the deep case frame acquired by the language information acquisition unit 135 and the deep case frame searched by the suspicious person information determination unit 860, the suspicious person can be identified efficiently. It is also possible to do this.

  As described above, in the present embodiment, the suspicious person information database 851 relates to the action name of the object, the verb representing the kind of action of the object, and the action of the object among the case elements displayed by the deep case frame. The suspicious person information database 851 includes a suspicious person information database 851 in a keyword or a sentence that the input receiving part 181 receives input. When it is determined whether or not the value of the case element to be stored is included, the language information acquisition unit 135 determines that the suspicious person information determination unit 860 determines that the value of the case element is included in the keyword or sentence. Searches and acquires a deep case frame including the value of the case element from the deep case frames stored in the language information database 161, so that a person such as a desired suspicious person can be identified more efficiently. Ukoto can.

  It should be noted that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above embodiments. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

It is a block diagram which shows the functional structure of the security apparatus of Embodiment 1. FIG. It is a figure which shows a mode that the surveillance camera of Embodiment 1 images a surveillance area | region. It is a figure which shows the specific example of the image which the monitoring camera shown in FIG. 2 images. It is a figure which shows the structure of the deep layer frame which the language information database shown in FIG. 1 memorize | stores. It is a figure which shows the specific example of a structure of the deep layer frame shown in FIG. It is a figure which shows the specific example of the operation panel shown in FIG. 3 is a flowchart illustrating a procedure from searching for a deep case frame, an image, and the like to displaying them on an operation panel in the first embodiment. It is a block diagram which shows the functional structure of the security apparatus of Embodiment 2. FIG. It is a figure which shows the structure of a suspicious person information database shown in FIG. 10 is a flowchart illustrating a procedure from searching for a deep case frame including suspicious person information to displaying it on an operation panel in the second embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 800 Security apparatus 110 Sensor 120 Monitoring camera 130 830 Main body part 131 Detection information receiving part 132 Image processing part 133 Security state switching part 134 Operation panel control part 135 Language information acquisition part 136 Image information acquisition part 137 Transmission / reception part 138 Security state storage part 139 Language structure construction unit 140 Storage unit 150 First storage unit 151 Video database 160 Second storage unit 161 Language information database 170 Timer 180 Operation panel 181 Input reception unit 182 Language information display unit 183 Video display unit 190 Third storage unit 191 Face Feature database 850 Fourth storage unit 851 Suspicious person information database 860 Suspicious person information determination unit 200 Monitoring center 300 Network

Claims (9)

Image information storage means for storing image information of the object imaged by the imaging means;
Analyzing the image information to extract the motion state of the object, information representing the motion state of the extracted object, a verb representing the type of motion of the object, and a noun associated with the motion of the object State information storage means for storing state information represented by a case frame that includes an element and indicating a semantic relationship between the verb and the case element in association with the image information stored in the image information storage means When,
Input receiving means for receiving input of key information for searching for the image information stored in the image information storage means;
State information acquisition for searching and acquiring the state information including the key information from the state information stored in the state information storage unit based on the key information received by the input receiving unit Means,
Image information acquisition means for acquiring the image information corresponding to the state information searched by the state information acquisition means from among the image information stored in the image information storage means;
Display means for displaying the status information acquired by the status information acquisition means and the image information acquired by the image information acquisition means;
A search device characterized by. Face feature information storage means for storing the person name and the face feature information of the person name in association with each other;
The input receiving means receives an input of a person name as the case element as the key information;
The state information acquisition means determines whether or not the person name is included in the key information received by the input reception means,
If the state information acquisition unit determines that the person name is included in the key information and the person name is included in the state information, the image information acquisition unit includes: The face information corresponding to the person name to be stored is acquired, and the image information stored in the image information storage means in association with the state information including face feature information similar to the acquired face feature information To get the
The search device according to claim 1. The input receiving means further receives an input of a person's face image,
When the face feature information corresponding to the person name included in the key information is not stored in the face feature information storage unit, the image information acquisition unit receives the input received by the input receiving unit. Analyzing an image to extract face feature information and acquiring the image information stored in the image information storage means in association with the state information including face feature information similar to the extracted face feature information ,
The search device according to claim 2. The status information displayed by the case frame includes case information, which is attribute information indicating the characteristics of the object, the type of motion of the object, position information indicating the place where the motion is performed, and the motion An operation time including an operation start time when the operation is started and an operation end time when the operation is ended, operation target information indicating the target of the operation, a starting point at which the object starts operating with respect to the operation target information, and the Including an end point at which the movement is finished, and means information indicating an object used when the object moves with respect to the movement target information.
The search device according to any one of claims 1 to 3. The image information includes a start time at which imaging of the object is started and an end time at which imaging of the object is ended,
The state information acquisition means sets the start time and the end time included in the image information as the operation start time and the operation end time included in the state information,
The state information in which the operation start time and the operation end time set by the state information acquisition unit are set from the image information stored in the image information storage unit. Obtaining the image information corresponding to
The search device according to claim 4. A timer for measuring the elapsed time from the start time when the monitoring area was imaged,
The image information acquisition means determines whether the elapsed time measured by the timer has passed a predetermined time from the start time, and the elapsed time from the start time is a predetermined time. Is deleted, the image information corresponding to the state information including the start time is deleted from the image information storage means,
The search device according to claim 4, wherein: The attribute information indicating the characteristics of the object includes the figure of the person and the appearance of the person,
The input receiving means further receives input of the figure of the person or / and the appearance of the person,
The status information acquisition means includes the status information including the body shape of the person and / or the appearance of the person that the input reception means has received from among the status information stored in the status information storage means. Search and get,
The image information acquisition unit includes the body shape of the person and / or the appearance of the person included in the state information acquired by the state information acquisition unit from the image information stored in the image information storage unit. Obtaining the image information based on:
The search device according to any one of claims 4 to 6. Of the state information displayed by the case frame, the type of motion of the object, a verb representing the type of motion of the object, a case element that is a noun related to the motion of the object, and a value of the case element Suspicious person information storing means for storing
Suspicious person information determination means for determining whether or not the value of the case element stored in the suspicious person information storage means is included in the key information received by the input receiving means,
If the suspicious person information determination unit determines that the key information includes the value of the case element, the state information acquisition unit includes the state information stored in the state information storage unit. From searching and obtaining the state information including the value of the case element,
The search device according to claim 1, wherein: Image information storage means for storing image information of the object imaged by the imaging means, and extracting the operation state of the object by analyzing the image information, the information indicating the extracted operation state of the object, State information represented by a case frame including a verb representing a type of action and a case element which is a noun related to the action of the object, and representing a semantic relationship between the verb and the case element, the image information A search method performed in a search device comprising: state information storage means for storing the image information in association with the image information stored therein; and display means for displaying the state information and the image information,
An input receiving step for receiving an input of key information for searching the image information stored in the image information storage unit;
State information acquisition means searches the state information including the key information from the state information stored in the state information storage means based on the key information received by the input reception means. Status information acquisition step to acquire,
An image information acquisition unit for acquiring the image information corresponding to the state information searched by the state information acquisition unit from the image information stored in the image information storage unit; Including,
A search method characterized by

Images