JP6576287B2 - Movie processing apparatus and movie processing method - Google Patents

Description

  The present invention relates to a moving image processing apparatus and a moving image processing method.

  The technology for tracking the movement of a person from a camera image is used for crime prevention and marketing purposes. In order to track the movement of a person over a wide range, a plurality of cameras are required instead of a single camera. In consideration of the number of cameras and the installation cost, it is desirable to install a plurality of cameras so that the overlapping range of the imaging ranges of each camera is minimized.

  In the technology that tracks the movement of a person from video obtained from multiple cameras, when a person in the imaging range of one camera moves to the imaging range of another camera, without worrying about the imaging range of each camera, It is desirable to keep tracking that person as the same person. If the camera is a fixed camera and the internal parameters of each camera and external parameters such as the world coordinate system in the imaging range are known, the position of the person moving between the cameras is accurately calculated from the video of each camera The identity of the person can be determined based on the calculated position. However, this requires that the video of each camera be time synchronized. If there is a deviation in the video time, accurate calculation and determination becomes difficult.

  Conventionally, the method described in Patent Document 1 is known. In Patent Literature 1, a visual event is detected by dividing a video, and visual event detection information including information indicating the detection time of the visual event is generated.

International Publication No. 13/094115 Pamphlet

  The technique described in Patent Document 1 assumes that the video frame rate is constant. However, the present inventors have found that even when the frame rate is described in the camera specifications, the frame rate may fluctuate depending on the imaging target, performance, and network environment of the camera. In particular, surveillance cameras are designed with an emphasis on power consumption and durability, so the output frame rate often fluctuates. According to the experiment by the present inventor, in one example, several frames were lost for an imaging time of 30 minutes. One of the reasons for such frame loss is as follows. The video is generally compression-encoded and transmitted from the camera. However, when the motion is intense or there are many textures, this compression-encoding process takes time, and as a result, the frame may be lost.

  When frame loss occurs, even if time synchronization is performed using the technique described in Patent Document 1, the accuracy of processing such as moving object tracking based on comparison of images from different cameras may be reduced.

  The present invention has been made in view of these problems, and an object of the present invention is to provide a moving image processing technique that can appropriately cope with fluctuations in the number of unintended image processing units.

  One embodiment of the present invention relates to a moving image processing apparatus. This moving image processing apparatus includes an acquisition unit that acquires a first moving image and a second moving image from a first imaging device and a second imaging device that can detect the same event, respectively, and a first moving image acquired by the acquiring unit. To specify a first part corresponding to a period between the first event and the second event, and specify a second part corresponding to the period from the second moving image acquired by the acquiring part; A changing unit that changes the number of processing units included in the first part according to the number of processing units included in the two parts.

  It should be noted that any combination of the above-described constituent elements, or those obtained by replacing the constituent elements and expressions of the present invention with each other between apparatuses, methods, systems, computer programs, recording media storing computer programs, and the like are also included in the present invention. It is effective as an embodiment of

  According to the present invention, it is possible to appropriately cope with a change in the number of unintended image processing units.

It is a mimetic diagram of a surveillance system provided with a video processing device concerning an embodiment. It is explanatory drawing of the gap | deviation of the time of a moving image. It is a block diagram which shows the function and structure of the moving image processing apparatus of FIG. It is a schematic diagram which shows the frame structure of the 1st moving image and 2nd moving image which are hold | maintained at the process moving image holding | maintenance part of FIG. It is a schematic diagram which shows the 1st part specified from the 1st moving image, and the 2nd part specified from the 2nd moving image. It is a schematic diagram which shows the state by which the flame | frame was thinned out from the 2nd part of a 2nd moving image. It is a flowchart which shows a series of processes in the moving image processing device of FIG.

  Hereinafter, the same or equivalent components, members, and processes shown in the drawings are denoted by the same reference numerals, and repeated description is appropriately omitted. In addition, in the drawings, some of the members that are not important for explanation are omitted.

  In the moving image processing technology according to the embodiment, when the same object is in the imaging range of a plurality of imaging devices, a change in the state of the object is defined as an event, and the event in the moving image obtained from each imaging device and the next The number of frames between the events is calculated. When the calculated number of frames differs depending on the imaging device, processing for equalizing the number of frames is performed. Thereby, the moving object recognition / tracking process based on the result of the process of comparing the moving image obtained from the imaging device and the moving image obtained from another imaging device for each frame becomes easier, and the accuracy can be improved.

  FIG. 1 is a schematic diagram of a monitoring system 10 including a moving image processing apparatus 100 according to an embodiment. The monitoring system 10 includes a plurality of monitoring cameras and a moving image processing apparatus 100. The plurality of monitoring cameras are connected to the moving image processing apparatus 100 via a network. Each of the plurality of monitoring cameras is fixed to the ceiling or wall of the facility 12 and images a part of the facility 12. The moving image processing apparatus 100 tracks a person 16 moving in the facility 12 by analyzing moving images sent from a plurality of monitoring cameras. The surveillance camera may be a wide-angle camera such as an omnidirectional camera. Note that the monitoring system 10 is an example, and it is obvious to those skilled in the art who have touched this specification that the technical idea of the present embodiment can be applied to other systems including a plurality of imaging devices.

  An automatic door 14 is provided at the entrance of the facility 12. The first surveillance camera 18 and the second surveillance camera 20 of the plurality of surveillance cameras both include an automatic door 14 in the imaging range. Therefore, the change in the state of the automatic door 14, that is, the opening / closing operation appears in both the moving image from the first monitoring camera 18 and the moving image from the second monitoring camera 20.

  The times of a plurality of surveillance cameras are synchronized. However, when the moving image is compressed and encoded in the monitoring camera or the image is distributed via the network, a time difference occurs between the moving images from the plurality of monitoring cameras on the moving image processing apparatus 100 side, and the time of the moving image Deviation may occur. In many cases, this deviation is several seconds at the maximum. Also, frame defects can occur.

  FIG. 2 is an explanatory diagram of a time lag of a moving image. The horizontal axis indicates the time of the moving image processing apparatus 100. The moving images of the first monitoring camera 18 are continuously stored in a holding unit such as an HDD of the moving image processing apparatus 100. The same applies to the moving image of the second monitoring camera 20. The hatched portion is a portion imaged during the same predetermined period (for example, 10 seconds) in the real world. In an ideal situation with no deviation, the start times t1 and t3 of the part are the same. However, there is a difference between t1 and t3 due to the above deviation. The same applies to the end times t2 and t4. In such a situation, even if the moving image processing apparatus 100 cuts out portions of the moving image processing device 100 from time t5 to time t6 from each moving image, it is difficult to compare those portions as they are.

  FIG. 3 is a block diagram showing functions and configuration of the moving image processing apparatus 100 according to the present embodiment. Each block shown here can be realized by hardware and other elements such as a computer CPU and a mechanical device, and software can be realized by a computer program or the like. Draw functional blocks. Therefore, it is understood by those skilled in the art who have touched this specification that these functional blocks can be realized in various forms by a combination of hardware and software.

  The moving image processing apparatus 100 includes a moving image acquisition unit 302, an event detection unit 304, a time synchronization unit 306, a frame number adjustment unit 308, a moving object recognition unit 310, a moving object tracking unit 312, a live moving image holding unit 314, A processed moving image holding unit 316.

  The moving image acquisition unit 302 acquires a moving image from each of the plurality of monitoring cameras via a network. The moving image acquisition unit 302 stores the acquired moving image in the live moving image holding unit 314. The moving image acquired from the first monitoring camera 18 is referred to as a first moving image, and the moving image acquired from the second monitoring camera 20 is referred to as a second moving image. In the following, the case where moving body tracking is performed by analyzing the first moving image and the second moving image will be described. Alternatively or additionally, when moving body tracking is performed using a moving image from another surveillance camera. It will be apparent to those skilled in the art who have touched this specification that the technical idea according to the present embodiment can be applied.

  The event detection unit 304 analyzes the first moving image and the second moving image acquired by the moving image acquisition unit 302 and detects an event from each of them. The event includes a moment when the automatic door 14 opens and a moment when the automatic door 14 closes. The former is called an open event, and the latter is called a closed event. The event detection unit 304 includes an object recognition unit 318 and an event frame identification unit 320.

  The object recognition unit 318 detects an image region corresponding to the object, that is, the automatic door 14 from each frame of the first moving image. This detection process may be realized using a known image recognition technique. For example, the motion region of the entire image may be calculated by optical flow, and the region showing the periodic motion on the left and right may be estimated as the image region corresponding to the automatic door 14. Alternatively, since the surveillance camera is fixed and the image area corresponding to the automatic door 14 does not change frequently, the image area corresponding to the automatic door 14 may be set manually. The object recognizing unit 318 also performs the same process as described above for the second moving image, and detects an image area corresponding to the automatic door 14 from each frame of the second moving image.

  The event frame specifying unit 320 specifies the open event frame Eh corresponding to the open event and the closed event frame Et corresponding to the closed event from the first moving image based on the detection result in the object recognition unit 318. The event frame specifying unit 320 similarly specifies the open event frame Eh and the closed event frame Et for the second moving image. The opening event frame Eh is a frame when an opening event of the automatic door 14 occurs, and may be a frame located at a boundary between a frame where the closed state of the automatic door 14 is reflected and a frame where the opened state is reflected. The closing event frame Et is a frame when a closing event of the automatic door 14 occurs, and may be a frame located at a boundary between a frame in which the open state of the automatic door 14 is reflected and a frame in which the closed state is reflected. The event frame specifying unit 320 separately stores an image of the automatic door closed state, an image when the door starts to open, and an image immediately before closing, and realizes the above-mentioned opening / closing event frame by supervised learning processing. Also good. In addition to the above opening / closing events, the moment when the automatic door 14 is fully opened or the moment when it starts to close may be used as an event.

  The event frame specifying unit 320 assigns an identifier indicating that it is the open event frame Eh to the open event frame Eh specified from the first moving image. The event frame specifying unit 320 assigns an identifier indicating that it is the closed event frame Et to the closed event frame Et specified from the first moving image. The event frame specifying unit 320 stores the first moving image assigned with the identifier in the processed moving image holding unit 316. Alternatively, the event frame specifying unit 320 separately generates a table indicating which frame of the first moving image is the open event frame Eh and which frame is the closed event frame Et, and the generated table is used as the first moving image. It may be stored in the processed moving image holding unit 316 in association with. The event frame specifying unit 320 performs the same process on the second moving image, and stores the second moving image assigned with the identifier in the processed moving image holding unit 316.

  The time synchronization unit 306 corrects the time lag between the first moving image and the second moving image based on the event frame specified by the event frame specifying unit 320. For example, the time synchronization unit 306 may offset the entire first moving image in the time direction so that the time of the open event frame Eh of the first moving image matches the time of the open event frame Eh of the second moving image, May be deleted or a new frame may be added. The time synchronization unit 306 stores the corrected first moving image and second moving image in the processed moving image holding unit 316.

  FIG. 4 is a schematic diagram showing a frame configuration of the first moving image and the second moving image held in the processed moving image holding unit 316. The first moving image includes a series of frames including an open event frame Eh and a closed event frame Et. The open event frame Eh of the first moving image is the third frame from the left, and the closed event frame Et is the eleventh frame. The open event frame Eh of the second moving image is the third frame from the left, and the closed event frame Et is the twelfth frame. The first moving image and the second moving image shown in FIG. 4 are corrected so that the times of the open event frame Eh are aligned.

  Hereinafter, a description will be given with reference to FIGS. 3 and 4. The frame number adjustment unit 308 identifies a first part corresponding to the period between the open event and the closed event from the first video, identifies a second part corresponding to the same period from the second video, And according to the number of frames included in one of the second parts, the number of frames included in the other is changed. The frame number adjustment unit 308 includes a part specifying unit 322, a frame configuration analysis unit 324, an adjustment necessity determination unit 326, an adjustment content determination unit 328, and a frame configuration change unit 330.

  The part specifying unit 322 specifies an open event frame Eh corresponding to the open event and a closed event frame Et corresponding to the closed event from the first moving image based on the identifier given by the event frame specifying unit 320. The part specifying unit 322 specifies the first part 402 to be adjusted based on the specified open event frame Eh and the closed event frame Et. For example, the part specifying unit 322 specifies a frame group having the open event frame Eh as the start point and the closed event frame Et as the end point as the first part 402. In the example of FIG. 4, the first portion 402 includes the third frame to the eleventh frame. The first portion 402 includes a frame group sandwiched between the open event frame Eh and the closed event frame Et. The part specifying unit 322 specifies the second part 404 to be adjusted from the second moving image by the same process. In the example of FIG. 4, the second portion 404 includes from the third frame to the twelfth frame.

  FIG. 5 is a schematic diagram showing a first portion 402 specified from the first moving image and a second portion 404 specified from the second moving image. Hereinafter, a description will be given with reference to FIGS. 3 and 5. The frame configuration analysis unit 324 analyzes the frame configurations of the first part 402 and the second part 404 specified by the part specifying unit 322. The frame configuration analysis unit 324 calculates the number of frames included in the first portion 402 and each frame interval. The frame configuration analysis unit 324 calculates the number of frames included in the second portion 404 and each frame interval. In the example of FIG. 5, the frame configuration analysis unit 324 calculates 9 as the number of frames included in the first portion 402 and calculates 10 as the number of frames included in the second portion 404.

  The adjustment necessity determination unit 326 determines whether adjustment of the number of frames is necessary. The adjustment necessity determination unit 326 determines that the adjustment of the number of frames is unnecessary when the number of frames of the first portion 402 and the number of frames of the second portion 404 calculated by the frame configuration analysis unit 324 are equal. The adjustment necessity determination unit 326 determines that the number of frames needs to be adjusted when the numbers are different. Note that the determination criterion in the adjustment necessity determination unit 326 is not limited to being equal or not, and other criteria such as whether or not the difference is within a predetermined range may be employed.

  When the adjustment necessity determination unit 326 determines that the number of frames needs to be adjusted, the adjustment content determination unit 328 equals the number of frames included in the second portion 404 and the number of frames included in the first portion 402. The contents of adjustment are determined so that The adjustment content determination unit 328 (1) whether to change the first part 402 or the second part 404, (2) whether to remove an existing frame or add a new frame, (3) the place of change, To decide. For example, the adjustment content determination unit 328 first determines (2) based on the minimum frame interval and the maximum frame interval. When (2) is determined, (1) and (3) are automatically determined. In the example of FIG. 5, the adjustment content determination unit 328 first obtains the maximum frame interval Δmax of the first portion 402 with few frames and the minimum frame interval Δmin of the second portion 404 with many frames. Next, the adjustment content determination unit 328 selects one of the minimum frame interval Δmin and the maximum frame interval Δmax that is further away from the average value Δave of the frame intervals. When Δmax−Δave> Δave−Δmin, the adjustment content determination unit 328 selects the maximum frame interval Δmax, selects addition of a new frame, and selects between frames having the maximum frame interval Δmax as a place to add the frame. To do. When Δmax−Δave <Δave−Δmin, the adjustment content determination unit 328 selects the minimum frame interval Δmin, selects removal of an existing frame, and sets one of two frames having the minimum frame interval Δmin as a removal target. select. The above selection method corresponds to determining the adjustment contents so as to suppress the variation in the frame interval. Or the adjustment content determination part 328 may determine said (1), (2), (3) at random.

  The frame configuration changing unit 330 adjusts the number of frames with the content determined by the adjustment content determining unit 328. When the adjustment content determination unit 328 determines to add a new frame to the first portion 402, the frame configuration change unit 330 determines that there is an interval between frames having the maximum frame interval Δmax of the first portion 402 (the ninth frame and the 10th frame). Insert a new frame (between the second frame). At this time, the frame configuration changing unit 330 may use a copy of the ninth or tenth frame as a new frame, or may synthesize a new frame based on the ninth and tenth frames. . When the adjustment content determination unit 328 determines to exclude the frame from the second portion 404, the frame configuration changing unit 330 has two frames (the ninth and tenth frames) having the minimum frame interval Δmin of the second portion 404. ) Is thinned out.

  When a frame is added, information is not lost, so that the accuracy of the subsequent recognition process is improved, while a process for generating a frame is required. When thinning out frames, processing for frame generation is not necessary, but information corresponding to the thinned out frames is lost.

  FIG. 6 is a schematic diagram showing a state where frames are thinned out from the second portion 404 of the second moving image. Hereinafter, a description will be given with reference to FIGS. 3 and 6. The frame configuration changing unit 330 removes the tenth frame from the second portion 404 of the second moving image. As a result, the number of frames of the first portion 402 and the number of frames of the second portion 404 are equal to nine. The frame configuration changing unit 330 further performs processing for making the frame interval uniform. The frame configuration changing unit 330 stores the changed first part 402 and second part 404 in the processed moving image holding unit 316.

  The moving body recognizing unit 310 reads the first part 402 and the second part 404 obtained as a result of the change by the frame configuration changing unit 330 from the processed moving image holding unit 316. The moving object recognition unit 310 performs image recognition processing on each frame of the first portion 402 and the second portion 404 to recognize moving objects such as the person 16.

  The moving object tracking unit 312 performs moving object tracking based on the image recognition result in the moving object recognition unit 310. The moving body tracking unit 312 compares the image recognition result of the frame included in the first portion 402 with the image recognition result of the corresponding frame included in the second portion 404, and the moving body recognized in the former frame and the latter It is determined whether or not the moving object recognized in the frame is the same. The moving body tracking unit 312 tracks the movement of the moving body recognized to be the same over both the first portion 402 and the second portion 404.

  The operation of the moving image processing apparatus 100 having the above configuration will be described. FIG. 7 is a flowchart showing a series of processes in the moving image processing apparatus 100. The moving image processing apparatus 100 acquires the first moving image and the second moving image from the first monitoring camera 18 and the second monitoring camera 20 via the network (S702). The moving image processing apparatus 100 identifies the open event frame Eh and the closed event frame Et from the first moving image and the second moving image (S704). The moving image processing apparatus 100 synchronizes the times of the first moving image and the second moving image based on the open event frame Eh or the closed event frame Et specified in step S704 (S706). The video processing device 100 specifies the first part 402 corresponding to the period between the open event frame Eh and the closed event frame Et from the first video, and specifies the second part 404 corresponding to the same period from the second video. (S708). The moving image processing apparatus 100 calculates the number of frames included in the first portion 402 and the number of frames included in the second portion 404 (S710). The moving image processing apparatus 100 determines whether the two calculated numbers are equal (S712). If equal (Y in S712), the moving image processing apparatus 100 performs the moving object recognition / tracking process using the first portion 402 and the second portion 404 (S718). If they are not equal (N in S712), the moving image processing apparatus 100 determines a change content such as where / from which part a frame is added / deleted (S714). The moving image processing apparatus 100 adds / deletes a frame according to the content determined in step S714 (S716). The moving image processing apparatus 100 performs moving object recognition / tracking processing using the first portion 402 and the second portion 404 having the same number of frames (S718).

  In the embodiment described above, examples of the holding unit are a hard disk and a semiconductor memory. Further, based on the description of the present specification, each unit is configured by a CPU (not shown), a module of an installed application program, a module of a system program, a semiconductor memory that temporarily stores the content of data read from the hard disk, or the like. It will be appreciated by those skilled in the art who have touched this specification that this can be achieved.

  According to the moving image processing apparatus 100 according to the present embodiment, more accurate video synchronization is realized by detecting the movement of a reliable object such as the automatic door 14 even in an environment where the installation interval of the surveillance cameras is relatively wide. it can. In addition, even if the frame rate of the video is not constant due to frame loss, etc., the number of frames between events is adjusted so that the accuracy of frame-based image recognition and tracking processing can be maintained or increased. it can.

  The configuration and operation of the moving image processing apparatus 100 according to the embodiment has been described above. This embodiment is an exemplification, and it will be understood by those skilled in the art that various modifications can be made to each component and combination of processes, and such modifications are within the scope of the present invention.

  In the embodiment, the case where the opening / closing of the automatic door 14 is defined as an event has been described. However, the present invention is not limited thereto, and a change in the state of an object other than the automatic door 14 may be used as an event. For example, LED flashing may be used as an event. A TV receiver screen or a PC display screen may be used. Alternatively, on / off of the lighting of the facility 12 instead of the object may be used. In this case, the moment when the luminance of the background changes from light to dark and the moment when the background luminance changes from dark to light may be defined as events.

  Since the automatic door 14 is opened and closed irregularly, the automatic door 14 opening and closing events are included in a series of irregular opening and closing events. In the above embodiment, it is preferable that the intervals at which events occur are irregular. This is because when it occurs regularly, appropriate measures are required when the time is shifted by the period.

  In the embodiment, the case where time synchronization is performed by the time synchronization unit 306 has been described. However, the present invention is not limited to this, and time synchronization may not be performed.

  In the embodiment, the case of arranging the number of frames has been described. However, the present invention is not limited to this, and the number of arbitrary image processing units other than the frame may be arranged.

  In the embodiment, when the event detected in the first portion 402 cannot be detected in the second portion 404, the number of frames may not be synchronized with respect to the event.

  In the embodiment, when adding a frame, a case where a new frame is generated and added has been described. However, the present invention is not limited to this. For example, if the first part 402 and the second part 404 are later used for human recognition and tracking processing, instead of generating a new frame, the position where the person will be recognized in the new frame May be calculated and the calculation result may be retained. In this case, the processing amount can be reduced.

  In the embodiment, the case where the surveillance camera is fixed to the wall or ceiling of the facility 12 has been described. However, the present invention is not limited thereto, and the camera may be mounted on a surveillance robot or the like, or may be a mobile terminal such as a smartphone. A video from the camera may be used.

  In the embodiment, a case has been described in which the times of a plurality of surveillance cameras are synchronized. However, the present invention is not limited to this, and a video from a camera in a LAN environment not connected to the Internet, a video from a mobile or robot camera, or the like. The technical idea according to the embodiment may be applied to this process.

  10 monitoring systems, 12 facilities, 14 automatic doors, 100 video processing devices.

Claims (11)

An acquisition unit that acquires the first moving image and the second moving image from the first imaging device and the second imaging device that can detect the same event, respectively, via a network;
The first part corresponding to the period between the first event and the second event is identified from the first moving picture obtained by the obtaining unit, and the second part is obtained from the second moving picture obtained by the obtaining part. A specifying unit for specifying the second part to be performed;
Regarding the processing unit for moving object recognition or tracking based on the comparison between the first moving image and the second moving image, the number of processing units included in the second part is equal to the number of processing units included in the first part. And a changing unit for changing the number of processing units included in the first part.   The specifying unit specifies a first processing unit corresponding to the first event and a second processing unit corresponding to the second event from the first moving image acquired by the acquiring unit, and the specified The moving image processing apparatus according to claim 1, wherein the first portion is specified based on a first processing unit and the second processing unit. When the number of processing units included in the second part is greater than the processing units included in the first part, the changing unit adds a new processing unit to the first part, and the processing unit included in the second part is If less than the processing units included in the first portion, moving picture processing device according to claim 1 characterized in that except for the processing unit from the first portion. The changing unit, when adding a new processing unit to the first part, adds the new processing unit between two processing units having a relatively wide interval in the first part. The moving image processing apparatus according to claim 3 . The changing unit, when excluding the processing unit from said first portion, according to claim 3, characterized in that except at least one of the two processing units having a relatively narrow spacing among the first part Video processing device. Moving image processing apparatus according to any one of claims 1-5 wherein the first event and the second event, characterized in that included in a series of events that occur irregularly. A comparison unit that compares an image recognition result of the processing unit included in the first part obtained as a result of the change by the changing unit and an image recognition result of the corresponding processing unit included in the second part; moving image processing apparatus according to any one of claims 1 6, characterized in. The at least one of the first event and the second event is related to an object that exists in both the imaging range of the first imaging device and the imaging range of the second imaging device. 8. The moving image processing apparatus according to any one of items 1 to 7 . Moving image processing apparatus according to any one of claims 1, wherein the processing unit is a frame 8. Acquiring a first moving image and a second moving image from a first image capturing device and a second image capturing device that can detect the same event, respectively, via a network;
The first part corresponding to the period between the first event and the second event is specified from the acquired first video, and the second part corresponding to the period is specified from the acquired second video To do
Regarding the processing unit for moving object recognition or tracking based on the comparison between the first moving image and the second moving image, the number of processing units included in the second part is equal to the number of processing units included in the first part. And changing the number of processing units included in the first part. A function of acquiring a first moving image and a second moving image from a first image capturing device and a second image capturing device capable of detecting the same event, respectively, via a network;
The first part corresponding to the period between the first event and the second event is specified from the acquired first video, and the second part corresponding to the period is specified from the acquired second video Function to
Regarding the processing unit for moving object recognition or tracking based on the comparison between the first moving image and the second moving image, the number of processing units included in the second part is equal to the number of processing units included in the first part. A computer program for causing a computer to realize a function of changing the number of processing units included in the first part.

Images