JP7670321B2 - Image monitoring device, method and program - Google Patents

Description

This disclosure relates to an image monitoring device, method, and program.

In recent years, digital signage has come to be used as a new form of advertising and in-store display. By displaying images on a screen, digital signage can display multiple contents and videos on a single display, allowing for a wider range of uses than display methods that use media such as paper. However, problems can arise due to display malfunctions, such as the image you want to display not being displayed. Malfunctions can occur for a variety of reasons, including breakdowns in the display, the terminal device that controls the display, or network equipment.

Not only does the inability to display images or other content on a display reduce the value of advertising using the display, but when multiple displays are used to create a single display format, for example, even if only one display is not displaying content, there are problems such as the visual value of the advertisement being significantly reduced. For this reason, it is desirable to automatically detect the problem as soon as possible and notify the administrator.

Patent document 1 discloses a technique for detecting freezes and black images in a video by extracting images from the video at a predetermined interval. Patent document 2 discloses a technique for monitoring an image displayed on a display device to determine whether the image is actually being displayed.

JP 2019-212970 A JP 2011-135477 A

When extracting images from a video at a specified interval, that is, when detecting changes in the actual display using captured images, the actual display and the captured image do not necessarily match. This is because the screen is captured by the device performing the display processing, and the image is not necessarily displayed correctly on the display, and if the display is broken, the actual display and the captured image may not match. Furthermore, if there is an abnormality on the screen due to a breakdown in the device performing the display processing, it is not possible to generate a captured image.

The present disclosure has been made to solve these problems, and aims to provide an image monitoring device, method, and program that can automatically detect when the display no longer changes due to an abnormality in the device used to display the image on the display.

The image monitoring device according to the present disclosure includes an acquisition unit that acquires images periodically captured of a display used in a signage system, an identification unit that identifies a display image displayed on the display that is included in the acquired images, and a determination unit that determines the display state of the display based on a result of comparing the display images identified by the identification unit in each of a plurality of images captured at different times.

The image monitoring method disclosed herein includes the steps of acquiring images periodically captured of a display used in a signage system, identifying a display image displayed on the display that is included in the acquired images, and determining the display state of the display based on a comparison between the identified display images for each of a plurality of images captured at different times.

The image monitoring program disclosed herein causes a computer to execute the following processes: acquiring images periodically captured of a display used in a signage system; identifying a display image included in the acquired images and displayed on the display; and judging the display state of the display based on a comparison between the identified display images for each of a number of images captured at different times.

The present disclosure provides an image monitoring device, method, and program that can automatically detect when a display no longer changes due to an abnormality in the device used to display the image on the display.

1 is a configuration diagram of an image monitoring device according to a first embodiment of the present disclosure. FIG. 11 is a configuration diagram of a display abnormality detection device according to a second embodiment of the present disclosure. FIG. 11 is a diagram showing an overview of an image generated by performing a mask process on an image acquired from a camera according to a second embodiment of the present disclosure. FIG. 11 is a diagram showing an image of calculating a difference value using a comparison image obtained by multiplexing a mask image according to a second embodiment of the present disclosure. FIG. 11 is a diagram showing an example of installation of a camera for photographing a display according to a second embodiment of the present disclosure. FIG. 11 is a flow diagram of an abnormality determination process performed by an abnormality determination unit according to a second embodiment of the present disclosure.

The following describes the embodiments with reference to the drawings. Note that the drawings are simplified, and the technical scope of the embodiments should not be interpreted narrowly based on the descriptions in the drawings. Also, the same elements are given the same reference numerals, and duplicate explanations will be omitted.

<Embodiment 1>
The configuration of an image monitoring device 1 according to this embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram of an image monitoring device 1 according to this embodiment. The image monitoring device 1 comprises an acquisition unit 2, a specification unit 3, and a determination unit 4.

The acquisition unit 2 acquires images captured periodically of a display used in a signage system. "Periodic" also includes images not captured at regular time intervals. The identification unit 3 identifies a display image that is included in the images acquired by the acquisition unit 2 and is displayed on the display. The determination unit 4 determines the display state of the display based on the results of comparing the display images identified by the identification unit 3 for each of a plurality of images captured at different times.

The image monitoring device 1 in this embodiment can automatically detect when the display no longer changes due to an abnormality in the equipment used to display the image on the display.

<Embodiment 2>
The configuration of the display abnormality detection device 10 in this embodiment will be described with reference to Fig. 2. Fig. 2 is a configuration diagram of the display abnormality detection device 10 in this embodiment. The display abnormality detection device 10 includes an image acquisition unit 11, an image storage unit 12, a difference calculation unit 13, an abnormality determination unit 14, and a reporting unit 15, and acquires images captured by a camera 20. The display abnormality detection device 10 automatically detects that no changes have occurred in the image of a display 16 used in a signage system.

The image acquisition unit 11 acquires images captured by the camera 20 periodically on the display 16. "Periodically" may be, for example, the number of seconds at which the still images are switched when still images are switched on the display 16 every few seconds, or it may be set arbitrarily. In addition, "periodically" also includes not being at a fixed time interval. It is preferable to set the timing at which the image acquisition unit 11 acquires images from the camera 20 so that the same still image is not acquired more than twice.

The image acquisition unit 11 identifies the display image displayed on the display 16 that is included in the acquired image in order to eliminate the influence of images captured from outside the display area of the display 16. The identification method may involve performing a masking process on the image acquired from the camera 20. The image acquisition unit 11 superimposes a frame that surrounds the area including the display image displayed on the display 16 on the image of the display 16 captured by the camera 20, and masks the peripheral portion of the image of the display 16 from the frame.

Here, an overview of an image generated by performing mask processing on an image acquired from camera 20 will be described with reference to FIG. 3. FIG. 3 is a diagram showing an overview of an image generated by performing mask processing on an image acquired from camera 20. As shown in FIG. 3, image acquisition unit 11 generates a mask image by painting black the outside of a trapezoidal frame formed by connecting four preset points with straight lines. Note that the mask image is not limited to a shape connecting four points with straight lines, and various polygonal shapes may be used. The mask image generated in image acquisition unit 11 is used as a comparison image in difference calculation unit 13, as described later.

The image storage unit 12 stores the image acquired by the image acquisition unit 11 and displayed on the display 16 as a comparison image. The image storage unit 12 may store a comparison image that has been masked with respect to the image displayed on the display 16. The image storage unit 12 stores two images: the acquired comparison image and the comparison image acquired one time before that. When a new image is saved, the image storage unit 12 may erase the older of the two stored images.

The difference calculation unit 13 compares the display images on the display 16 contained in each of the two images stored in the image storage unit 12 to calculate a difference value. The difference value is the sum of the differences in pixel values obtained by calculating component values corresponding to each of the different colors in the pixels of the two display images. The pixel values may use each of the RGB color components. When the RGB values range from 0 to 255, for example, white is expressed as RGB = (255, 255, 255). The difference value may be calculated based on the difference for each pixel of the two images using the pixel values of each of the RGB color components. The difference value may also be calculated from all pixels in the comparison image, or may be calculated from only pixels extracted at equal intervals, for example.

Here, an overview of the process performed by the difference calculation unit 13 to calculate a difference value using a comparison image that has been subjected to mask processing will be described with reference to FIG. 4. FIG. 4 is a diagram showing an image of calculating a difference value using a comparison image that has been subjected to mask processing. As shown in FIG. 4, a comparison can be made to calculate a difference value for unmasked areas of the image acquired at the previous timing and the image acquired this time. This is because the difference value of the masked area will be 0.

The difference calculation unit 13 can calculate the difference value only for the unmasked areas included in the previous captured image and the current captured image. The difference value calculated by the difference calculation unit 13 is a value that excludes the influence of reflection of the image captured by the camera 20 onto the image displayed on the display 16.

The abnormality determination unit 14 determines whether the screen displayed on the display 16 is changing normally based on the difference value calculated by the difference calculation unit 13. The abnormality determination unit 14 notifies the notification unit 15 of the determination result. Details of the abnormality determination process performed by the abnormality determination unit 14 will be described later.

The reporting unit 15 receives notification from the abnormality determination unit 14 of the result of determining whether the screen displayed on the display 16 is changing normally, and reports the result. Various means of reporting can be used, such as textual means such as an email to the administrator, audio means, and display means.

An image of the installation of the camera 20 on the display 16 is shown in FIG. 5. As shown in FIG. 5, the camera 20 is installed facing the display 16 and can capture images displayed on the display 16. As shown in FIG. 5, the camera 20 captures images displayed on the display 16 directly from diagonally above the display 16, but it is not limited to a diagonally above direction, as long as it can capture images of the display 16 from various directions. Depending on the installation location of the display 16, it is possible that people may pass in front of the display 16. Therefore, in this embodiment, the camera 20 is fixed to the top of the display 16 and installed at an angle that looks down on the display 16 from diagonally above. The camera 20 may also be equipped with a polarizing filter on the lens to reduce reflections of lighting in the shooting environment on the display 16.

The abnormality determination process performed by the abnormality determination unit 14 will now be described with reference to FIG. 6. FIG. 6 is a flow diagram of the abnormality determination process performed by the abnormality determination unit 14. α is a difference value threshold for determining whether or not the screen displayed on the display 16 is changing normally. β is an abnormality count threshold, which is the number of times when the difference value is the threshold α, i.e., the number of times it is estimated that the image has not changed. The thresholds α and β can be set arbitrarily.

The abnormality determination unit 14 acquires the difference value calculated by the difference calculation unit 13 (step 101). When the difference value between the previously acquired image and the currently acquired image exceeds the threshold value α, the abnormality determination unit 14 determines that the screen has changed normally (NO in step 102) and clears the abnormality count number to 0 (step 103). When the calculated difference value is equal to or less than the threshold value α (YES in step 102), the abnormality determination unit 14 increments the abnormality count number by 1, assuming that the screen has not changed (step 104). When the abnormality count number is less than the threshold value β (NO in step 105), the abnormality determination unit 14 holds the current abnormality count number and waits until the next difference value is input. When the abnormality count number is equal to or greater than the threshold value β (YES in step 105), the abnormality determination unit 14 determines that an abnormality has occurred on the screen displayed on the display 16, notifies the notification unit 15 of the occurrence of the abnormality (step 106), and ends the process.

As described above, the display anomaly detection device 10 detects changes in the image displayed on the display 16 based on an image captured by the camera 20 installed outside the display 16. This makes it possible to prevent the processing of the display anomaly detection device 10 from affecting the display processing of the display 16 performed by other devices. Furthermore, the display anomaly detection device 10 detects anomalies based on the difference between the images captured by the camera 20. This means that the display anomaly detection device 10 does not need to store in advance the videos or still images displayed on the display 16, and does not require the effort of storing in advance a huge number of images to compare with the image actually displayed. Furthermore, the camera 20 does not need to have a high resolution, and does not need to have functions such as distortion correction.

The display abnormality detection device 10 in this embodiment can automatically detect when the display on the display 16 no longer changes due to an abnormality in the device used to display the information on the display 16.

<Embodiment 3>
The difference calculation unit 13 of the display abnormality detection device 10 in this embodiment divides an image into a plurality of regions, and for each divided region, compares the image acquired by the image acquisition unit 11 with the image stored in the image storage unit 12 to calculate a difference value. The abnormality determination unit 14 determines the presence or absence of a display abnormality on the display 16 based on the difference value calculated by the difference calculation unit 13.

The abnormality determination unit 14 therefore compares the calculated difference value with a difference threshold value α set for each corresponding region in the previously acquired image and the currently acquired image. The abnormality determination unit 14 determines that the image displayed on the display 16 has changed normally only when the differences for all regions included in each image exceed the threshold value α.

By configuring the display anomaly detection device 10 in this embodiment, it is possible to detect an anomaly even when an anomaly occurs in only a part of the image displayed on the display 16.

<Other embodiments>
The image monitoring device 1 in the present disclosure includes, for example, an embodiment as an image monitoring method. That is, the image monitoring method includes a step of acquiring images periodically captured of a display used in a signage system, a step of identifying a display image included in the acquired images and displayed on the display, and a step of determining a display state of the display based on a result of comparing the identified display images in each of a plurality of the images captured at different times.

In the above example, the program can be stored and supplied to the computer using various types of non-transitory computer readable media. Non-transitory computer readable media include various types of tangible storage media. Non-transitory computer readable media include, for example, magnetic recording media, optical magnetic recording media, CD-ROM (Read Only Memory), CD-R, CD-R/W, and semiconductor memory. Semiconductor memory is, for example, mask ROM, PROM (Programmable ROM), EPROM, flash ROM, RAM (Random Access Memory), etc. The program may also be supplied to the computer by various types of transitory computer readable media. Examples of transitory computer readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer readable media can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or via a wireless communication path.

The above program is an image monitoring program that causes a computer to execute the following processes: acquiring images periodically captured of a display used in a signage system; identifying display images contained in the acquired images and displayed on the display; and judging the display state of the display based on the results of comparing the identified display images for each of a number of images captured at different times.

Note that this disclosure is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit and scope of the present disclosure.

Reference Signs List 1 Image monitoring device 2 Acquisition unit 3 Identification unit 4 Determination unit 10 Display abnormality detection device 11 Image acquisition unit 12 Image storage unit 13 Difference calculation unit 14 Abnormality determination unit 15 Report unit 16 Display 20 Camera

Claims (7)

an acquisition unit that acquires images of a display periodically using a camera installed above the display at an angle that faces the direction of the display used in the signage system and looks down on the display from diagonally above ;
an identification unit that identifies a display image displayed on the display and that is included in the acquired image;
a determination unit that determines a display state of the display based on a result of comparing the display images identified by the identification unit in each of the plurality of images captured at different times;
Equipped with
the identification unit superimposes a trapezoidal frame having an upper base longer than a lower base, surrounding an area including the display image displayed on the display, on the captured image of the display, and performs mask processing on a peripheral portion of the image of the display from the frame.
Image surveillance equipment. the determination unit compares the display images included in the two images captured at different times to calculate a difference value, and determines the display state based on the calculated difference value.
2. The image monitoring device according to claim 1. The determination unit calculates the number of times that the difference value is equal to or less than a threshold, and determines that a display abnormality exists in the display when the number of times is equal to or greater than the threshold.
3. An image monitoring device according to claim 2. the determination unit calculates, as a difference value, a total value of differences between pixel values obtained by calculating component values corresponding to different colors in pixels in areas corresponding to the two display images;
4. An image monitoring device according to claim 2 or 3. the determination unit divides each of the two display images into a plurality of regions, and determines the presence or absence of a display abnormality on the display by calculating the difference value for each corresponding region.
An image monitoring device according to any one of claims 2 to 4. A step of periodically acquiring images of a display used in a signage system using a camera installed above the display at an angle looking down on the display from diagonally above so as to face the direction of the display;
Identifying a display image displayed on the display included in the acquired image;
determining a display state of the display based on a result of comparing the identified display images in each of the plurality of images captured at different times;
Equipped with
In the step of specifying, a trapezoidal frame having an upper base longer than a lower base and surrounding an area including the display image displayed on the display is superimposed on the captured image of the display, and a peripheral portion of the image of the display is masked from the frame.
Image monitoring method. A process of acquiring images of a display periodically taken by a camera installed above the display at an angle facing the direction of the display used in the signage system and looking down on the display from above;
A process of identifying a display image displayed on the display included in the acquired image;
determining a display state of the display based on a result of comparing the identified display images in each of the plurality of images captured at different times;
on the computer,
In the process of specifying, a trapezoidal frame having an upper base longer than a lower base and surrounding an area including the display image displayed on the display is superimposed on the captured image of the display, and a peripheral portion of the image of the display is masked from the frame.
Image surveillance program.

Images