JP7505192B2 - IMAGE PROCESSING APPARATUS, IMAGE PROCESSING PROGRAM, AND IMAGE PROCESSING METHOD - Google Patents

Description

The present invention relates to an image processing device, an image processing program, and an image processing method, and can be applied to, for example, an image processing device that determines the state of a subject based on an image.

In recent years, surveillance cameras have become more widespread, and technology for determining the condition of a subject using images has also come into practical use. For example, Patent Document 1 discloses a system that, in order to determine the condition of a subject, acquires a reference image that serves as a reference for assisting in the determination, separate from the measurement image, and determines the condition of the measurement image by comparing the reference image with the measurement image.

Figure 4 is a block diagram showing an example of a conventional image processing device as typified by Patent Document 1. The image processing device 100 shown in Figure 4 is composed of a reference image acquisition unit A1 that acquires and updates a reference image when an external trigger is received, a measurement image acquisition unit A2 that acquires a measurement image, and an image comparison unit A3 that calculates a status signal by comparing the reference image with the measurement image. The comparison between the reference image and the measurement image can be carried out by image analysis processing using a background subtraction method, as shown in Patent Document 1, for example.

As described above, by adopting the configuration shown in Figure 4, it is possible to determine through image analysis how the subject has changed since the reference image was captured.

JP 2008-77379 A

However, as shown in FIG. 5, when the image processing device 100 is installed in a closed space that can be opened and closed, and image analysis is used to determine how the subject has changed while being illuminated by light, a problem arises with the timing of acquiring the reference image.

For example, if the trigger is input using a button or the like attached to the image processing device 100, if the reference image is acquired when the closed space is in an open state, the measurement image will be acquired when it is in a closed state, and the difference between opening and closing, i.e., the difference in lighting conditions, will dominate the image changes, making it difficult to determine by analysis image changes that are smaller than the difference in lighting conditions.

One way to avoid this problem is to set a timer so that a trigger is input a certain amount of time after a button or the like is operated, but this method does not guarantee that a reference image will be acquired immediately after the camera is closed. Therefore, if a change occurs in the subject of interest to be judged immediately after the camera is closed, the reference image will be acquired after the change has occurred, making it impossible to detect the change. Conversely, if the waiting time set by the timer is not long enough, there is a possibility that a reference image will be acquired while the camera is still in the open state, even though the camera is not yet in the closed state.

Therefore, there is a need for an image processing device, an image processing program, and an image processing method that can reduce the problem of erroneous judgment of the subject's condition caused by comparing an undesired reference image as the standard.

The first invention is an image processing device that determines the state of a subject in a closed space that can be opened and closed based on a reference image while controlling an external device having at least an illumination unit and an imaging unit, characterized in that it comprises: (1) an external device control unit that controls the external device to acquire unilluminated data or an illuminated image; (2) an open determination unit that calculates a representative value of brightness from the unilluminated data acquired by the external device control unit and determines the open state if the representative value is greater than an open state threshold ; (3) a closed determination unit that calculates a representative value of brightness from the unilluminated data acquired by the external device control unit and determines the closed state if the representative value is less than a closed state threshold ; and (4) a reference image acquisition core unit that controls the external device control unit to acquire the illuminated image based on the determination results of the open determination unit and the closed determination unit, and outputs the illuminated image as the reference image.

The second image processing program of the present invention is characterized in that a computer mounted on an image processing device that determines the state of a subject in a closed space that can be opened and closed based on a reference image while controlling an external device having at least a lighting unit and a photographing unit functions as: (1) an external device control unit that controls the external device to acquire unilluminated data or an illuminated image; (2) an open judgment unit that calculates a representative value of brightness from the unilluminated data acquired by the external device control unit and determines an open state if the representative value is greater than an open state threshold ; (3) a closed judgment unit that calculates a representative value of brightness from the unilluminated data acquired by the external device control unit and determines a closed state if the representative value is less than a closed state threshold ; and (4) a reference image acquisition core unit that controls the external device control unit to acquire the illuminated image based on the judgment results of the open judgment unit and the closed judgment unit and outputs the illuminated image as the reference image.

The third invention is an image processing method used in an image processing device that determines the state of a subject in a closed space that can be opened and closed based on a reference image while controlling an external device having at least a lighting unit and a photographing unit, characterized in that (1) an external device control unit controls the external device to acquire unilluminated data or an illuminated image, (2) an open judgment unit calculates a representative value of brightness from the unilluminated data acquired by the external device control unit, and determines the open state if the representative value is greater than an open state threshold , (3) a closed judgment unit calculates a representative value of brightness from the unilluminated data acquired by the external device control unit, and determines the closed state if the representative value is less than a closed state threshold , and (4) a reference image acquisition core unit controls the external device control unit to acquire the illuminated image based on the judgment results of the open judgment unit and the closed judgment unit, and outputs the illuminated image as the reference image.
The fourth aspect of the present invention is an image processing device that determines a state of a subject in an openable and closable closed space based on a reference image while controlling an external device having at least a lighting unit and a photographing unit, the image processing device including: (1) an external device control unit that controls the external device to acquire unilluminated data or an illuminated image; (2) an open determination unit that calculates a representative value of brightness from the unilluminated data acquired by the external device control unit to determine an open state; (3) a closed determination unit that calculates a representative value of brightness from the unilluminated data acquired by the external device control unit to determine a closed state; and (4) a process that determines a closed state based on the determination results of the open determination unit and the closed determination unit. and a reference image acquisition core unit that controls the external device control unit to acquire the illuminated image and output the illuminated image as the reference image; (5) the external device control unit acquires the unilluminated data to be used in the open determination unit and the closed determination unit in response to a trigger that notifies of a sudden change in a physical phenomenon caused by light; and (6) after the open determination unit determines an open state and then the closed determination unit determines a closed state, the reference image acquisition core unit controls the external device control unit to acquire the illuminated image and output the illuminated image as the reference image.
The image processing program of the fifth invention includes a computer mounted on an image processing device that determines a state of a subject in an openable and closable closed space based on a reference image while controlling an external device having at least a lighting unit and an image capturing unit, the computer comprising: (1) an external device control unit that controls the external device to acquire no-illumination data or an image with illumination; (2) an open determination unit that calculates a representative value of brightness from the no-illumination data acquired by the external device control unit and determines an open state; (3) a closed determination unit that calculates a representative value of brightness from the no-illumination data acquired by the external device control unit and determines a closed state; and (4) the open determination unit and the closed determination unit. Based on the determination result, (5) the external device control unit controls the external device control unit to acquire the illuminated image and functions as a reference image acquisition core unit that outputs the illuminated image as the reference image, and (6) after the open determination unit determines an open state and the closed determination unit subsequently determines a closed state, the reference image acquisition core unit controls the external device control unit to acquire the illuminated image and output the illuminated image as the reference image.
A sixth aspect of the present invention is an image processing method used in an image processing device that determines the state of a subject in an openable and closable closed space based on a reference image while controlling an external device having at least an illumination unit and an image capture unit, wherein (1) an external device control unit controls the external device to acquire no-illumination data or an image with illumination, (2) an open determination unit calculates a representative value of brightness from the no-illumination data acquired by the external device control unit to determine an open state, (3) a closed determination unit calculates a representative value of brightness from the no-illumination data acquired by the external device control unit to determine a closed state, and (4) a reference image acquisition core unit controls the open state. (5) based on the judgment results of the determination unit and the closed judgment unit, the external device control unit is controlled to acquire the illuminated image and output the illuminated image as the reference image; (6) after the open judgment unit judges that the state is open and then the closed judgment unit judges that the state is closed, the reference image acquisition core unit controls the external device control unit to acquire the illuminated image and output the illuminated image as the reference image.

The present invention can reduce the problem of erroneous judgment of the subject's condition caused by comparing an unwanted reference image as the standard.

1 is a block diagram showing an example of an image processing apparatus according to an embodiment; 4 is a flowchart showing the overall operation of the image processing apparatus according to the embodiment. 10 is a flowchart illustrating a characteristic operation of a reference image acquisition unit according to the embodiment. FIG. 1 is a block diagram showing an example of a conventional image processing device. FIG. 1 is an explanatory diagram showing an example of the arrangement of a conventional image processing device.

(A) Main embodiment Hereinafter, an embodiment of an image processing device, an image processing program, and an image processing method according to the present invention will be described in detail with reference to the drawings.

(A-1) Configuration of the embodiment Fig. 1 is a block diagram showing an example of an image processing device according to the embodiment. The image processing device may be configured with hardware (for example, a dedicated semiconductor chip, etc.), or may be configured in part or in whole by software.

In FIG. 1, the image processing device 10 includes a reference image acquisition unit A1, a measurement image acquisition unit A2, and an image comparison unit A3, similar to the image processing device 100 in FIG. 4 described above. However, the function and internal configuration of the reference image acquisition unit A1 of the image processing device 10 differs from that of the image processing device 100, and the following detailed description will focus on these differences.

The image processing device 10 also includes a camera 20 and lighting 30 as external devices. The camera 20 and lighting 30 may be integrated into the image processing device 10. In addition, in this embodiment, it is assumed that the image processing device 10 is installed in a closed space that can be opened and closed, as shown in FIG. 5 above.

The reference image acquisition unit A1 of the image processing device 10 includes an external device control unit B11, an open determination unit B12, a closed determination unit B13, a reference image acquisition core unit B14, and a state variable V.

The external device control unit B11 has input/output functions for controlling external devices (camera 20 and lighting 30) and acquiring data such as images, and has a function for inputting a trigger that notifies of a sudden change in a physical phenomenon such as light and a trigger from the closed determination unit B13. The external device control unit B11 also has a function for outputting no-illumination data to the open determination unit B12 and the closed determination unit B13, and a function for outputting an illuminated image to the reference image acquisition core unit B14. The trigger is not limited to this and can be input, for example, from a human sensor.

The open determination unit B12 has a function of inputting no-illumination data from the external device control unit B11, a function of inputting an open state threshold from the outside, and a function of comparing a representative value of the no-illumination data with the open state threshold.

The closed determination unit B13 has a function of inputting no-illumination data from the external device control unit B11, a function of inputting a closed state threshold from the outside, and a function of comparing a representative value of the no-illumination data with the closed state threshold.

The reference image acquisition core unit B14 has the function of outputting the illuminated image input from the external device control unit B11 to the image comparison unit A3 as a reference image.

(A-2) Operation of the embodiment Next, the operation of the image processing device 10 according to this embodiment will be described in detail with reference to the drawings. Fig. 2 is a flowchart showing the overall operation of the image processing device according to the embodiment.

First, the image processing device 10 controls the external devices (camera 20, lighting 30) in the reference image acquisition unit A1 to acquire a reference image, and continues to provide the reference image to the image comparison unit A3 (S1).

Next, the measurement image acquisition unit A2 acquires an image using the camera 20 with the light 30 turned on, and outputs the acquired measurement image to the image comparison unit A3 (S2).

Then, the image comparison unit A3 compares the measurement image with the reference image and calculates a status signal (S3).

Next, the detailed operation of the reference image acquisition unit A1 (the detailed operation of step S1 described above) will be described. FIG. 3 is a flowchart showing the characteristic operation of the reference image acquisition unit according to the embodiment.

First, the external device control unit B11 initializes the state variable V by waiting for an open determination (S11-1).

Next, the external device control unit B11 waits until a trigger is input, and when the trigger is input, branches the process according to the state variable V (step S11-2). If the state variable V is waiting for an open determination, the external device control unit B11 proceeds to the next step S11-3, if the state variable V is waiting for a closed determination, the external device control unit B11 proceeds to step S11-4, which will be described later, and if the state variable V is waiting for a reference image to be acquired, the external device control unit B11 proceeds to step S11-5, which will be described later.

The external device control unit B11 obtains no-lighting data by controlling the external device and outputs it to the open determination unit B12 (step S11-3).

The open determination unit B12 calculates a representative brightness value from the no-illumination data, compares it with the open state threshold, and if it determines that the state is open, changes the state variable to waiting for a closed determination (step S12).

Here, the representative value is, for example, the average pixel value or median value found from the pixel values of the entire image or the pixel values of a part of the image. For example, when unilluminated data is compressed in a compression format such as JPEG or MPEG-4/AVC, it is the amount of compressed data. When the representative value is a statistical value of pixel values, the pixel values represent the intensity of light, so in closed spaces, the representative value will be a small value. Also, when the representative value is the amount of compressed data, in closed spaces, the entire image approaches solid black, in other words, the entropy of the image information is small, so the amount of data will also be a small value. Also, because of the above-mentioned tendency, the entropy of the image can be found from the pixel value sequence and used as the representative value. In this case as well, the representative value will be a small value when closed spaces are present.

If the representative value is the average pixel value or median value calculated from the pixel values of the entire image or part of the image, or if it is the amount of data or entropy, the representative value tends to be small in the case of a closed space, so the state is determined to be open if the representative value is higher than the open state threshold.

In addition, since the no-illumination data does not need to be a high-resolution image, an external device may be controlled to acquire images with lower resolution parameters than, for example, the reference image or measurement image. By keeping the resolution low, data transfer time can be reduced, resulting in lower power consumption.

On the other hand, if the external device control unit B11 determines in step S11-2 that the state variable V is waiting for a closed determination, it controls the external device to obtain no-illumination data and outputs it to the closed determination unit B13 (step S11-4).

The closed determination unit B13 calculates a representative brightness value from the no-illumination data, compares it with the closed state threshold, and if it determines that the state is closed, it changes the state variable V to waiting for acquisition of a reference image and outputs a trigger to the external device control unit B11 (step S13).

Specifically, the determination in step S13 can be made using the method described in step S12 above. However, in this step, since the closed state is sought instead of the open state, if the representative value is the average pixel value or median value found from the pixel values of the entire image or a portion of the image, or if it is the amount of data or entropy, the representative value tends to be small in the case of an enclosed space, so the state is determined to be closed when the representative value is lower than the closed state threshold value.

In addition, if the external device control unit B11 determines in step S11-2 described above that the state variable V is waiting for reference image acquisition, it controls the external device to acquire an illuminated image and outputs it to the reference image acquisition core unit B14 (step S11-5).

Then, the reference image acquisition core unit B14 changes the state variable to waiting for an open determination, updates the reference image to an illuminated image, and continues to provide the reference image to the image comparison unit A3 (S14).

(A-3) Effects of the embodiment As described above, according to the present embodiment, a reference image for the closed state can be obtained by acquiring an image with illumination immediately after confirming that the state is closed based on unilluminated data, thereby avoiding the problem of erroneous judgment caused by comparing an undesirable reference image as the standard.

In addition, the image processing device 10 of this embodiment confirms that the door is in the open state based on the no-illumination data before confirming that the door is in the closed state. This eliminates the problem of erroneously updating the reference image due to erroneous detection of a trigger that notifies of a sudden change in a physical phenomenon such as light when the door is actually in the closed state.

(B) Other Embodiments Although various modified embodiments have been mentioned in the above embodiment, the present invention can also be applied to the following modified embodiments.

(B-1) In the above embodiment, no-illumination data is used to determine whether the door is open or closed, and the no-illumination data is described assuming a no-illumination image. However, the no-illumination data may be determined using, for example, an illuminance sensor.

(B-2) In the above embodiment, the external device control unit B11 determines whether the device is in the closed state only when the trigger is input. However, due to erroneous detection of the trigger or erroneous determination by the closed determination unit B13, there is a risk that the device will never be determined to be in the closed state in step S13 and the reference image will never be updated. For this reason, the effects of the present invention may be more reliably achieved by setting a short-term timer in the processing of the open determination unit B12 and the closed determination unit B13, and outputting a trigger to the external device control unit B11 after a certain period of time has elapsed, as shown by the solid line (or dotted line) in Figure 2.

10, 100...image processing device, 20...camera, 30...lighting, A1...reference image acquisition unit, A2...measurement image acquisition unit, A3...image comparison unit, B11...external device control unit, B12...open judgment unit, B13...close judgment unit, B14...reference image acquisition core unit, V...state variable.

Claims (8)

An image processing device that determines a state of a subject in an openable and closable closed space based on a reference image while controlling an external device including at least a lighting unit and a photographing unit,
an external device control unit that controls the external device to acquire unilluminated data or illuminated images;
an open determination unit that calculates a representative value of brightness from the no-illumination data acquired by the external device control unit, and determines that the sensor is in an open state if the representative value is greater than an open state threshold ;
a close determination unit that calculates a representative value of brightness from the no-illumination data acquired by the external device control unit, and determines that the light source is in a closed state if the representative value is smaller than a closed state threshold value ;
and a reference image acquisition core unit that controls the external device control unit to acquire the illuminated image based on the judgment results of the open judgment unit and the closed judgment unit, and outputs the illuminated image as the reference image. An image processing device that determines a state of a subject in an openable and closable closed space based on a reference image while controlling an external device including at least a lighting unit and a photographing unit,
an external device control unit that controls the external device to acquire unilluminated data or illuminated images;
an open determination unit that calculates a representative value of brightness from the no-illumination data acquired by the external device control unit and determines an open state;
a close determination unit that calculates a representative value of brightness from the no-illumination data acquired by the external device control unit and determines a closed state;
a reference image acquisition core unit that controls the external device control unit to acquire the illuminated image based on the judgment results of the open judgment unit and the closed judgment unit, and outputs the illuminated image as the reference image;
Equipped with
the external device control unit acquires the no-illumination data used by the open determination unit and the close determination unit in response to a trigger notifying that a sudden change has occurred in a physical phenomenon caused by light;
an image processing apparatus characterized in that, after the open determination unit determines an open state and then the closed determination unit subsequently determines a closed state, the reference image acquisition core unit controls the external device control unit to acquire the illuminated image and output the illuminated image as the reference image. The image processing device according to claim 2, further comprising a trigger generating unit that outputs the trigger to the external device control unit after a predetermined time has elapsed after the determination process by the open determination unit. 4. The image processing apparatus according to claim 1, wherein the non-illuminated data has a lower resolution than the illuminated image. A computer installed in an image processing device that determines a state of a subject in an openable and closable closed space based on a reference image while controlling an external device having at least a lighting unit and a photographing unit,
an external device control unit that controls the external device to acquire unilluminated data or illuminated images;
an open determination unit that calculates a representative value of brightness from the no-illumination data acquired by the external device control unit, and determines that the sensor is in an open state if the representative value is greater than an open state threshold value ;
a close determination unit that calculates a representative value of brightness from the no-illumination data acquired by the external device control unit, and determines that the light source is in a closed state if the representative value is smaller than a closed state threshold value ;
an image processing program for controlling the external device control unit to acquire the illuminated image based on the judgment results of the open judgment unit and the closed judgment unit, and causing the external device control unit to function as a reference image acquisition core unit that outputs the illuminated image as the reference image. An image processing method for use in an image processing device that determines a state of a subject in an openable and closable closed space based on a reference image while controlling an external device having at least a lighting unit and a photographing unit,
The external device control unit controls the external device to acquire unilluminated data or illuminated images;
an open determination unit that calculates a representative value of brightness from the no-illumination data acquired by the external device control unit, and determines an open state if the representative value is greater than an open state threshold value ;
a close determination unit that calculates a representative value of brightness from the no-illumination data acquired by the external device control unit, and determines that the light source is in a closed state if the representative value is smaller than a closed state threshold value ;
a reference image acquisition core unit controls the external device control unit to acquire the illuminated image based on the judgment results of the open judgment unit and the closed judgment unit, and outputs the illuminated image as the reference image. A computer installed in an image processing device that determines a state of a subject in an openable and closable closed space based on a reference image while controlling an external device having at least a lighting unit and a photographing unit,
an external device control unit that controls the external device to acquire unilluminated data or illuminated images;
an open determination unit that calculates a representative value of brightness from the no-illumination data acquired by the external device control unit and determines an open state;
a close determination unit that calculates a representative value of brightness from the no-illumination data acquired by the external device control unit and determines a closed state;
a reference image acquisition core unit that controls the external device control unit to acquire the illuminated image based on the judgment results of the open judgment unit and the closed judgment unit, and outputs the illuminated image as the reference image;
and make it work.
the external device control unit acquires the no-illumination data used by the open determination unit and the close determination unit in response to a trigger notifying that a sudden change has occurred in a physical phenomenon caused by light;
After the open determination unit determines an open state and the close determination unit subsequently determines a closed state, the reference image acquisition core unit controls the external device control unit to acquire the illuminated image and outputs the illuminated image as the reference image.
2. An image processing program comprising: An image processing method for use in an image processing device that determines a state of a subject in an openable and closable closed space based on a reference image while controlling an external device having at least a lighting unit and a photographing unit,
The external device control unit controls the external device to acquire unilluminated data or illuminated images;
an open determination unit that calculates a representative value of brightness from the no-illumination data acquired by the external device control unit and determines an open state;
a closed state determination unit that calculates a representative value of brightness from the no-illumination data acquired by the external device control unit and determines a closed state;
a reference image acquisition core unit that controls the external device control unit to acquire the illuminated image based on the determination results of the open determination unit and the closed determination unit, and outputs the illuminated image as the reference image;
the external device control unit acquires the no-illumination data used by the open determination unit and the close determination unit in response to a trigger notifying that a sudden change has occurred in a physical phenomenon caused by light;
After the open determination unit determines an open state and the close determination unit subsequently determines a closed state, the reference image acquisition core unit controls the external device control unit to acquire the illuminated image and outputs the illuminated image as the reference image.
13. An image processing method comprising:

Images