JP7560950B2 - Image processing system and control program - Google Patents

Description

The present invention relates to an image processing system and control program for detecting human gestures.

In recent years, image processing systems have been developed that detect human gestures based on images captured in a monitored space.

Patent document 1 discloses an information input device that emits light with varying intensity toward an object, separates the reflected light from the object from external light, and detects an image of the reflected light from the object.

Patent document 2 discloses a portable computer that recognizes the reciprocating motion of an object at a predetermined cycle. The portable computer receives multiple successive frames from an image sensor that captures an image of the object. The portable computer then compares the background image with the blurred image contained in each frame, calculates the difference in the gradation values of corresponding pixels, and recognizes the object's motion from the binarized difference image.

Japanese Patent Application Publication No. 10-177449 Patent No. 5782061

In an image processing system, it is desirable to accurately detect gestures made by people in a monitored space.

The problem that this invention aims to solve is to provide an image processing system and control program that generates an image in which the movement of a moving object in a captured image is easily recognized. Also, to provide an image processing system and control program that can accurately recognize a gesture made by a person in a monitored space with their hand out in front of them.

In order to solve the above-mentioned problems, as one aspect of the present invention, an image processing system is provided that includes: a distance image acquisition means for sequentially acquiring distance images whose gradation values are information relating to the distance from a reference position to an object in a monitored space; a two-dimensional image acquisition means for sequentially acquiring two-dimensional images whose gradation values are information relating to the shading in the monitored space corresponding to the sequentially acquired distance images; and a synthesis means for identifying, for each group of pixels or regions arranged at the same position in a plurality of distance images acquired by the distance image acquisition means over a predetermined period of time, a pixel or region having a relatively small gradation value in the group, and generating a processed image in which a plurality of two-dimensional images acquired over a predetermined period of time are synthesized using a pixel or region corresponding to the identified pixel or region in a two-dimensional image corresponding to the distance image including the pixel or region identified for each group.

In the above image processing system, it is preferable that the synthesis means generates a processed image by using the gradation value of a pixel or area corresponding to a specified pixel or area in a two-dimensional image corresponding to a distance image including the pixels or areas specified for each group as the gradation value of the pixel or area corresponding to that group.

In the above image processing system, it is preferable that the synthesis means generates a processed image such that the gradation value of a pixel or region corresponding to a specified pixel or region in a two-dimensional image corresponding to a distance image including the pixels or regions specified for each group is set to the gradation value of the first component of the pixel or region corresponding to that group, and the gradation value of a pixel or region specified for each group in the distance image is set to the gradation value of the second component of the pixel or region corresponding to that group.

In the above image processing system, it is preferable that the system further includes a detection means for inputting the processed image into a model trained to output information about the gesture movements of a person contained in the input learning processed image, and detecting the gesture movements of a person in the monitored space based on the information output from the model.

In the above image processing system, it is preferable that the system further includes a person area detection means for detecting a person area including a person in the distance image or the two-dimensional image, and the synthesis means generates a processed image using pixels or areas corresponding to the identified pixels or areas in the two-dimensional image corresponding to the distance image including the pixels or areas identified for each group, only for the groups corresponding to the detected person areas.

In the above image processing system, it is preferable that the system further includes a detection means for detecting a predetermined gesture movement made in front of a person present in the monitored space based on the processed image.

In addition, in order to solve the above-mentioned problems, in another aspect, the present invention provides an image processing system characterized by having a distance image acquisition means for sequentially acquiring distance images in which information relating to the distance from a reference position to an object in a monitored space is used as a gradation value, and a processed image generation means for identifying, for each group of pixels or regions located at the same position in multiple distance images acquired by the distance image acquisition means over a predetermined period of time, a pixel or region with a relatively small gradation value within the group, and using the pixels or regions identified for each group, synthesizing the multiple distance images acquired over the predetermined period of time to generate a processed image.

In addition, in order to solve the above-mentioned problems, in another aspect, the present invention provides a control program that causes a computer to execute the following steps: sequentially acquire distance images in which information relating to the distance from a reference position to an object in a monitored space is used as a gradation value; sequentially acquire two-dimensional images in which information relating to the shading in the monitored space is used as a gradation value corresponding to the sequentially acquired distance images; identify, for each group of pixels or regions arranged at the same position in a plurality of distance images acquired over a predetermined period, a pixel or region in the group that has a relatively small gradation value; and generate a processed image by synthesizing the plurality of two-dimensional images acquired over the predetermined period using pixels or regions corresponding to the identified pixels or regions in the two-dimensional images corresponding to the distance images including the pixels or regions identified for each group.

In order to solve the above-mentioned problems, in another aspect, the present invention provides a control program that causes a computer to execute the following steps: sequentially acquire distance images in which information relating to the distance from a reference position to an object in a monitored space is used as a gradation value; for each group of pixels or regions located at the same position in multiple distance images acquired over a specified period, identify pixels or regions with relatively small gradation values within the group; and use the pixels or regions identified for each group to synthesize the multiple distance images acquired over the specified period to generate a processed image.

The present invention provides an image processing system and control program that generates an image in which the movement of a moving object in a captured image is easily recognized. It also provides an image processing system and control program that can accurately recognize a gesture made by a person in a monitored space with their hand out in front of them.

FIG. 1 is a block diagram of an image processing system. 4 is a flowchart showing the operation of the image processing system. FIG. 2 is a diagram for explaining a correspondence relationship between pixels in each processed image. FIG. 11 is a conceptual diagram for explaining a processed image. FIG. 11 is a conceptual diagram for explaining a processed image. 1 is an example of a processed image. 1 is an example of a processed image. 1 is an example of a processed image.

Various embodiments of the present invention will be described below with reference to the drawings. However, please note that the technical scope of the present invention is not limited to these embodiments, but extends to the inventions described in the claims and their equivalents. In addition, in each drawing, parts having the same or equivalent functions are given the same reference numerals, and their description may be omitted or simplified.

(Overview of image processing system 1)
1 is a block diagram of an image processing system 1. The image processing system 1 is used for watching over a person in a monitored space, such as a hospitalized patient or a person receiving care, and detects a gesture such as a hand wave made by the person being watched over as a gesture, and notifies an external device used by the person watching over the gesture. The image processing system 1 includes an imaging device 2, a distance sensor 3, an image processing device 4, and the like.

The imaging device 2 is an example of an image generating means, and sequentially generates two-dimensional images by capturing images of the monitored space. A two-dimensional image is an image in which multiple pixels, whose gradation values are information about the shading in the monitored space (such as brightness values or color values), are arranged in two dimensions. The imaging device 2 includes a light emitter, a two-dimensional detector, an imaging optical system, an A/D converter, and the like. The light emitter irradiates the monitored space with near-infrared light having a wavelength of, for example, about 890 nm. The two-dimensional detector includes a photoelectric converter that is sensitive to near-infrared light, such as a CCD (Charge-Coupled Device) element or a C-MOS (Complementary MOS). The imaging optical system forms an image of the monitored location on the two-dimensional detector. The A/D converter amplifies the electrical signal output from the two-dimensional detector and converts it to analog/digital (A/D).

The imaging device 2 captures images of the monitored space while irradiating the light emitter with near-infrared light at regular time intervals (e.g., 1/30 seconds), generates a near-infrared image as a two-dimensional image in which each pixel has a brightness value representing the intensity of the near-infrared light as a gradation value, and outputs it to the image processing device 4. Since humans cannot directly see near-infrared light, the imaging device 2 does not affect the vision of people in the monitored space. Therefore, when monitoring, for example, a hospitalized patient or a person receiving care, the image processing system 1 can monitor the patient without disturbing the patient or person receiving care while they sleep.

The two-dimensional detector may have a photoelectric converter that is sensitive to visible light, and generate a two-dimensional visible light image in which each pixel has a luminance value, RGB value, or CMY value of visible light as a gradation value. In this case, the light emitter may be omitted.

The distance sensor 3 is an example of a distance image generating means, and generates distance images sequentially. The distance image is an image in which a plurality of pixels, whose gradation values are information about the distance from a reference position in the monitored space to the corresponding position of the object, are arranged two-dimensionally. The reference position is the arrangement position of the distance sensor 3. The distance sensor 3 irradiates near-infrared rays toward the shooting range of the imaging device 2 at a timing shifted from the timing at which the light emitter of the imaging device 2 irradiates near-infrared rays every time the imaging device 2 performs shooting. The distance sensor 3 sequentially irradiates each position in the monitored space corresponding to each pixel in the two-dimensional image. For example, the distance sensor 3 divides the shooting range of the imaging device 2 in the horizontal and vertical directions at equal intervals by the number of pixels in the horizontal and vertical directions of the two-dimensional image, and sets the position in each divided area as the position corresponding to each pixel in the two-dimensional image. The distance sensor 3 receives a reflected signal arriving along the scanning direction in which the exploration signal is irradiated, and generates a received light signal having a value according to the intensity of the reflected signal.

Based on the phase information of the exploration signal, the angle information indicating the direction in which the exploration signal is currently being irradiated, and the received light signal, the distance sensor 3 measures the distance from the distance sensor 3 to the object that reflected the reflected signal for each scanning direction, and generates distance measurement data indicating the relationship between the scanning direction and the distance. For example, the distance sensor 3 calculates the difference between the phase of the reflected signal obtained from the received light signal and the phase of the exploration signal according to the time of flight method, and measures the distance based on the difference. The distance sensor 3 generates a distance image in which a value corresponding to the distance corresponding to each scanning direction shown in the distance measurement data is set as the gradation value of the pixel corresponding to each scanning direction, and outputs the image to the image processing device 4. For example, the distance sensor 3 divides a predetermined distance range (for example, 0.5 m to 7 m) into 256 equal steps and assigns each value from 0 to 255. The distance sensor 3 sets the value assigned to the category to which the distance corresponding to each scanning direction shown in the distance measurement data belongs as the gradation value of the pixel corresponding to each scanning direction. Each gradation value is set so that the shorter the distance to the corresponding object, the smaller the gradation value, and the longer the distance to the corresponding object, the larger the gradation value.

In addition, the distance sensor 3 may measure distance according to other known methods, such as a TOF/LiDAR method that irradiates near-infrared light, millimeter waves, laser, etc. and measures the time it takes for the light to reflect off an object and return, or a method that performs triangulation using a stereo camera, etc.

In this way, the distance sensor 3 sequentially generates distance images corresponding to the sequentially generated two-dimensional images. That is, the imaging device 2 sequentially generates two-dimensional images corresponding to the sequentially generated distance images.

The imaging device 2 and distance sensor 3 may be placed apart to capture and measure. In this case, the processing unit 12 corrects the two-dimensional image or distance image so that pixels corresponding to the same position in the monitored space are positioned at the same position in the two-dimensional image and distance image. The image processing device 4 stores in advance in the memory unit 9 a table showing the relationship between each pixel in the two-dimensional image and distance image, and the processing unit 12 corrects the image by referring to the table stored in the memory unit 9.

Furthermore, a part or all of the imaging device 2 and the distance sensor 3 may be used in common. For example, the imaging device 2 and the distance sensor 3 may generate a two-dimensional image and a distance image using a common light emitter and/or light receiver.

The image processing device 4 is a general computer such as a desktop computer, a workstation, or a notebook computer. The image processing device 4 has an interface unit 5, an input unit 6, a display unit 7, a communication unit 8, a memory unit 9, a processing unit 12, and a data bus B.

The interface unit 5 has an interface circuit for performing data communication with the imaging device 2 and the distance sensor 3, and is electrically connected to the imaging device 2 and the distance sensor 3 to transmit and receive various control signals or image signals. Note that the image processing device 4 may also have the imaging device 2 and the distance sensor 3.

The input unit 6 has an input device (such as a keyboard or mouse) and an interface circuit that acquires signals from the input device, and accepts input operations from an operator who operates the image processing device 4.

The display unit 7 has a display such as a liquid crystal display or an organic EL (Electro-Luminescence) display, and an interface circuit that outputs image data to the display, and displays various information on the display.

The communication unit 8 is an example of an output means, and has a communication interface circuit that complies with, for example, TCP/IP, and connects to a communication network such as the Internet. The communication unit 8 outputs data received from the communication network to the processing unit 12, and transmits data input from the processing unit 12 to the communication network.

The storage unit 9 has semiconductor memory such as ROM and RAM, and a magnetic disk or an optical disk drive such as a CD-ROM or DVD-ROM, and a recording medium thereof. The storage unit 9 also stores a control program and various data for controlling the image processing device 4, and inputs and outputs this information between the processing unit 12. A computer program may be installed into the storage unit 9 from a computer-readable portable recording medium such as a CD-ROM or DVD-ROM using a known setup program or the like. The storage unit 9 also stores a model 10 and a background image 11 as data.

Model 10 is a judgment model that has been pre-trained to output an evaluation value that indicates the likelihood that an input image contains a gesture to be detected. The evaluation value is set to be higher the more likely the image contains a gesture to be detected.

The background image 11 is a two-dimensional image generated by photographing an unoccupied monitored space. The background image 11 may be updated periodically or as needed when it is determined that no people are present in the monitored space.

The processing unit 12 has a processor such as a CPU or MPU, memories such as a ROM or RAM, and peripheral circuits, and executes various signal processing of the image processing device 4. Note that a DSP, an LSI, an ASIC, an FPGA, etc. may be used as the processing unit 12. The processing unit 12 has a distance image acquisition means 13, a two-dimensional image acquisition means 14, a person area detection means 15, an extraction means 16, a processed image generation means 17, a detection means 18, an output control means 19, a learning means 20, etc.

(Gesture Detection Operation of Image Processing System 1)
2 is a flowchart showing an operation sequence of the image processing system 1. This operation sequence is executed mainly by the processing unit 12 in cooperation with each element of the image processing device 4, based on a control program stored in the storage unit 9. This operation sequence is executed at each time interval when a distance image and a two-dimensional image are generated.

First, the distance image acquisition means 13 acquires the latest distance image generated by the distance sensor 3 (step S1). The distance image acquisition means 13 sequentially acquires distance images in which information relating to the distance from a reference position to an object in the monitored space is used as a gradation value. The distance image acquisition means 13 stores the acquired distance images in the memory unit 9 in association with the time of acquisition.

Then, the two-dimensional image acquisition means 14 acquires the latest two-dimensional image generated by the imaging device 2 (step S2). The two-dimensional image acquisition means 14 sequentially acquires two-dimensional images whose gradation values are information about the shading in the monitored space, corresponding to the sequentially acquired distance images. The two-dimensional image acquisition means 14 stores the acquired two-dimensional images in the memory unit 9 in association with the time of acquisition. In this way, the distance image acquisition means 13 sequentially acquires the distance images sequentially generated by the distance sensor 3, and the two-dimensional image acquisition means 14 sequentially acquires the two-dimensional images sequentially generated by the imaging device 2 corresponding to the distance images sequentially generated by the distance sensor 3.

Next, the person area detection means 15 detects a person area that includes a person within the two-dimensional image (step S3).

The person area detection means 15 calculates the absolute value of the difference between the gradation value of each pixel in the two-dimensional image and the gradation value of each corresponding pixel in the background image 11 stored in the memory unit 9, and extracts the pixel area where the calculated absolute value of the difference is equal to or greater than a predetermined threshold as a difference area. The person area detection means 15 groups difference areas caused by the same object by labeling, and detects them as changed areas. That is, the person area detection means 15 groups adjacent (8-connected) pixels from the difference areas extracted from a single two-dimensional image, combines groups that are close to each other (located within a predetermined range) based on their size or positional relationship, and combines the combined areas as changed areas.

The person area detection means 15 may detect changed areas using inter-frame differences. In this case, the person area detection means 15 calculates the absolute value of the difference between the luminance value of each pixel in the latest two-dimensional image and the luminance value of each corresponding pixel in the immediately preceding two-dimensional image, and extracts, as a difference area, a pixel area where the absolute value of the calculated difference is equal to or greater than a predetermined threshold value.

Then, the person area detection means 15 judges whether the object in the changed area is likely to be a person based on the feature quantities such as the size and aspect ratio of the changed area. The person area detection means 15 judges whether the object in the changed area is likely to be a person based on whether the size of the changed area is within a predetermined range corresponding to the size of a person and whether the aspect ratio of the changed area is within a predetermined range corresponding to the aspect ratio of a person. The size of each changed area is converted to an actual size using the position in the two-dimensional image and the installation information of the imaging device 2 stored in the memory unit 9. If the changed area is likely to be a person, the person area detection means 15 detects the changed area as a person area.

Note that the person area detection means 15 may detect a person area in the distance image in the same manner as when detecting a person area in a two-dimensional image. The person area detection means 15 may also detect a person area included in an image according to a judgment model. In this case, the image processing device 4 stores in the storage unit 9 a judgment model trained using a plurality of learning images including people, for example, by a known machine learning technique such as deep learning. The judgment model is pre-trained so that when a learning image is input, the position of the person area included in the learning image is output. As the machine learning technique, for example, a multi-layered neural network composed of an input layer, a plurality of intermediate layers, and an output layer is used. A learning image is input to the input layer. Each node in the intermediate layer extracts a feature vector from the image output from each node in the input layer, and outputs the sum of values obtained by multiplying each extracted feature vector by a weight. The output layer outputs the sum of values obtained by multiplying each feature vector output from each node in the intermediate layer by a weight. The judgment model learns while adjusting each weight so that the difference between the output value from the output layer and the position of the person area included in the learning image becomes small. The person area detection means 15 inputs a two-dimensional image or a distance image into a judgment model, and detects a person area in the two-dimensional image or the distance image from the output value output from the judgment model.

Next, the extraction means 16 extracts the pixel with the smallest gradation value in each group of pixels that are arranged at the same position in a predetermined number of distance images generated during a predetermined period (step S4). The predetermined number is 2 or more, for example 10.

The extraction means 16 reads out a predetermined number of the most recent distance images from the distance images stored in the memory unit 9. The extraction means 16 groups pixels that are located at the same position within an area of each read distance image that corresponds to a person area detected by the person area detection means 15. That is, each group contains a predetermined number of pixels (the same number as the read distance image). The extraction means 16 may also group pixels that are located at the same position within the entire area of each read distance image. The extraction means 16 extracts the pixel with the smallest gradation value in each group, i.e. the pixel with the shortest distance to the corresponding object.

The extraction means 16 may extract the pixel with the smallest gradation value in each group only from pixels whose gradation value is equal to or less than the threshold value. The extraction means 16 may also extract the pixel with the smallest gradation value in each group only from pixels in which an object located in front of the background and the person is captured. In this case, the image processing device 4 stores in advance in the storage unit 9 a background distance image generated by measuring the distance in an unmanned monitored space. The extraction means 16 extracts the pixel with the smallest gradation value in each group only from pixels in the distance image that have a gradation value smaller than the gradation value of the corresponding pixel in the background distance image. Furthermore, the extraction means 16 extracts the pixel with the smallest gradation value in each group only from pixels in the distance image that have a gradation value smaller than the average gradation value of each pixel in the distance image corresponding to the person area detected by the person area detection means 15 a predetermined time ago (for example, the distance image immediately before the predetermined number of distance images read out by the extraction means 16). If there is no pixel in each group whose gradation value is equal to or less than the threshold value, the extraction means 16 extracts a predetermined pixel (e.g., a pixel in the latest distance image) instead of the pixel with the smallest gradation value. This allows the extraction means 16 to exclude pixels in which a moving background (e.g., plants swaying in the wind) is captured from the extraction target. As a result, the image processing system 1 can detect a person's gestures with greater accuracy. Note that the average gradation value of each pixel in the distance image corresponding to the person region may be the average gradation value of the upper body (upper half) or head region of the person region, rather than the average gradation value of the entire person region.

Note that the pixel extracted by the extraction means 16 for each group does not have to be the pixel with the smallest gradation value. For example, when extracting pixels, the extraction means 16 may extract pixels with relatively small gradation values in the group. For example, the extraction means 16 extracts pixels with relatively small gradation values, such as a predetermined (second or third) smallest gradation value, rather than the smallest gradation value in the group. For example, if there are a predetermined number or more pixels (for example, differences between three of the pixels on the top, bottom, left and right sides of the pixel to be extracted that are greater than or equal to a predetermined value) that have a difference greater than or equal to a predetermined value with respect to the surrounding pixels (for example, differences between the three pixels on the top, bottom, left and right sides of the pixel to be extracted that are greater than or equal to a predetermined value), the pixel to be extracted is the second or third pixel, or a pixel with a relatively small gradation value, such as the median or average value of the surrounding pixels with similar gradation values. In this way, for example, a pixel that is supposed to be measuring an object located at the same distance from the reference position but that temporarily has a minimum gradation value different from the gradation value of the surrounding area due to noise of the distance sensor 3 or the like can be excluded from the extraction target. Furthermore, when extracting pixels for each group, the extraction means 16 may refer to the gradation values of other pixels adjacent to the pixel in addition to the gradation value of the pixel to determine whether or not to extract the pixel. In this case, the extraction means 16 refers to the gradation values of the pixels adjacent to the pixel in addition to the gradation value of the pixel itself as an index when extracting pixels for each group. For example, the extraction means 16 calculates a representative value (average value, median value, mode value, etc.) from the gradation value of the pixel and the gradation values of four pixels adjacent to the pixel above, below, left and right of the pixel as an index for a certain pixel. Furthermore, the extraction means 16 may group regions consisting of multiple pixels instead of pixels arranged at the same position in multiple distance images acquired by the distance image acquisition means during a predetermined period. In this case, the extraction means 16 calculates a representative value of the gradation values of the pixels belonging to the region for each region. When extracting pixels belonging to the region, the extraction means 16 selects a pixel to be extracted using a representative value of the region to which the pixel belongs instead of the gradation value of the pixel itself.

Then, the processed image generating means 17 generates a processed image from the two-dimensional images generated by the imaging device 2 during a predetermined period and/or the distance images generated by the distance sensor 3 during a predetermined period (step S5). The processed image generating means 17 generates a processed image in which multiple two-dimensional images and/or distance images acquired during a predetermined period are synthesized using pixels corresponding to the extracted pixels of the two-dimensional images corresponding to the distance images including the pixels with the smallest gradation values extracted by the extraction means 16 (two-dimensional images corresponding to the distance images including the pixels or regions specified for each group) and/or pixels with the smallest gradation values extracted in the distance images.

For example, the processed image generating means 17 generates a processed image from two-dimensional images generated by the imaging device 2 during a predetermined period. In this case, the processed image generating means 17 extracts each distance image including each pixel extracted for each group by the extraction means 16. The processed image generating means 17 identifies the gradation value of a pixel corresponding to the pixel extracted by the extraction means 16 for a two-dimensional image corresponding to the extracted distance image. The processed image generating means 17 generates a processed image by setting the gradation value identified in the two-dimensional image as the gradation value of a pixel corresponding to that group in the processed image.

When the extraction means 16 extracts areas for each group of areas, the processed image generation means 17 extracts each distance image including each area identified for each group by the extraction means 16. The processed image generation means 17 generates a processed image by setting the gradation value of each pixel corresponding to the area extracted by the extraction means 16 as the gradation value of each pixel in the area corresponding to that group in the processed image for a two-dimensional image corresponding to the extracted distance image.

The processed image generating means 17 may also generate a processed image from a distance image generated by the distance sensor 3 during a predetermined period of time. In this case, the processed image generating means 17 identifies the gradation value of each pixel in the distance image extracted for each group by the extraction means 16. The processed image generating means 17 generates a processed image by setting the gradation value identified in the distance image as the gradation value of the pixel corresponding to that group in the processed image.

In addition, when the extraction means 16 extracts areas for each group of areas, the processed image generation means 17 generates a processed image by setting the gradation value of each pixel corresponding to each area in the distance image identified for each group by the extraction means 16 as the gradation value of each pixel in the area corresponding to that group in the processed image.

The processed image generating means 17 may generate a processed image from a two-dimensional image generated by the imaging device 2 during a predetermined period and a distance image generated by the distance sensor 3 during a predetermined period. In this case, the processed image generating means 17 extracts each distance image including each pixel extracted by the extraction means 16 for each group. The processed image generating means 17 specifies the gradation value of a pixel corresponding to the pixel extracted by the extraction means 16 for a two-dimensional image corresponding to the extracted distance image. The processed image generating means 17 also specifies the gradation value of each pixel in the distance image extracted by the extraction means 16 for each group. The processed image generating means 17 generates a processed image by setting the gradation value specified in the two-dimensional image as the gradation value of the first component of the pixel corresponding to that group in the processed image and setting the gradation value specified in the two-dimensional image as the gradation value of the second component of the pixel corresponding to that group in the processed image. The processed image is, for example, an image having RGB color components, where the first component is, for example, a G component and the second component is, for example, an R component. The first and second components may be other components among the RGB color components. The first and second components may also be any of the CMY color components. The first and second components may also be components that are not determined in relation to human vision.

As described above, the extraction means 16 extracts pixels with the smallest gradation value from the group only for the group corresponding to the person area detected by the person area detection means 15. That is, the processed image generation means 17 generates a processed image only for the group corresponding to the detected person area so that the gradation value of the pixel extracted for each group and/or the pixel in the two-dimensional image corresponding to the pixel is set to the gradation value of the pixel corresponding to the group. For pixels not corresponding to the detected person area, the processed image generation means 17 sets the gradation value of the pixel in a predetermined image (e.g., the latest two-dimensional image and/or distance image) to the gradation value of the pixel corresponding to the group. In this way, the processed image generation means 17 can reduce the processing load of generating the processed image and generate a processed image in which multiple images are synthesized, limited to the area corresponding to the person. Note that a combination means includes both the extraction means 16 and the processed image generation means 17. The combination means generates a processed image in which multiple two-dimensional images acquired during a predetermined period are synthesized using a pixel or area corresponding to a specified pixel or area in a two-dimensional image corresponding to a distance image including the pixel or area specified for each group. In particular, the synthesis means generates a processed image by using the gradation value of a pixel or region corresponding to a specified pixel or region in a two-dimensional image corresponding to a distance image including a pixel or region specified for each group as the gradation value of the pixel or region corresponding to the group. Alternatively, the synthesis means generates a processed image by synthesizing a plurality of distance images acquired during a predetermined period using the pixel or region specified for each group. Also, the synthesis means generates a processed image by using the gradation value of a pixel or region specified for each group as the gradation value of a pixel or region corresponding to the group. Alternatively, the synthesis means generates a processed image by using the gradation value of a pixel or region corresponding to a specified pixel or region in a two-dimensional image corresponding to a distance image including a pixel or region specified for each group as the gradation value of the first component of the pixel or region corresponding to the group, and the gradation value of a pixel or region specified for each group in the distance image as the gradation value of the second component of the pixel or region corresponding to the group. The synthesis means generates a processed image by using the pixel or region corresponding to a specified pixel or region in a two-dimensional image corresponding to a distance image including a pixel or region specified for each group only for the group corresponding to the detected person region.

Figure 3 is a diagram for explaining the correspondence between distance images, two-dimensional images, and processed images. Figure 3 shows distance images D1-D3 and two-dimensional images E1-E3 generated at times T1, T2, and T3, respectively, and a processed image F3 generated from the distance images D1-D3 and two-dimensional images E1-E3.

In the example shown in FIG. 3, pixels P1 and P2 in distance images D1 to D3 are located at the same position and are classified into the same group. Suppose that the pixel with the smallest gradation value among pixels P1 in distance images D1 to D3 is pixel P1 in distance image D1, and the pixel with the smallest gradation value among pixels P2 in distance images D1 to D3 is pixel P2 in distance image D3. In this case, the gradation value of pixel P1 in two-dimensional image E1 corresponding to distance image D1 is set as the gradation value of the first component (G component) of pixel P1 in processed image F3, and the gradation value of pixel P1 in distance image D1 is set as the gradation value of the second component (R component) of pixel P1 in processed image F3. In addition, the gradation value of pixel P2 in two-dimensional image E3 corresponding to distance image D3 is set as the gradation value of the first component (G component) of pixel P2 in processed image F3, and the gradation value of pixel P2 in distance image D3 is set as the gradation value of the second component (R component) of pixel P2 in processed image F3.

Figure 4 shows an example of a processed image generated from a two-dimensional image. Two-dimensional images 21 to 23 are images captured at times T1, T2, and T3 of a person waving his or her hand towards the imaging device 2 in the monitored space. Generally, when a person waves his or her hand towards a specific position, the person waves by pushing their hand towards the specific position. Therefore, the hand is positioned closer to the specific position than the background or the person. Therefore, processed image 24 is generated so as to include areas 25 to 27 in which the hands are captured in two-dimensional images 21 to 23, respectively.

Figure 5 shows an example of a processed image generated from a distance image. Distance images 31 to 33 are distance images generated by measuring the situation in which a person is waving their hand towards distance sensor 3 in the monitored space at times T1, T2, and T3. Generally, when a person waves their hand towards a specific position, they wave by pushing their hand towards the specific position. For this reason, the hand is positioned closer to the specific position than the background or person (in Figure 5, the darker the color, the closer). Therefore, processed image 24 is generated in which areas 35 to 37 where the hands are captured in distance images 31 to 33 are different gradation values from the background and person.

Figure 6 is an example of a processed image generated from a two-dimensional image and a distance image. In this processed image, the gradation value extracted from the distance image is set as the gradation value of the R component, and the gradation value extracted from the two-dimensional image is set as the gradation value of the G component. In general, the shorter the distance from the imaging device to the object, the clearer the image in which the object is captured, and the longer the distance from the imaging device to the object, the blurrier the image in which the object is captured, and the unclearer the edges of the object. Therefore, in this processed image, the G component extracted from the two-dimensional image is blurred for the background away from the imaging device. However, the edges of this background are made clear by the R component extracted from the distance image. On the other hand, in this processed image, for a person present near the imaging device, the G component extracted from the two-dimensional image makes the texture of the person's clothing, etc. clear, and the appearance of the person waving his/her hand and the posture of his/her arm moving little by little with his/her elbow as the fulcrum are clearly expressed. In this way, by generating a processed image from a two-dimensional image and a distance image, image processing system 1 can complement information about distant details that are easily lost in two-dimensional images with distance information, thereby clarifying the edges of the background.

Figure 7 is an example of a processed image generated from a two-dimensional image. As shown in Figure 6, the processed image generated in this way has clearer texture of the person due to the information on shading, and clearly shows the person waving their hand and the posture of their arm moving little by little with their elbow as the fulcrum.

Figure 8 is an example of a processed image generated from a distance image. As shown in Figure 7, the processed image generated in this way contains information about distance, so the edges between the background and the person are clearly depicted, and furthermore, the edges between the background and the person and their hands are also clearly depicted.

Then, the detection means 18 obtains an evaluation value for the processed image generated by the processed image generation means 17 (step S6). The detection means 18 inputs the processed image to the model 10, which has been trained to output information about the gesture movements of a person contained in the input learning processed image, and detects the gesture movements of a person in the monitored space based on the information output from the model 10.

For example, the detection means 18 obtains an evaluation value for the processed image using the model 10 stored in the storage unit 9. The model 10 is generated by the learning means 20. The learning means 20 learns the relationship between the multiple learning processed images, the evaluation value indicating the likelihood that the gesture to be detected is included in each learning processed image, and the position of the area including the gesture to be detected, using a known machine learning technique such as deep learning. The gesture to be detected is, for example, a waving motion. In particular, the detection means 18 detects a predetermined gesture motion made in front of the human body (an area where the gradation value of the distance image is smaller than the gradation value of the human body) among gesture motions made using a part of the human body such as a human hand. The gesture to be detected may be a periodic motion such as a beckoning motion. In addition, the gesture to be detected may be multiple, and may be, for example, both a waving motion and a beckoning motion. Each learning processing image is an image generated in the same manner as the processing image from images including gestures of various sizes by objects in various states (standing, sitting, lying, etc.) or images including no gestures. The learning means 20 stores the learned relationships as a model 10 in the storage unit 9.

The learning processing image is input to the input layer. Each node of the intermediate layer extracts a feature vector from the image output from each node of the input layer, and outputs the sum of values obtained by multiplying each extracted feature vector by a weight. The output layer outputs the sum of values obtained by multiplying each feature vector output from each node of the intermediate layer by a weight. The learning means 20 adjusts each weight while learning so that the difference between the output value from the output layer, the correct answer value, and the position of the area including the gesture to be detected is small. For example, the correct answer value is set to 1 when the learning processing image includes the gesture to be detected, and is set to 0 when the gesture to be detected is not included. Note that the model 10 may be trained by other machine learning techniques such as DPM (Deformable Part Model), R-CNN (Regions with Convolutional Neural Networks), and YOLO. The model 10 may also be generated by an external computer other than the image processing device 4 and transmitted to the image processing device 4. In that case, the learning means 20 may be omitted.

The detection means 18 inputs the processed image into the model 10 stored in the memory unit 9, and obtains the output value output from the model 10 as an evaluation value for the processed image and the position of the area where the gesture was detected.

The detection means 18 may cut out an area in the processed image where there is movement, and input the cut-out image to the model 10 to calculate the evaluation value. For a stationary object, the distance from the distance sensor 3 does not change, so there is a possibility that the distance image in which the pixel with the smallest gradation value is extracted by the extraction means 16 for each position in the object may vary. On the other hand, for a moving object, it is highly likely that the pixel with the smallest gradation value is extracted collectively from one distance image for the entire area in the object. Therefore, the detection means 18 identifies the pixel extracted by the extraction means 16 in step S4 from each distance image for each of the multiple distance images generated during a predetermined period. The detection means 18 detects, as an area in the processed image where there is movement, an area of pixels that are closely adjacent to each other, connected, and have a predetermined size or more, among the pixels identified in each distance image. This allows the detection means 18 to detect gestures only in areas where there is movement, and to detect gestures more accurately.

The detection means 18 may also calculate the evaluation value using a pattern matching technique. In this case, the image processing device 4 stores in advance in the storage unit 9 a number of image patterns in which the gesture to be detected is captured within the sample processed image. The detection means 18 cuts out an area of a predetermined size within the processed image generated in step S5 while shifting its position, and obtains the degree of similarity with the image pattern stored in the storage unit 9 as an evaluation value. The degree of similarity is, for example, a normalized cross-correlation value.

Then, the detection means 18 detects the gesture of the person to be detected based on the acquired evaluation value (step S7). If the evaluation value is equal to or greater than a predetermined threshold, the detection means 18 determines that the processed image contains the gesture of the person to be detected, and if the evaluation value is less than the threshold, the detection means 18 determines that the processed image does not contain the gesture of the person to be detected. In this way, the detection means 18 detects the gesture of the person in the monitored space based on the processed image. In particular, the detection means 18 inputs the processed image to the model 10 that has been trained to output information regarding the gesture of the person included in the processed image for learning when the processed image for learning is input, and detects the gesture of the person in the monitored space based on the information output from the model 10.

Next, the detection means 18 determines whether or not the gesture to be detected has been detected (step S8). If the gesture to be detected has not been detected, the detection means 18 ends the series of steps without performing any particular processing.

On the other hand, when the detection means 18 detects a gesture to be detected, the detection means 18 judges whether or not a person is present in the vicinity of the area where the gesture was detected (step S9). The detection means 18 calculates the distance between the position of the area where the gesture was detected acquired in step S6 and the area in the processed image corresponding to the person area detected in step S3. If the calculated distance is less than a predetermined distance threshold, the detection means 18 judges that a person is present in the vicinity of the area where the gesture was detected and that the detected gesture is a gesture performed by a person. On the other hand, if the calculated distance is equal to or greater than the distance threshold, the detection means 18 judges that no person is present in the vicinity of the area where the gesture was detected and that the detected gesture is not a gesture performed by a person, and ends the series of steps. This allows the detection means 18 to prevent erroneous detection of the movement of an object other than a person in the monitored space as the gesture to be detected.

When it is determined that a person is present near the area where the gesture is detected, the output control means 19 outputs information about the detected gesture via the communication unit 8 to notify an external device (step S10), and ends the series of steps. The information about the gesture includes the fact that a gesture of the detection target by a person has been detected, the type of gesture, the time when the gesture was detected, the duration of the gesture, the area where the gesture was detected, etc. The output control means 19 may display the information about the detected gesture on the display unit 7 or output it from a sound output device (not shown).

The process of step S3 may be omitted. In this case, the extraction means 16 performs the process of step S4 on all pixels in the distance image and the two-dimensional image.

The processing of step S9 may also be omitted. In this case, the model 10 may be pre-trained to output an evaluation value indicating the likelihood that each learning processing image contains a gesture to be detected by a person, and the detection means 18 may detect a gesture made by a person using the determination model 10.

(Effects of Image Processing System 1)
As described above, the image processing system 1 detects a gesture to be detected based on a processed image generated using pixels captured or measured when an object in a monitored space was located at the closest position within a predetermined period of time. This allows the image processing system 1 to focus on an area where a movement occurs at the closest position and accurately detect a gesture in which a movement occurs at the closest position. Therefore, the image processing system 1 can accurately detect a gesture of a person in a monitored space. In particular, when a person to be monitored is asked to put his/her hand out in front of his/her body and wave as a means of calling a nurse, the image processing system 1 can accurately detect the movement and notify the person to be monitored.

The image processing system 1 generates a processed image based on the distance image and the two-dimensional image. By generating a processed image based on the two-dimensional image, the image processing system 1 can include information about the shape and texture of the object in the processed image. Although the distance sensor 3 may not be able to measure the distance to an object with low reflectance of near-infrared light, the image processing system 1 can generate a highly reliable processed image by generating a processed image based on the two-dimensional image. On the other hand, the image processing system 1 generates a processed image based on the distance image, so that even if the brightness of the background and the person are similar, the image processing system 1 can generate a processed image in which the background and the person are clearly distinguished from each other. In this way, the brightness and the distance play complementary roles when generating a processed image, so that the image processing system 1 can detect gestures more accurately by using a processed image generated based on the distance image and the two-dimensional image. Furthermore, by using the distance image, a processed image in which the person and the hand are clearly distinguished from each other can be generated, even if the person and the hand are moving while overlapping, and the gesture can be detected more accurately.

1 Image processing system, 2 Imaging device, 3 Distance sensor, 4 Image processing device, 8 Communication unit, 9 Memory unit, 12 Processing unit

Claims (9)

a distance image acquisition means for sequentially acquiring distance images in which information regarding a distance from a reference position to an object in a monitored space is used as a gradation value;
a two-dimensional image acquisition means for sequentially acquiring two-dimensional images, the two-dimensional images having gradation values representing information on shading within the monitored space, corresponding to the sequentially acquired distance images;
a synthesis means for identifying, for each group of pixels or regions arranged at the same position in a plurality of distance images acquired by the distance image acquisition means during a predetermined period of time, a pixel or region having a relatively small gradation value within the group, and generating a processed image in which the plurality of two-dimensional images acquired during the predetermined period of time are synthesized using a pixel or region corresponding to the identified pixel or region in a two-dimensional image corresponding to the distance image including the pixel or region identified for each group;
1. An image processing system comprising: The image processing system according to claim 1, wherein the synthesis means generates a processed image by using the gradation value of a pixel or region corresponding to the specified pixel or region in a two-dimensional image corresponding to a distance image including the pixel or region specified for each group as the gradation value of the pixel or region corresponding to the group. The image processing system according to claim 1 or 2, wherein the synthesis means generates the processed image such that the gradation value of a pixel or region corresponding to the identified pixel or region in a two-dimensional image corresponding to a distance image including the pixel or region identified for each group is set to the gradation value of the first component of the pixel or region corresponding to the group, and the gradation value of a pixel or region identified for each group in the distance image is set to the gradation value of the second component of the pixel or region corresponding to the group. The image processing system according to any one of claims 1 to 3 further comprises a detection means for inputting the processed image for learning into a model trained to output information regarding the gesture movements of a person contained in the input processed learning image, and detecting the gesture movements of a person in the monitored space based on information output from the model. a person area detection unit that detects a person area including a person in the distance image or the two-dimensional image,
The image processing system according to any one of claims 1 to 4, wherein the synthesis means generates the processed image using pixels or areas corresponding to the identified pixels or areas in a two-dimensional image corresponding to a distance image including the pixels or areas identified for each group, only for the group corresponding to the detected person area. 4. The image processing system according to claim 1 , further comprising a detection unit that detects, based on the processed image, a predetermined gesture made in front of a person present within the monitored space. a distance image acquisition means for sequentially acquiring distance images in which information regarding a distance from a reference position to an object in a monitored space is used as a gradation value;
identifying, for each group of pixels or regions arranged at the same position in a plurality of distance images acquired by the distance image acquisition means during a predetermined period, a pixel or region having a gradation value indicating a relatively short distance within the group;
a processed image generating means for generating a processed image by synthesizing a plurality of distance images acquired during a predetermined period using the gradation values of the pixels or regions specified for each group;
1. An image processing system comprising: Sequentially acquiring distance images in which information regarding the distance from a reference position to an object in a monitored space is used as a grayscale value;
Sequentially acquiring two-dimensional images in which information regarding shading in the monitored space is used as a gradation value in response to the sequentially acquired distance images;
Identifying, for each group of pixels or regions arranged at the same position in a plurality of distance images acquired during a predetermined period, a pixel or region having a relatively small gradation value within said group;
generating a processed image by synthesizing a plurality of two-dimensional images acquired during a predetermined period using pixels or regions corresponding to the identified pixels or regions in a two-dimensional image corresponding to a distance image including the pixels or regions identified for each group;
A control program that causes a computer to execute the above steps. Sequentially acquiring distance images in which information regarding the distance from a reference position to an object in a monitored space is used as a grayscale value;
identifying, for each group of pixels or regions that are arranged at the same position in a plurality of distance images acquired during a predetermined period, a pixel or region that has a gradation value that indicates a relatively short distance within the group;
generating a processed image by synthesizing a plurality of distance images acquired during a predetermined period using the gradation values of the pixels or regions identified for each group;
A control program that causes a computer to execute the above steps.

Images