JP7799514B2 - Observation system, observation system control method, and program - Google Patents

Description

The present invention relates to an observation system for observing a target person and related technologies.

There are observation systems that observe a person being observed (see Patent Document 1, etc.). For example, Patent Document 1 describes a monitoring system (observation system) that monitors residents (persons being observed) in care facilities and the like using images captured by a camera.

Japanese Patent Application Laid-Open No. 2022-010581

In the above-mentioned observation systems, it is preferable to monitor the entire room in which the person being observed lives (with as few blind spots as possible). In order to create such technology (technology that ensures a wider observation range), one idea is to drive the observation (monitoring) camera to change its orientation and change the camera's shooting range (camera orientation, etc.).

However, for example, if the position of a person (person being observed) is detected and the camera direction (direction of the camera's line of sight) is constantly tracked so that it is always directed at that person (position), the camera's operating noise will be generated frequently (continuously) in response to the person's movements. This could potentially cause stress (psychological stress) to the person.

The objective of this invention is to provide an observation system and related technologies that can ensure a wider observation range while avoiding or minimizing stress in the person being observed.

In order to solve the above problem, the observation system of the present invention comprises an imaging unit that captures an image of a space to be observed, a drive unit that rotates the imaging unit to change the imaging range of the imaging unit, an illumination unit that emits illumination light to illuminate the imaging range of the imaging unit, and a control unit that controls the rotational driving of the imaging unit by the drive unit to change the imaging range of the imaging unit to one of a predetermined number of imaging ranges that have been set in advance , and controls the emission of illumination light by the illumination unit, wherein the control unit changes the imaging range of the imaging unit to one of the predetermined number of imaging ranges and stops the rotational driving of the imaging unit by the drive unit, and then causes the illumination unit to emit illumination light with an amount of light emission corresponding to the one imaging range from the amount of light emission predetermined for each of the predetermined number of imaging ranges, and performs observation processing of the person to be observed based on the captured image captured by the imaging unit.

The control unit may detect the position of the person being observed within the captured image, and if the position of the person being observed falls within a predetermined area in the captured image, may perform an observation process of the person being observed based on the captured image captured by the imaging unit, without driving the drive unit and maintaining the imaging range of the imaging unit at the one imaging range.

The predetermined number of shooting ranges may include a first shooting range and a second shooting range, and the second shooting range may be a shooting range that can capture the person being observed that is farther away from the imaging unit than the first shooting range.

The predetermined number of shooting ranges may include a first shooting range and a second shooting range, the imaging unit may be installed near and above the bed, the first shooting range may be a shooting range that can capture the person being observed near the bed, and the second shooting range may be a shooting range that can capture the person being observed near an entrance away from the bed.

The amount of light emitted determined in correspondence with the second imaging range may be greater than the amount of light emitted determined in correspondence with the first imaging range.

The program of the present invention is characterized in that it is a program for causing a computer that controls an observation system having an imaging unit that captures an image of a space to be observed, a drive unit that rotates and drives the imaging unit to change the imaging range of the imaging unit, and an illumination unit that emits illumination light to illuminate the imaging range of the imaging unit, to execute the following steps: (a) controlling the rotational drive of the imaging unit by the drive unit to change the imaging range of the imaging unit to one of a predetermined number of imaging ranges that have been set in advance; and (b) in a state where the imaging range of the imaging unit is changed to one of the predetermined number of imaging ranges and the rotational drive of the imaging unit by the drive unit is stopped, causing the illumination unit to emit illumination light of an amount of light emission corresponding to the one imaging range from among the light emission amounts that are predetermined for each of the predetermined number of imaging ranges, and performing an observation process of the person to be observed based on the captured image captured by the imaging unit.

The control method of the present invention is a control method for an observation system comprising an imaging unit that captures an image of a space to be observed, a drive unit that rotates and drives the imaging unit to change the imaging range of the imaging unit, and an illumination unit that emits illumination light to illuminate the imaging range of the imaging unit , and is characterized by comprising the steps of: a) controlling the rotational drive of the imaging unit by the drive unit to change the imaging range of the imaging unit to one of a predetermined number of imaging ranges that have been set in advance; and b) in a state where the imaging range of the imaging unit has been changed to one of the predetermined number of imaging ranges and the rotational drive of the imaging unit by the drive unit has been stopped, causing the illumination unit to emit illumination light with an amount of light emission corresponding to the one imaging range from among the light emission amounts that are predetermined for each of the predetermined number of imaging ranges, and performing observation processing of the person to be observed based on the captured image captured by the imaging unit.
The observation system of the present invention comprises an imaging unit that captures an image of a space to be observed, a drive unit that rotates the imaging unit to change the imaging range of the imaging unit, and a control unit that controls the rotation of the imaging unit by the drive unit to change the imaging range of the imaging unit to one of a predetermined number of imaging ranges that have been set in advance, wherein the control unit sets the imaging range of the imaging unit to one of the predetermined number of imaging ranges, and then detects the position of the person to be observed in the captured image corresponding to the one imaging range, and if the position of the person to be observed falls within a predetermined area in the captured image, performs an observation process of the person to be observed based on the captured image while maintaining the imaging range of the imaging unit at the one imaging range without rotating the imaging unit.
When the position of the person being observed no longer falls within the specified area, the control unit may rotate the imaging unit to change the shooting range of the imaging unit to a shooting range other than the one shooting range.
The observation process of the observed person may include a process of detecting a behavior-related event of the observed person.
The program of the present invention is characterized in that it is a program for causing a computer that controls an observation system having an imaging unit that captures an image of a space to be observed and a drive unit that rotates the imaging unit to change the imaging range of the imaging unit to execute the following steps: a) controlling the rotational drive of the imaging unit by the drive unit to set the imaging range of the imaging unit to one imaging range that is one of a predetermined number of imaging ranges that have been set in advance; b) detecting the position of the person to be observed in the captured image that corresponds to the one imaging range; and c) if the position of the person to be observed falls within a predetermined area in the captured image, performing an observation process of the person to be observed based on the captured image while maintaining the imaging range of the imaging unit at the one imaging range without rotating the imaging unit.
The control method of the present invention is a control method for an observation system that includes an imaging unit that captures an image of a space to be observed and a drive unit that rotates the imaging unit to change the imaging range of the imaging unit, and is characterized by comprising the steps of: a) controlling the rotational drive of the imaging unit by the drive unit to set the imaging range of the imaging unit to one imaging range that is one of a predetermined number of imaging ranges that have been set in advance; b) detecting the position of the person to be observed in the captured image that corresponds to the one imaging range; and c) when the position of the person to be observed falls within a predetermined area in the captured image, performing an observation process of the person to be observed based on the captured image while maintaining the imaging range of the imaging unit at the one imaging range without rotating the imaging unit.

This invention makes it possible to ensure a wider observation range while avoiding or minimizing stress in the person being observed.

FIG. 1 is a schematic diagram showing an observation system. FIG. 2 is a functional block diagram showing a schematic configuration of an observation device. 10 is a conceptual diagram showing that information on the three-dimensional position of each specific part of a photographed person is obtained (calculated) based on a photographed image and depth information. FIG. 10 is a diagram showing three attitude angles (three preset attitude angles) of the camera unit. FIG. 10 is a diagram showing three shooting ranges (three preset shooting ranges). 10A and 10B are diagrams illustrating a process of changing the shooting range (selected range). 10A and 10B are diagrams illustrating a process of changing the shooting range (selected range). 10A and 10B are diagrams showing the light emission amounts (light emission amounts for each shooting range) previously set by the pre-adjustment process. 10 is a flowchart showing the observation process (watching process) in detail.

Embodiments of the present invention will be described below with reference to the drawings.

<1. System Overview>
Fig. 1 is a schematic diagram showing an observation system 1. As shown in Fig. 1, the observation system 1 includes a plurality of observation devices 10 and a plurality of terminal devices 70, 80. The terminal device 70 is also referred to as a management device 70, and the terminal device 80 is also referred to as a mobile terminal device 80. Note that Fig. 1 illustrates only some of the plurality of observation devices 10 and the plurality of terminal devices 70, 80, namely, the devices 10, 70, 80.

Here, we will mainly use an example of how the observation system 1 is used in a nursing care facility. However, this is not limited to this, and the observation system 1 may also be used in nursing care facilities (hospitals, etc.) or private homes, etc.

As shown in FIG. 1, each observation device 10 and each terminal device 70, 80 are connected to each other via a network 108. The network 108 is configured by a LAN (Local Area Network) and the Internet, etc. Furthermore, the connection to the network 108 may be wired or wireless. For example, the management device 70 is wired to the network 108, and each observation device 10 and each mobile terminal device 80 is wirelessly connected to the network 108. Alternatively, all devices 10, 70, 80 may be wirelessly connected to the network 108.

Each observation device 10 is placed in each room 90 (e.g., each care recipient's private room) of the person being observed (here, a care recipient). Each observation device 10 (and observation system 1) is a device that observes the person being observed (e.g., a care recipient) based on captured images of the person being observed. In detail, each observation device 10 detects (acquires) skeletal information (positional information of multiple specific body parts, etc.) (described below) of the person being observed, positional information of the person being observed, and various "behavioral-related events" related to the person being observed, based on the captured images, etc.

"Behavior-related events" include a person's behavior (movement) itself and/or states related to the person's behavior. "Behavior-related events" are events that should be detected (events that are the target of detection processing) regarding the observed person, and are also called detection target events. Examples of "behavior-related events" of a person include events such as "normal lying position (normal lying position (state of lying normally on the bed))", "sitting up (upper body) on the bed", "sitting on the edge of the bed", "sliding off the bed", "normal standing position", "normal sitting position", and "falling".

The observation device 10 is also referred to as a detection device, and the observation system 1 is also referred to as a detection system. Furthermore, because the observation device 10 and the observation system 1 are used to monitor the behavior of people, they are also referred to as a monitoring device and a monitoring system.

2. Overview of the observation device 10
1 and 2, the observation device 10 includes a camera unit 20, a drive unit 27, and a processing unit 30. Note that FIG. 2 is a functional block diagram showing a schematic configuration of the observation device 10.

The camera unit 20 is a camera that captures images of the space to be observed (here, the space within the living room 90).

As shown in FIG. 1, the camera unit 20 is installed near or above the bed 92 in the room 90 (on the ceiling or upper part of the wall), and is capable of capturing images of the interior of the room 90. The processing unit 30 is capable of acquiring the three-dimensional positions (particularly time-series information) of multiple specific body parts (eyes, ears, nose, neck, chest, waist, shoulders, elbows, wrists, knees, ankles, etc.) of the person being observed based on images (particularly moving images) captured by the camera unit 20. Note that while the camera unit 20 and the processing unit 30 are provided separately here, this is not limiting, and the camera unit 20 and the processing unit 30 may also be provided as an integrated unit.

The camera unit 20 also includes an imaging unit (sensor unit) 23 and an illumination unit (light-emitting unit) 25 (see Figure 2).

The drive unit 27 is a processing unit that can (mechanically) drive the camera unit 20 (more specifically, the imaging unit 23, lighting unit 25, etc.) to change the attitude of the camera unit 20. The drive unit 27 is configured with a drive mechanism (motors, gears, etc.). The drive operation by the drive unit 27 is controlled by the controller 31 (described below) of the processing unit 30.

The drive unit 27 is configured with a drive mechanism (motor, gears, etc.) capable of achieving rotational drive operation around a predetermined axis. The attitude (attitude angle) of the camera unit 20 is changed by the drive (rotational drive operation) of the drive unit 27. For example, the camera unit 20 is installed (placed) on a ceiling or the like in a state where it can rotate around a single axis parallel to the horizontal direction (e.g., the X axis in Figure 4). The rotational drive operation of the drive unit 27 causes the camera unit 20 to rotate around the X axis. The rotational drive operation also changes the shooting angle (attitude angle) of the camera unit 20, changing the field of view (shooting range) of the imaging unit 23. However, the present invention is not limited to this. The drive unit 27 may drive the camera unit 20 (imaging unit 23) by rotational drive operation around two axes (or three axes), or may also involve translational drive operation. The drive unit 27 is not limited to being provided external to the camera unit 20, but may be built into the camera unit 20 to drive the imaging unit 23, etc.

Here, the imaging range (field of view) of the imaging unit 23 can, in principle, change to an infinite number of imaging ranges (field of view) as the rotational drive operation proceeds. However, as will be described later, in this embodiment, of the infinite number of imaging ranges, only a predetermined number of imaging ranges (here, three imaging ranges F1, F2, and F3) are used for the imaging range (field of view) of the imaging unit 23 (see Figure 5). In other words, of the infinite number of attitude angles for the camera unit 20, only three attitude angles θ1, θ2, and θ3 for the camera unit 20 are used (see Figure 4).

The camera unit 20 is a three-dimensional camera. The camera unit 20 acquires captured images with depth information. Specifically, the camera unit 20 captures a captured image (e.g., infrared image) 110 (see Figure 3) of a subject object (e.g., a person, a wall, a floor 91, a bed 92), and acquires depth information (depth distance information) 120 for each pixel in the captured image 110. The depth information 120 for each pixel in the captured image is information about the distance (distance from the camera unit 20) to the subject object (e.g., a person, a wall, a floor, a bed) corresponding to each pixel in the captured image 110, and is information about the distance in a direction perpendicular to the captured image 110. In other words, the depth information is distance information (depth distance information) in the normal direction of the plane of the captured image. Depth information 120 for multiple pixels can also be expressed as a captured image with depth information.

Here, a TOF (Time of Flight) 3D camera is used as the camera unit 20. As described above, the camera unit 20 includes an imaging unit (sensor unit) 23 (Figure 2) and an illumination unit (light-emitting unit) 25. More specifically, the imaging unit 23 is configured as a camera (infrared camera) having optical elements (optical system) such as lenses and an imaging element (infrared image sensor) that receives light (infrared light). The illumination unit 25 emits illumination light (infrared light in this case) to illuminate the imaging range of the imaging unit 23. Specifically, the illumination unit 25 is a light-emitting unit that emits (emits) infrared light (also as illumination light) for distance measurement in the TOF method. The imaging unit 23 and the illumination unit 25 are integrated and have a fixed positional relationship. The illumination unit 25 is driven by a drive unit 27 in conjunction with the imaging unit 23, enabling it to illuminate the changed imaging range. Furthermore, the camera unit 20 also includes a controller 21. Controller 21 has the same hardware configuration as controller 31 (described below).

A TOF (Time of Flight) 3D camera is a camera that measures the distance to a subject using the time of flight of light (infrared light in this case). Specifically, the distance from the imaging unit 23 to the subject (depth distance information) is calculated (acquired) for each pixel in the captured image based on the time from when infrared light is emitted (irradiated) from the illumination unit (light-emitting unit) 25 to when the infrared light reaches the subject and the reflected light from the subject returns to the imaging unit (image sensor). The calculation process for this depth distance information is performed by a controller 21 or the like incorporated in the camera unit 20. The sensor unit 23 may also be provided with an RGB image sensor or the like that captures visible light images (such as color images), allowing color images to be captured.

In this way, the camera unit 20 acquires a captured image (captured image information) 110 (see Figure 3) of the subject person and depth information 120 for each pixel in the captured image (information on the distance to the object corresponding to each pixel (depth distance information)).

Figure 3 is a conceptual diagram showing how information on the three-dimensional position of each specific part of a subject person is obtained (calculated) based on a captured image 110 and depth information 120. Note that Figure 3 shows the captured image 110 and depth information 120 in a schematic manner. For example, in an actual captured image 110, the person is captured exactly as it was captured, whereas in the captured image 110 of Figure 3, the person is represented (simplified) by a figure made up of connected ellipses. The same is true for other depth information 120. Furthermore, while the actual depth information 120 contains information on the distance to the object corresponding to each pixel, in the depth information 120 of Figure 3, the distance to the object corresponding to each pixel is expressed by converting it into the density of each pixel. In more detail, the magnitude of the distance is expressed by the magnitude of the density (shade).

First, the processing unit 30 (particularly the controller 31) analyzes the captured image 110 and acquires skeletal information (skeletal model information) 140 (see the lower left portion of Figure 3) of the subject person based on the texture information, etc., of the captured image 110. This skeletal information 140 represents the subject person's skeleton (in a simplified form) using multiple specific body parts (specifically, the chest, nose, neck, shoulders, elbows, wrists, hips, knees, ankles, eyes, ears, etc.) (mainly joints) of the subject person and skeletal lines (links) connecting these multiple specific body parts. In the skeletal information 140 of Figure 3, each specific body part is represented by a "point," and each skeletal line (connecting line) is represented by a "line segment." The skeletal information 140 based on the captured image contains information about multiple specific body parts (here, B0 to B17) (such as information about the planar position of each body part within the image).

In addition, the processing unit 30 acquires planar position information (two-dimensional position information within the captured image in the camera coordinate system) of each specific part of the subject person (for example, each representative position) within the captured image 110 based on the skeletal information 140 of the subject person.

Furthermore, the processing unit 30 also acquires distance information (depth position information in the camera coordinate system) from the camera unit 20 to each specific part based on depth information 120 of one or more pixels at a planar position (within the captured image 110) corresponding to each specific part. This distance information can also be expressed as depth information in the normal direction (camera line of sight direction) of the captured image plane.

In this way, the processing unit 30 obtains information regarding the planar position of each specific part of the subject person within the captured image (two-dimensional position information within the captured image) and distance information (depth information) to each specific part of the subject person.

The processing unit 30 then acquires three-dimensional position information 150 (three-dimensional position information within the living space) of each specific body part of the subject person, using coordinate transformation or the like, based on information regarding the planar position of each specific body part within the captured image and information regarding the distance (depth information) to each specific body part. Specifically, the processing unit 30 converts the position information in the camera coordinate system Σ1 (planar position within the captured image and depth position in the normal direction of the captured image) into three-dimensional position information 150 in a coordinate system Σ2 fixed relative to the living room 90 (more specifically, the bed 92). The camera coordinate system Σ1 is, for example, a three-dimensional Cartesian coordinate system based on three orthogonal axes, including two orthogonal axes parallel to the plane of the captured image and one axis extending perpendicular to the plane of the captured image. Furthermore, the transformed coordinate system Σ2 is, for example, a three-dimensional Cartesian coordinate system based on three orthogonal axes, including two orthogonal axes parallel to the horizontal plane and one axis extending vertically (in the height direction). In this observation system 1 (controller 31, etc.), the position of the camera unit 20 in real space is acquired in advance, and adjustments based on the position of the camera unit 20 (calibration related to the camera position, etc.) are performed in advance. In other words, the positional relationship between the two coordinate systems Σ1 and Σ2 is adjusted in advance.

More specifically, this calibration between the coordinate systems Σ1 and Σ2 is performed for each of the three attitude angles θi (i = 1, 2, 3) of the camera unit 20 (see Figure 4 (described below)). In other words, this calibration is performed for each of the three imaging ranges Fi.

Here, the controller 31 of the processing unit 30 generates skeletal information 140 and three-dimensional position information 150 of each specific body part, but this is not limited to this.

For example, the controller 21 of the camera unit 20 may generate the skeletal information 140. The controller 21 of the camera unit 20 may also generate three-dimensional position information 150 of each specific body part. Specifically, the controller 21 may acquire information about the position of each specific body part of the subject person within the captured image and distance information (depth information) to each specific body part, and acquire the three-dimensional position information 150 of each specific body part through coordinate transformation or the like. The controller 31 of the processing unit 30 may then acquire the three-dimensional position information 150 of each specific body part from the camera unit 20. Alternatively, the controllers 21, 31 of the camera unit 20 and the processing unit 30 may cooperate to perform these various processes. In other words, the skeletal information 140 and the three-dimensional position information 150 may be generated by both or one of the controllers 21, 31.

Furthermore, while a TOF (Time of Flight) camera is used here as an example of a 3D camera, this is not limiting. The 3D camera may be a stereoscopic camera, or a pattern illumination 3D camera that acquires depth information for each pixel based on the irradiation of a dot pattern of infrared light. Any type of 3D camera may acquire a captured image of the person being observed and depth information (such as depth information for each pixel) that is information on the distance to the subject object in the captured image.

Furthermore, since the observation device 10 is a device that performs image processing on captured images, etc., it is also referred to as an image processing device. The observation system 1 is also referred to as an image processing system, etc.

3. Processing unit 30
As shown in FIG. 2, the processing unit 30 of the observation device 10 includes a controller (also referred to as a control unit) 31, a storage unit 32, a communication unit 34, and an operation unit 35.

The controller 31 is built into the processing unit 30 and is a control device that controls the observation device 10.

The controller 31 is configured as a computer system equipped with one or more hardware processors (e.g., a central processing unit (CPU) and a graphics processing unit (GPU)). The controller 31 performs various processes by executing, in the CPU or the like, a predetermined software program (hereinafter simply referred to as a program) stored in a storage unit 32 (a non-volatile storage unit such as a ROM and/or a hard disk). The program (more specifically, a group of program modules) may be recorded on a portable storage medium such as a USB memory, read from the storage medium, and installed on the observation device 10. Alternatively, the program may be downloaded via a communication network or the like and installed on the observation device 10.

The controller 31 acquires captured image information about the subject person (such as the person being observed) from the camera unit 20, and also acquires distance information (depth information) to each specific part of the subject from the camera unit 20. The controller 31 then acquires three-dimensional position information for each specific part based on the captured image information (particularly the planar position information of each specific part of the subject within the captured image) and the distance information (depth information of the subject).

The controller 31 also uses the learning model 400 (Figure 2) to detect behavior-related events of the target person.

For example, a neural network model consisting of multiple layers is used as the learning model 400. Then, using a predetermined machine learning method, weighting coefficients (learning parameters) between the multiple layers in the neural network model (input layer, (one or more) intermediate layers, and output layer) are adjusted. The learning model 400 after being trained by machine learning is also referred to as a trained model. Specifically, the learning parameters of the learning model (learner) 400 are adjusted using a predetermined machine learning method, and a trained learning model (trained model) 400 is generated.

To this end, first, the controller 31 executes the learning stage of machine learning. Specifically, the controller 31 performs machine learning on the learning model 400 in advance using multiple pieces of training data. Each piece of training data uses training data that takes information indicating the positional relationships of multiple specific body parts of a person as input and outputs behavior-related events related to the person. Then, using a predetermined machine learning method, weighting coefficients (learning parameters) between multiple layers in the neural network model (input layer, (one or more) intermediate layers, output layer), etc. are adjusted. As a result, a trained learning model 400 (trained model) is generated (produced).

The controller 31 then executes the inference stage process using the trained learning model 400. Specifically, the controller 31 uses the learning model 400 (trained model) that has been machine-learned using the multiple pieces of training data to detect behavior-related events related to the target person (also referred to as the person to be detected or the person to be determined) based on information indicating the positional relationships of multiple specific parts of the target person. More specifically, the controller 31 inputs the positional relationships of multiple specific parts of the target person into the trained model 400, and obtains output from the trained model 400 (the processing result of the event detection process related to the behavior-related events of the target person).

Here, the observation device 10 (controller 31) performs both the learning stage processing and the inference stage processing in machine learning. However, this is not limited to this, and for example, the learning stage processing and the inference stage processing may be performed by separate devices.

In addition, the controller 31 cooperates with the communication unit 34 and the like to transmit the detection results to the terminal devices 70, 80, which then output the detection results (e.g., display output and/or audio output).

The memory unit 32 is composed of a storage device such as a hard disk drive (HDD) or a solid state drive (SSD). The memory unit 32 stores (memorizes) the above-mentioned programs and various data. For example, the memory unit 32 stores (memorizes) time-series data such as the three-dimensional positions of multiple specific parts of the observed person, as well as various data and programs used for learning and using the learning model 400.

The communication unit 34 is capable of performing network communication via the network 108. This network communication uses various protocols, such as TCP/IP (Transmission Control Protocol/Internet Protocol). By using this network communication, the observation device 10 can send and receive various types of data to and from desired destinations (e.g., terminal devices 70, 80).

The operation unit 35 includes an operation input unit 35a that accepts operation input for the observation device 10, and a display unit 35b that displays and outputs various information.

4. Processing Overview
Next, an outline of the processing in this embodiment will be described.

As a technology different from the embodiments of the present application (also referred to as a comparative example), as described above, it is conceivable to detect the position of a person (person being observed) and perform constant tracking control so that the camera direction (direction of the camera's line of sight) is always directed at that person (the person's position). More specifically, it is conceivable to perform a process (constant tracking process) that constantly tracks the person being observed by driving the camera unit 20 using a drive unit 27 (motor, etc.) so that the person being observed is always in the center of the captured image. However, with such a technology (comparative example), the driving sound of the camera is generated frequently (continuously) in response to the person's movements. Therefore, there is a high possibility that this will cause stress (psychological burden) to the person.

In this embodiment, of the infinite number of shooting ranges, only a predetermined number (three in this case) of shooting ranges are used as the shooting range (field of view) of the imaging unit 23 (see FIG. 5). In other words, of the infinite number of attitude angles related to the camera unit 20, only three attitude angles related to the camera unit 20 (three predetermined attitude angles) are used (see FIG. 4). In detail, the controller 31 changes the shooting range of the imaging unit 23 to one of the predetermined number (three in this case) of shooting ranges. Then, the controller 31 basically performs a specific observation operation (particularly, event detection processing related to the person being observed) only in one of the three shooting ranges.

More specifically, the controller 31 changes the imaging range of the imaging unit 23 to one of the predetermined number of imaging ranges and stops driving the imaging unit 23 by the drive unit 27, and then observes the observed person based on the captured images (newly captured sequentially) by the imaging unit 23. Specifically, during the period when driving of the imaging unit 23 is stopped, the controller 31 executes, as the observation process of the observed person, processes such as detecting skeletal information of the observed person (specific body part detection process, etc.), detecting the position of the observed person (person position detection process), and detecting behavior-related events related to the observed person (event detection process). However, it is not necessary to execute all of these various processes, and only some of them may be executed. Alternatively, the observation process of the observed person may only involve acquiring the captured images (newly captured sequentially) by the imaging unit 23 and displaying the captured images on the display unit 35b, etc.

In this embodiment, the process of acquiring captured images by the imaging unit 23, as well as the process of detecting skeletal information and the process of detecting a person's position based on the captured images, continue even during the driving period (non-stop period) of the imaging unit 23 (camera unit 20) by the driving unit 27. On the other hand, the process of detecting an event related to the person being observed is temporarily suspended during the driving period (non-stop period) of the imaging unit 23, and resumes after driving is completed (after stopping).

Figure 4 is a diagram showing three attitude angles (three preset attitude angles) θ1, θ2, and θ3 for the camera unit 20. Of the infinite number of attitude angles for the camera unit 20, only the three attitude angles θ1, θ2, and θ3 shown in Figure 4 are used here. Note that the attitude angle θ of the camera unit 20 is expressed as the angle between the horizontal direction and the line of sight of the camera unit 20 (imaging unit 23) (depression angle (downward angle relative to the horizontal direction)).

In the bottom row of Figure 4, the camera unit 20 installed on the ceiling captures an image of the area around the bed directly below (see the bottom row of Figure 5). The angle θ1 of the camera unit 20 at this time is, for example, 90 degrees (depression angle).

In the middle of Figure 4, the camera unit 20 has rotated several tens of degrees (e.g., 25 degrees) counterclockwise around the X axis (in a direction that reduces the depression angle) from a state having an angle θ1, transitioning to a state having an angle θ2. The angle θ2 of the camera unit 20 is, for example, 65 degrees (depression angle). At this time, the camera unit 20 (θ = θ2) is primarily capturing an area that is shifted a certain amount toward the door (entrance) 93 (to the right in Figure 4) from the bed (see the middle of Figure 5).

In the top row of Figure 4, the camera unit 20 has rotated a further few tens of degrees (e.g., 25 degrees) around the X axis (in a direction that reduces the depression angle) from a state having an angle θ2, transitioning to a state having an angle θ3. The angle θ3 of the camera unit 20 is, for example, 40 degrees (depression angle). At this time, the camera unit 20 (θ = θ3) is primarily capturing an area further shifted by a certain amount toward the door (entrance/exit) 93 (to the right in Figure 4) (see the top row of Figure 5).

Note that the specific values of these three angles θ1, θ2, and θ3 are not limited to those described above, and other values may be used as appropriate.

Figure 5 also shows three imaging ranges (three preset imaging ranges) for the imaging unit 23. Of the infinite number of imaging ranges (field of view ranges) for the imaging unit 23, only three imaging ranges (field of view ranges) F1, F2, and F3 shown in Figure 5 are used. The three imaging ranges F1, F2, and F3 correspond to the attitude angles θ1, θ2, and θ3 (of the camera unit 20) in Figure 4, respectively. That is, imaging range F1 is the imaging range corresponding to attitude angle θ1, imaging range F2 is the imaging range corresponding to attitude angle θ2, and imaging range F3 is the imaging range corresponding to attitude angle θ3. In other words, these imaging ranges F1, F2, and F3 are realized by changing the attitude angle θ of the camera unit 20 to angles θ1, θ2, and θ3, respectively.

The shooting range F1 in the bottom row of Figure 5 is a shooting range that shows the view from the ceiling to the area around the bed directly below. Shooting range F1 is a shooting range that can capture a person near the bed (at a relatively close position), and in particular, a shooting range that can capture the entire view of the bed. In other words, shooting range F1 is a shooting range of a close view captured by a camera installed above the room (on the ceiling or upper part of a wall, etc.) looking down on the area directly below.

Shooting range F2 in the middle of Figure 5 is a shooting range that can capture an image of a person who is slightly away from the bed, facing the door (room entrance) 93. Shooting range F2 is also a shooting range that can capture an image of a person who is near the bed. However, compared to shooting range F1, shooting range F2 is a shooting range that can capture an image of a person who is farther away from the bed (and from the imaging unit 23).

The shooting range F3 at the top of Figure 5 is a shooting range that can capture a person who has moved significantly away from the bed (especially a person who has moved closer to the door 93). Shooting range F3 can also be described as a shooting range that can capture a person (at a relatively distant location) near an entrance (room entrance) away from the bed. In other words, shooting range F3 is a distant shooting range captured by a camera installed at the top of the room, looking slightly downward at the distant entrance to the room. In this way, shooting range F3 is a shooting range that can capture a person who is farther away from the bed (and from the imaging unit 23 as well) compared to shooting ranges F1 and F2.

In this way, the three imaging ranges F1, F2, and F3 are imaging ranges (discontinuous imaging ranges) of predetermined stages (three stages) that change in stages from the close view (see the bottom row of Figure 5) to the distant view (see the top row of Figure 5). In other words, imaging ranges F3, F2, and F1 are imaging ranges that can capture people that are increasingly far away from the imaging unit 23, in this order. In more detail, imaging range F3 is the imaging range among imaging ranges F3, F2, and F1 that can capture a person located farthest from the imaging unit 23 (or an origin position closest to the imaging unit 23 (here, the bed position)). Furthermore, imaging range F1 is the imaging range among imaging ranges F3, F2, and F1 that can capture a person located closest to the imaging unit 23 (or an origin position closest to the imaging unit 23 (here, the bed position)).

These three imaging ranges F1, F2, and F3 are set in advance in a pre-adjustment process (described below) that is performed before the observation process (described below) in step S20 (see Figure 9). Furthermore, in this pre-adjustment process, light emission amount setting processing, image pre-processing, bed area setting processing, etc. are performed for each of these three imaging ranges. The pre-adjustment process will be described later.

Note that although three shooting ranges are used as an example of the predetermined number of shooting ranges set in advance, this is not limiting. For example, the predetermined number of shooting ranges may be two shooting ranges, or four or more shooting ranges.

<5. Details of observation processing>
<Changing the shooting range>
In this embodiment, one shooting range is selected from a predetermined number (three in this case) of shooting ranges, and the person to be observed is observed based on the one shooting range in a stopped state where the image capture unit 23 is not driven by the drive unit 27. In other words, the shooting range of the image capture unit 23 is changed to one shooting range (one selected range) out of the predetermined number of shooting ranges, and driving of the image capture unit 23 by the drive unit 27 is stopped, and the person to be observed is observed based on a (newly) captured image by the image capture unit 23.

In more detail, the position of the person being observed within the captured image is detected as needed (at a predetermined sampling time interval) based on the captured image. The imaging unit 23 is driven by the driving unit 27 only when the detected position (position of the person being observed) moves within a trigger area G (described below) in the captured image. Specifically, a process of changing the imaging range of the imaging unit 23 (a process of changing from one imaging range to another) is performed. The person being observed is then observed based on the captured image captured within the changed imaging range.

On the other hand, if the position of the person being observed is within the non-trigger area C in the captured image (area C excluding trigger area G (described below)), the driving unit 27 does not drive the imaging unit 23. Then, with the driving unit 27 not being driven and the imaging range of the imaging unit 23 maintained at that single imaging range (camera drive stopped state), the person being observed is observed based on the captured image (further) captured by the imaging unit 23.

This type of processing suppresses the occurrence of drive operations. Therefore, it is possible to avoid or suppress the occurrence of stress (in the observed person) caused by drive sounds (such as motor drive sounds).

Figures 6 and 7 are diagrams explaining this process of changing the shooting range (selected range). This change process can also be expressed as a process of switching (selecting) between the three shooting ranges F1, F2, and F3.

As shown in Figures 6 and 7, a "trigger area" (area that triggers an update of the shooting area) G and a "non-trigger area" (area other than the trigger area) C are set for each shooting area F. Note that in Figures 6 and 7, trigger areas G are marked with diagonal hatching or cross hatching, and non-trigger areas C are marked with sand hatching.

Specifically, within the imaging range F1, the upper end area within the imaging range F1 (for example, the upper 20%) is set as the trigger area G (more specifically, G11). On the other hand, the area outside the trigger area G11 (the area excluding the upper edge G11) is set as the non-trigger area C (more specifically, C1).

Within the shooting range F2, the upper end area (e.g., the upper 20%) of the shooting range F2 is set as the trigger area G (more specifically, G21). Furthermore, the lower end area (e.g., the lower 20%) of the shooting range F2 is set as the trigger area G (more specifically, G22). Meanwhile, the area outside the trigger areas G21 and G22 (the central area) is set as the non-trigger area C (more specifically, C2).

In the shooting range F3, the lower end area of the shooting range F1 (for example, 20% from the top to the bottom) is set as the trigger area G (specifically, G32). Meanwhile, the area outside the trigger area G32 is set as the non-trigger area C (specifically, C3).

The trigger areas G11 and G21 on the upper end are trigger areas used when changing the shooting range (sequentially) to a relatively distant view (F1 → F2 → F3). Conversely, the trigger areas G32 and G22 on the lower end are trigger areas used when changing the shooting range (sequentially) to a relatively close view (F3 → F2 → F1).

The position of the observed person is obtained as planar position information of a specified part within the captured image based on the skeletal information 140 of the observed person. Furthermore, when changing the shooting range to a relatively distant view and when changing the shooting range to a relatively close view, the positions of different parts of the body may be used as the position of the observed person. Here, when changing the shooting range to a relatively distant view, the position of the person's neck is used as the position of the observed person, and the position of the person's waist is used as the position of the observed person.

<Example of operation>
A more specific example of the operation will be described below.

As shown in the bottom row of Figure 6, when the imaging range of the imaging unit 23 is imaging range F1, the positional relationship between the position of the person's neck and trigger area G11 in the captured image captured in imaging range F1 is determined. Then, only when the detected position (position of the person's neck) moves into trigger area G11 in the captured image is the driving unit 27 driven to change the imaging range of the imaging unit 23 to imaging range F2.

For example, when the imaging range of the imaging unit 23 is imaging range F1 (see the bottom row in Figure 6), consider a situation in which a person (person being observed) is sleeping in bed, or the person has gotten out of bed and is now standing next to the bed. In such a situation, the position of the person (more specifically, the position of their neck) is determined to still be within the non-trigger area C1. Then, the driving unit 27 does not drive the imaging unit 23, and the imaging range of the imaging unit 23 is maintained at imaging range F1.

Now, imagine a situation in which the person gets off the bed and walks toward the center of the room. In this situation, if it is determined that the person's position (specifically, the position of their neck) has moved into trigger area G11, the driving unit 27 drives the imaging unit 23, and the shooting range of the imaging unit 23 is changed from shooting range F1 to shooting range F2. Within the updated shooting range F2, the person is present (photographed) near the center. The person's position is within the non-trigger area C2 of shooting range F2.

Similarly, when the imaging range of the imaging unit 23 is imaging range F2 (see the middle of Figure 6), the positional relationship between the position of the person's neck and trigger areas G21 and G22 in the captured image captured in imaging range F2 is determined. Then, only when the detected position (position of the person's neck) moves into trigger area G21 (, G22) in the captured image is the driving unit 27 driving the imaging unit 23 and the imaging range of the imaging unit 23 changing to imaging range F3 (, F1).

For example, consider a situation in which the imaging range of the imaging unit 23 is imaging range F2 (see the middle of Figure 6) and a person is present near a chair in the center of a living room. In this situation, the position of the person (specifically, the position of their neck) is determined to still be within the non-trigger area C2. The driving unit 27 does not drive the imaging unit 23, and the imaging range of the imaging unit 23 is maintained at imaging range F2. In a situation in which a person is sitting and relaxing in the chair, no driving operation is performed by the driving unit 27.

On the other hand, if the person (person being observed) moves further toward door 93, it is determined that the position of the person (specifically, the position of their neck) has moved into trigger area G21. In response, the drive unit 27 drives the imaging unit 23, and the shooting range of the imaging unit 23 is changed from shooting range F2 to shooting range F3. Within the updated shooting range F3, the person is present (photographed) near the center. The person's position is within non-trigger area C3 of shooting range F3.

Furthermore, in situations where the person is located within the non-trigger area C3 of the imaging range F3 (such as when the person is doing exercises in the non-trigger area C3 or sitting in a chair (not shown)), the driving unit 27 does not drive the imaging unit 23, and the imaging range of the imaging unit 23 is maintained within the imaging range F3.

Conversely, as shown in Figure 7, when a person moves from near the door 93 to the bed 92, the positional relationship between the lower trigger areas G32, G22 and the person's position (more specifically, the position of the person's waist) is determined, and the imaging range of the imaging unit 23 is sequentially changed from imaging range F3 to imaging range F1 (F3 → F2 → F1).

For example, assume that the imaging range of the imaging unit 23 is imaging range F3 (see the top row of Figure 7), and a person is walking from the door (entrance/exit) 93 toward the bed.

In this situation, when the person's position (more specifically, the position of the person's waist) is within non-trigger area C3, no drive is performed by drive unit 27 and imaging range F3 is maintained. On the other hand, when the person's position (more specifically, the position of the person's waist) moves into trigger area G32, drive is performed by drive unit 27 and the imaging range of image capture unit 23 is changed from imaging range F3 to imaging range F2 (see middle section of Figure 7). Immediately after the change, the person's position is within non-trigger area C2 of imaging range F2.

After that, when the person's position (more specifically, the position of the person's waist) is within the non-trigger area C2, no driving is performed by the driving unit 27, and the imaging range F2 is maintained. On the other hand, when the person's position (more specifically, the position of the person's waist) moves into the trigger area G22, the driving unit 27 is driven, and the imaging range of the imaging unit 23 is changed from imaging range F2 to imaging range F1 (see the bottom row of Figure 7).

Note that the example shown here mainly illustrates a transition in which the shooting range of the imaging unit 23 changes from shooting range F1 to shooting range F3 (F1 → F2 → F3) (see Figure 6), or from shooting range F3 to shooting range F1 (F3 → F2 → F1) (see Figure 7). However, this is not limited to this, and the shooting range of the imaging unit 23 may change from shooting range F1 to shooting range F2 and then return to shooting range F1. Alternatively, the shooting range of the imaging unit 23 may change from shooting range F3 to shooting range F2 and then return to shooting range F3.

<Processing flow details>
Fig. 9 is a flowchart showing the above-described observation process (monitoring process) in detail. The process of Fig. 9 (also referred to as the process of step S20) is executed by the controller 31. It is assumed that the pre-adjustment process (the process of step S10) described below has already been executed prior to the process of step S20.

First, in step S21, various initial processes are performed. Specifically, processes such as setting the initial imaging range (for example, when the care recipient is lying in bed 92, setting imaging range F1 as the imaging range of the imaging unit 23 in response to input from the caregiver) are performed. Various processes corresponding to the initial imaging range are also performed. For example, various settings prepared in advance for imaging range F1 are employed. In detail, the light emission amount (light emission amount of the illumination unit 25), image pre-processing parameters (brightness adjustment parameters, etc.), and bed area prepared in advance for imaging range F1 are employed.

Next, in step S22, a captured image (a captured image of the capturing range F1) is acquired by the imaging unit 23, and an observation process is performed to observe the person based on the captured image. This observation process may include, for example, a process for detecting the person's skeletal information, a process for detecting the person's position, and a process for detecting events related to the person's behavior.

In step S23, it is determined whether the person (observed person) is located in the trigger area G or the non-trigger area C.

If it is determined that a person is present in the non-trigger area C, the camera remains stopped (step S24) and the process returns to step S22. The observation process continues with the camera stopped by looping through steps S22, S23, and S24.

On the other hand, if it is determined that a person is present within trigger area G, the process proceeds from step S23 to step S25, where the driving operation by drive unit 27 changes the shooting range of image capture unit 23. For example, a transition from shooting range F1 to shooting range F2 is performed. In addition, in step S26, a change process is performed to various settings that have been prepared in advance to correspond to the changed shooting range. In detail, a change process is performed to the light emission amount (light emission amount of illumination unit 25), image pre-processing parameters, bed area, etc. that have been prepared in advance to correspond to the changed shooting range (for example, shooting range F2).

<Effects of the embodiment>
As described above, in the above embodiment, the shooting range of the imaging unit 23 is changed to one of a predetermined number of shooting ranges (one selected range) and the driving of the imaging unit 23 by the driving unit 27 is stopped, and the person to be observed is observed based on the captured image captured by the imaging unit 23.

This type of processing makes it possible to avoid or reduce stress (for the person being observed) caused by the drive noise (motor drive noise, etc.) of the drive unit 27. Furthermore, by switching between a predetermined number of shooting ranges, it is possible to ensure a wider observation range.

In addition, a relatively small number of cameras (one in this case) are required to ensure a wide observation range, which helps reduce costs compared to installing multiple cameras.

In particular, in the above embodiment, if the detected position of the person being observed falls within the non-trigger area C in the captured image, the drive unit 27 is not driven and the capture range of the image capture unit 23 is maintained at one capture range. In other words, the drive unit 27 drives the image capture unit 23 only when the detected position (detected position of the person being observed) moves into the trigger area G (described below) in the captured image.

This ensures that the drive remains stopped even if the person moves slightly within the shooting range. This makes it possible to more reliably avoid or reduce stress (in the person being observed) caused by the drive noise (motor drive noise, etc.) of the drive unit 27.

<7. Pre-adjustment processing>
Next, a pre-adjustment process (step S10 (not shown)) for a predetermined number of shooting ranges (here, three shooting ranges F1, F2, and F3) will be described. This pre-adjustment process (also referred to as a pre-setting process) is executed prior to the observation process (monitoring process) (step S20 in FIG. 9) described above.

First, as described above, the camera attitude angle θi corresponding to each of the three imaging ranges Fi (i = 1, 2, 3) is set. Specifically, angle θ1 is set for imaging range F1, angle θ2 is set for imaging range F2, and angle θ3 is set for imaging range F3.

In addition, settings such as light emission intensity, image preprocessing parameters, and bed area settings are performed for each shooting range Fi.

The light emission intensity setting is a process for adjusting and setting the light emission intensity of the illumination unit 25 (more specifically, the intensity of infrared light used for distance measurement in the TOF method) for each imaging range Fi. The light emission intensity Li (see Figure 8) appropriate for each imaging range Fi is set in advance. By setting the light emission intensity in this way, the light emission intensity Li corresponding to the imaging range Fi is determined in advance (before step S20).

Shooting range F1 is a shooting range where the distance to the person being observed is relatively close (closer than the other shooting ranges), while shooting range F3 is a shooting range where the distance to the person being observed is relatively far (farther than the other shooting ranges). Reflecting this situation, the light emission amount L3 corresponding to shooting range F3 is set to a value greater than the light emission amount L1 corresponding to shooting range F1. Furthermore, the light emission amount L2 corresponding to shooting range F2 is set to an intermediate value (L1 < L2 < L3). Note that in cases where there is no significant difference in the appropriate light emission amount, L1 = L2 or L2 = L3 may be acceptable.

This makes it possible to simplify adjustment of the light emission amount, since adjustments only need to be made for a predetermined number of shooting ranges Fi (here, three), rather than an infinite number of shooting ranges. It is also possible to appropriately set the light intensity suitable for each of the limited number of shooting ranges Fi. For example, when shooting range F3 is photographed in step S20 (see Figure 9), a sufficiently large light emission amount L3 is used, thereby ensuring a sufficient amount of light (infrared light intensity for distance measurement) that reaches relatively distant subjects (people, etc.). On the other hand, when shooting range F1 is photographed, a (relatively small) light emission amount L1 is used, thereby making it possible to appropriately suppress the light emission amount.

Image pre-processing parameter setting is a process for adjusting the brightness of the captured image. Because the distance to the subject differs for each shooting range Fi and the light emission amount Li differs for each shooting range Fi, it is preferable to set parameters that appropriately adjust the brightness of the captured image. In image pre-processing parameter setting, appropriate parameters are set for each shooting range Fi. For example, adaptive histogram smoothing processing is performed, and appropriate parameters (brightness adjustment parameters) are set.

In this embodiment, image preprocessing parameter settings only require adjustment for a predetermined number of shooting ranges Fi (here, three), rather than an infinite number of shooting ranges. This simplifies adjustment.

Bed area setting is a process for setting (specifying) a bed area within a captured image. Bed area setting is performed by, for example, drawing and specifying the range of the upper surface area of the bed 92 within each capturing range Fi using a polygon (rectangle) or the like within the captured image of each capturing range Fi. Furthermore, the three-dimensional position of the bed 92 is identified by, for example, specifying the bed height by numerical input. Behavioral events related to the person (observed person) are detected using the positional relationship between the set bed area (three-dimensional bed space) and each specific part of the person.

In this embodiment, bed area setting requires only setting operations for three imaging ranges Fi, rather than an infinite number of imaging ranges. This simplifies bed area setting. In particular, when bed area setting is performed by a caregiver (care staff), the burden on the caregiver can be reduced.

Furthermore, for a limited number of imaging ranges Fi, it is possible to specify each bed area relatively easily by specifying it two-dimensionally.

<8. Modifications, etc.>
Although the embodiment of the present invention has been described above, the present invention is not limited to the above-described contents.

<Unmoving object determination>
For example, in the above embodiment, a moving object determination (stationary object determination) may be further performed using differential images in each shooting range Fi, which makes it possible to accurately distinguish a person (stationary object) in a poster posted on a wall (or floor, etc.) from a living person (moving object).

Specifically, in each shooting range Fi, moving objects are detected based on a difference image between an image (still image) taken at a certain point in time and an image (still image) taken at another point in time. More specifically, the difference-occurring area (different area) in the difference image is detected as the moving object.

In more detail, when a person's skeleton is detected in a certain area within the shooting range Fi based on image features, a difference image is calculated between the area where the skeleton is detected in an image captured at a certain point in time (detection area) and an area at the same position in an image captured at a different point in time. Then, if it is determined that there is a significant difference between the two areas based on the difference image, it is determined that a moving object is present in the detection area. In other words, it is determined that a living person is present.

When only a predetermined number of (here, three) shooting ranges Fi are used for person detection, rather than an infinite number of shooting ranges, the same shooting range Fi continues for a certain period of time or longer, and images captured at different times using the same shooting range Fi are obtained. Therefore, it is possible to perform moving object (stationary object) detection using differential images in each shooting range Fi. In other words, by employing a detection process using differential images, it is possible to perform moving object (stationary object) detection relatively easily.

<2D camera>
Furthermore, in the above embodiment, a three-dimensional camera is used as the camera unit 20, but this is not limitative, and the camera unit 20 may be a two-dimensional camera.

<Learning model for each shooting range>
In addition, in the above embodiment, the same learning model 400 is used in common for the three shooting ranges Fi, but this is not limited to this. For example, a different learning model 400 may be used for each of the three shooting ranges Fi. Specifically, an individual learning model specialized for each shooting range Fi may be used. Such a modified example will be described below.

In this modified example, for the imaging range F1, a learning model 400 (401 (not shown)) suitable for the person's pose (positional relationships between joints) in the image captured from a top-down perspective is used. On the other hand, for the imaging range F3, a learning model 400 (403 (not shown)) suitable for the person's pose (positional relationships between joints) in the image captured from a side-on perspective is used. Furthermore, for the imaging range F2, a learning model 400 (402 (not shown)) suitable for the person's pose (positional relationships between joints) in the image captured from an intermediate perspective is used. Note that it is not necessary to use a different learning model 400 for each imaging range Fi. For example, for the imaging range F2, the same learning model 401 (or 403) as for the imaging range F1 (or F3) may be used.

For example, in a captured image of imaging range F1, the joints of a person lying on a bed (a person in a recumbent position), such as the neck, shoulders, waist, knees, and ankles, are arranged in a plane with a relatively large gap between them and a specific positional relationship (two-dimensional (planar) positional relationship). On the other hand, the joints of a person in a standing position, such as the neck, shoulders, waist, knees, and ankles, are located relatively close together and have a specific positional relationship within the captured image. Furthermore, the joints of the neck, chest, and waist of a person in a sitting position have a positional relationship similar to that in a standing position, but the knee and ankle joints are located slightly away from the waist (in a certain direction). For imaging range F1, a learning model 401 may be used that machine-learns such specific positional relationships within the captured image in a planar manner (specific positional relationships when viewed from above).

Conversely, in the captured image of imaging range F3, the joints of a person in a standing position, such as the neck, shoulders, waist, knees, and ankles, are arranged at relatively large intervals in the vertical direction (so as to have a specific positional relationship). Furthermore, the joints of the neck, chest, and waist of a person in a sitting position are arranged in the vertical direction, while the joints of the waist and knees are arranged in a direction other than the vertical direction (such as the horizontal direction). For imaging range F3, a learning model 403 may be used that machine-learns specific planar positional relationships (specific positional relationships when viewed from the side) within such captured images. Furthermore, because imaging range F3 hardly includes the area near the bed, learning model 403 may not need to detect some events, such as "slipping off the bed."

If one learning model (common learning model) were to be constructed that corresponds to an infinite number of shooting ranges F corresponding to an infinite number of shooting angles θ, the learning model 400 would often become a relatively large (complex) model (a model with a large number of parameters and/or a heavy processing load) in order to appropriately reflect the diverse situations corresponding to the extremely wide variety of shooting angles.

On the other hand, in the above-described modified example, a learning model 400 is constructed according to an image captured from a specific shooting angle (specific shooting range), more specifically, a learning model 400 specialized for each shooting range Fi. This makes it possible to construct relatively small (simple) learning models 401, 402, 403 (models with a low processing load and/or a small number of parameters) that appropriately reflect the unique characteristics of each shooting range Fi.

This makes it possible to reduce the processing load on the CPU and/or reduce memory capacity. As a result, it is possible to operate each of the learning models 401, 402, and 403 at high speed (by reducing the processing time, etc.) on a CPU with the same performance.

REFERENCE SIGNS LIST 1 Observation system 10 Observation device 20 Camera unit 23 Imaging unit 25 Lighting unit 27 Driving unit 30 Processing unit 90 Living room 91 Floor 92 Bed 93 Door (entrance/exit)
θi Camera unit attitude angle (photography angle)
Fi, F1, F2, F3 Shooting range C, C1, C2, C3 Non-trigger area G, G11, G21, G22, G32 Trigger area Li Light output

Claims (12)

an imaging unit that captures an image of an observation target space;
a drive unit that rotates the imaging unit to change the imaging range of the imaging unit;
an illumination unit that emits illumination light for illuminating a photographing range of the imaging unit;
a control unit that controls the rotational driving of the imaging unit by the drive unit to change the imaging range of the imaging unit to one of a predetermined number of imaging ranges that have been set in advance, and also controls the emission of illumination light by the illumination unit;
Equipped with
an observation system characterized in that the control unit changes the shooting range of the imaging unit to one of the predetermined number of shooting ranges and stops the rotational driving of the imaging unit by the drive unit, causes the lighting unit to emit illumination light of an amount of light emission corresponding to the one shooting range from among the light emission amounts predetermined for each of the predetermined number of shooting ranges, and performs observation processing of the person being observed based on the captured image captured by the imaging unit. The control unit
Detecting the position of the person to be observed within the captured image;
The observation system described in claim 1, characterized in that when the position of the person being observed falls within a predetermined area in the captured image, the driving unit is not driven and the capturing range of the imaging unit is maintained at the one capturing range, and an observation process of the person being observed is performed based on the captured image captured by the imaging unit. the predetermined number of imaging ranges includes a first imaging range and a second imaging range,
3. The observation system according to claim 1, wherein the second shooting range is a shooting range that is capable of capturing an image of the person being observed that is farther away from the imaging unit than the first shooting range. the predetermined number of imaging ranges includes a first imaging range and a second imaging range,
The imaging unit is installed near and above the bed,
the first imaging range is an imaging range in which the person to be observed who is present near the bed can be imaged,
3. The observation system according to claim 1, wherein the second imaging range is a imaging range that can capture an image of the person to be observed who is present near an entrance away from the bed. The observation system described in claim 3 or claim 4, wherein the light emission amount determined for the second imaging range is greater than the light emission amount determined for the first imaging range. a computer that controls an observation system that includes an imaging unit that captures an image of a space to be observed, a drive unit that rotates and drives the imaging unit to change the imaging range of the imaging unit, and an illumination unit that emits illumination light to illuminate the imaging range of the imaging unit;
a) controlling the rotational driving of the imaging unit by the driving unit to change the imaging range of the imaging unit to one of a predetermined number of imaging ranges set in advance;
b) changing the imaging range of the imaging unit to one of the predetermined number of imaging ranges and stopping the rotational driving of the imaging unit by the drive unit, causing the lighting unit to emit illumination light of an amount of light emission corresponding to the one imaging range from among light emission amounts predetermined for each of the predetermined number of imaging ranges, and executing an observation process of the person to be observed based on the captured image captured by the imaging unit;
A program to execute. A control method for an observation system including an imaging unit that captures an image of an observation target space, a drive unit that rotationally drives the imaging unit to change an imaging range of the imaging unit, and an illumination unit that emits illumination light to illuminate the imaging range of the imaging unit,
a) controlling the rotational driving of the imaging unit by the driving unit to change the imaging range of the imaging unit to one of a predetermined number of imaging ranges set in advance;
b) changing the imaging range of the imaging unit to one of the predetermined number of imaging ranges and stopping the rotational driving of the imaging unit by the drive unit, causing the lighting unit to emit illumination light of an amount of light emission corresponding to the one imaging range from among light emission amounts predetermined for each of the predetermined number of imaging ranges, and executing an observation process of the person to be observed based on the captured image captured by the imaging unit;
A control method for an observation system, comprising: an imaging unit that captures an image of an observation target space;
a drive unit that rotates the imaging unit to change the imaging range of the imaging unit;
a control unit that controls the rotational driving of the imaging unit by the drive unit to change the imaging range of the imaging unit to one of a predetermined number of imaging ranges that are set in advance;
Equipped with
The control unit
After setting the imaging range of the imaging unit to one of the predetermined number of imaging ranges, detecting the position of an observed person in a captured image corresponding to the one imaging range;
An observation system characterized in that, when the position of the person being observed falls within a predetermined area in the captured image, the imaging unit is not rotated, and the imaging range of the imaging unit is maintained at the one imaging range, and observation processing of the person being observed is performed based on the captured image. The observation system described in claim 8, characterized in that, when the position of the person being observed no longer falls within the predetermined area, the control unit rotates the imaging unit to change the imaging range of the imaging unit to a different imaging range from the first imaging range. The observation system described in claim 8 or claim 9, wherein the observation process of the observed person includes a process of detecting behavior-related events of the observed person. a computer for controlling an observation system including an imaging unit that captures an image of an observation target space and a drive unit that rotationally drives the imaging unit to change the imaging range of the imaging unit;
a) controlling the rotational driving of the imaging unit by the drive unit to set the imaging range of the imaging unit to one imaging range that is one of a predetermined number of imaging ranges that have been set in advance;
b) detecting the position of a person to be observed within a captured image corresponding to the one capturing range;
c) when the position of the person to be observed is within a predetermined area in the captured image, performing an observation process of the person to be observed based on the captured image while maintaining the imaging range of the imaging unit at the one imaging range without rotating the imaging unit;
A program to execute. A control method for an observation system including an imaging unit that captures an image of an observation target space and a drive unit that rotationally drives the imaging unit to change an imaging range of the imaging unit,
a) controlling the rotational driving of the imaging unit by the drive unit to set the imaging range of the imaging unit to one imaging range that is one of a predetermined number of imaging ranges that have been set in advance;
b) detecting the position of a person to be observed within a captured image corresponding to the one capturing range;
c) when the position of the person to be observed is within a predetermined area in the captured image, performing an observation process of the person to be observed based on the captured image while maintaining the imaging range of the imaging unit at the one imaging range without rotating the imaging unit;
A control method for an observation system, comprising: