JP7799644B2 - Moving object estimation system, moving object estimation method, and program - Google Patents

Description

This disclosure relates to a moving object estimation system, a moving object estimation method, and a program, and more specifically to a moving object estimation system, a moving object estimation method, and a program that use a radio wave sensor to make estimations regarding moving objects (moving objects) such as human bodies.

Patent Document 1 describes an FMCW radar device that transmits a frequency-modulated transmission signal so that the frequency changes linearly over time, receives a received signal that is the transmission signal reflected by an object, and measures the distance to the object from the frequency of a beat signal obtained by multiplying the received signal by a frequency-modulated local signal.

Japanese Patent Application Laid-Open No. 2022-108639

Radio wave sensors that use radio waves modulated by FMCW or other methods can be used to make inferences about moving objects (for example, to estimate whether an object is moving or not (stationary), to estimate whether the moving object is a human or a moving object other than a human, to identify the position of a moving object (especially a human), and even to estimate the posture of a human). However, it is generally not easy to achieve enough precision with radio wave sensors to make inferences about moving objects, especially humans. For example, to be able to estimate the posture of a human body using an FMCW radio wave sensor, it was necessary to increase the number of antennas (for example, to use an expensive radio wave sensor with dozens of antennas).

The purpose of the present disclosure is to provide a moving object estimation system, a moving object estimation method, and a program that can improve the accuracy of estimations regarding moving objects using radio wave sensors.

A moving object estimation system according to one aspect of the present disclosure is a moving object estimation system that uses a radio wave sensor to make estimations regarding a moving object. The moving object estimation system comprises the radio wave sensor, a conversion unit, a storage unit, a difference acquisition unit, a moving object estimation unit, and a point placement unit. The radio wave sensor performs a transmission/reception operation of transmitting a transmission wave, which is a radio wave modulated using a predetermined method, toward real space, receiving a reflected wave from the real space, and outputting an output signal based on the transmission wave and the reflected wave. The conversion unit performs a conversion process of converting the output signal output by the radio wave sensor into location-identifiable information that can identify the position of each of one or more objects present in the real space. The storage unit performs a storage process of storing the location-identifiable information converted by the conversion process. The difference acquisition unit performs a difference acquisition process of acquiring the difference between multiple pieces of location-identifiable information stored by the storage process. The moving object estimation unit performs a moving object estimation process in which it estimates whether each of the one or more objects is a moving object or a stationary object based on the differences acquired by the difference acquisition process, identifies the position of the object estimated to be a moving object, and acquires moving object position information indicating the identified position. The point placement unit performs a point placement process in which it virtually places points corresponding to the moving object position information acquired by the moving object estimation process in a virtual space corresponding to the real space. The difference acquisition process acquires two or more differences with different time differences for three or more pieces of position identifiable information at different times that are stored in the storage process. The moving object estimation process acquires two or more pieces of moving object position information corresponding to the two or more differences acquired by the difference acquisition process. The point placement process virtually places two or more points in the virtual space that correspond to the two or more pieces of moving object position information with different time differences that are acquired by the moving object estimation process.

A moving object estimation method according to one aspect of the present disclosure is a moving object estimation method that uses a radio wave sensor to make an estimation regarding a moving object. The moving object estimation method comprises a transmission/reception step, a conversion step, a retention step, a difference acquisition step, a moving object estimation step, and a point placement step. In the transmission/reception step, the radio wave sensor performs a transmission/reception operation in which it transmits a transmission wave, which is a radio wave modulated using a predetermined method, toward real space, receives a reflected wave from the real space, and outputs an output signal based on the transmission wave and the reflected wave. The conversion step performs a conversion process in which the output signal output by the radio wave sensor is converted into location-identifiable information that can identify the position of each of one or more objects present in the real space. The retention step performs a retention process in which the location-identifiable information converted by the conversion process is retained. The difference acquisition step performs a difference acquisition process in which the difference between multiple pieces of location-identifiable information retained by the retention process is acquired. The moving object estimation step performs moving object estimation processing to estimate whether each of the one or more objects is a moving object or a stationary object based on the differences acquired by the difference acquisition processing, identify the position of the object estimated to be a moving object, and acquire moving object position information indicating the identified position. The point placement step performs point placement processing to virtually place points corresponding to the moving object position information acquired by the moving object estimation processing in a virtual space corresponding to the real space. The difference acquisition processing acquires two or more differences with different time differences for three or more pieces of position identifiable information at different times stored in the storage processing. The moving object estimation processing acquires two or more pieces of moving object position information corresponding to the two or more differences acquired by the difference acquisition processing. The point placement processing virtually places two or more points in the virtual space corresponding to the two or more pieces of moving object position information acquired by the moving object estimation processing.

A program according to one aspect of the present disclosure causes one or more processors to execute the moving object estimation method.

The moving object estimation system, moving object estimation method, and program disclosed herein have the effect of improving the accuracy of estimation of moving objects using radio wave sensors.

FIG. 1 is a block diagram of a device control system (human body posture estimation system) according to an embodiment of the present disclosure. FIG. 2 is a conceptual diagram of a room in which the above-mentioned device control system is used. FIG. 3 is a graph showing changes in frequency of a transmission wave transmitted by a radio wave sensor constituting the device control system. FIG. 4 is a flowchart illustrating the operation of the control device constituting the device control system. FIG. 5 is a flowchart illustrating the one-to-multiple frame difference acquisition process included in the above operation. FIG. 6 is a flowchart illustrating a posture estimation process included in the above operation. FIG. 7 is a waveform diagram for explaining an example of inter-frame difference (one-to-one inter-frame difference). FIG. 8A is a frequency spectrum diagram showing the FFT result in the current frame Fr0, FIG. 8B is a frequency spectrum diagram showing the FFT result in the subsequent frame Fr1, and FIG. 8C is a frequency spectrum diagram showing the difference between the FFT results. FIG. 9 is a waveform diagram for explaining another example of inter-frame difference (one-to-multiple inter-frame difference). FIG. 10 is a conceptual diagram for explaining the characteristics of various movements of the human body. FIG. 11 is a waveform diagram for explaining an example of obtaining one-to-multiple frame differences according to various movements in the embodiment. FIG. 12A is a distribution diagram showing an example of cluster distribution in a standing position, which is one of the postures of the human body; FIG. 12B is a distribution diagram showing an example of cluster distribution in a sitting position; and FIG. 12C is a distribution diagram showing an example of cluster distribution in a lying position. Figure 13A is a conceptual diagram showing a three-dimensional figure surrounding a cluster group in a standing position, Figure 12B is a conceptual diagram showing a three-dimensional figure surrounding a cluster group in a sitting position, and Figure 12C is a conceptual diagram showing a three-dimensional figure surrounding a cluster group in a lying position. FIG. 14 is a data structure diagram of the automatic control information group. FIG. 15 is a data structure diagram of the control history information. FIG. 16 is a block diagram of a modified example of the device control system.

(1) Overview of the Device Control System In this embodiment, the moving object estimation system of the present disclosure is a device control system 100 that has a device control function in addition to a moving object estimation function (e.g., a human body estimation function, particularly a human body posture estimation function). As shown in Fig. 2 , the device control system 100 of this embodiment uses a radio wave sensor 1 to perform estimation of a moving object (hereinafter, sometimes referred to as a "moving object"), particularly a human body 301.

Estimation regarding the human body 301 includes, for example, estimating whether it is the human body 301, a moving object other than the human body 301, or a non-moving object (stationary object), as well as estimating the posture of the human body 301.

The posture of the human body 301 may be, for example, standing, sitting, lying down, etc. (see Figures 12A to 12C), but is not limited to these. The device control system 100 then detects various movements of the human body 301, and ultimately the person's behavior, based on the estimated changes in posture, and controls the device 200 according to the detection results.

Note that a change in posture may be, for example, a change between a standing position, a sitting position, and a lying position, but is not limited to this. A change in posture also includes a continuation of no change in posture (for example, when a predetermined period of time has passed since changing to a sitting position, but there is no change to a standing position, etc.). Furthermore, the behavior to be detected is, in particular, a non-operation behavior (described below), but may also be an operation behavior (described below).

(1-1) Radio wave sensor The radio wave sensor 1 transmits radio waves (transmission waves Tr) from an antenna, receives reflected waves Re that are reflected by an object from the transmission waves Tr, and outputs information that can identify the position of the object (hereinafter referred to as "position-identifying information").

(1-1-1) Position Identifiable Information Position Identifiable Information is information that can identify the position of an object. The position of the object is preferably a three-dimensional position, but may also be a two-dimensional or one-dimensional position. The position Identifiable Information is, for example, an FFT result group (information that can identify a three-dimensional position) described below, but may also be each of multiple (e.g., three) FFT results that make up the FFT result group (information that can identify a one-dimensional position).

Furthermore, for example, a signal indicating the time difference Δt between the transmission of the transmitted wave Tr and the reception of the reflected wave Re, or a signal indicating the frequency difference Δf corresponding to the time difference Δt (for example, an IF signal indicating the frequency difference Δf between the transmitted wave Tr modulated by the FMCW method and the reflected wave Re: see Figure 3), may also be considered a type of location-identifying information. Furthermore, the distance calculated from the time difference Δt or frequency difference Δf, etc., may itself be location-identifying information.

(1-1-2) Antenna The antenna used to transmit the transmission wave Tr and the antenna used to receive the reflected wave Re may be the same antenna (hereinafter referred to as a "shared antenna") or may be separate antennas (hereinafter referred to as a "transmitting antenna" and a "receiving antenna"). In other words, the transmission wave Tr may be transmitted from the shared antenna, and the reflected wave Re of the transmission wave Tr may be received by the shared antenna, or the transmission wave Tr may be transmitted from the transmitting antenna, and the reflected wave Re of the transmission wave Tr may be received by the receiving antenna.

Generally, to identify the three-dimensional position of an object (for example, to locate a point corresponding to a human body in a virtual three-dimensional space 500), the radio wave sensor 1 must have, for example, three or more shared antennas, or one transmitting antenna and three or more receiving antennas. Note that when the radio wave sensor 1 has multiple antennas, this does not necessarily mean that the multiple antennas are housed in a single housing, but also includes cases where they are located in multiple locations separated from one another. In the latter case, each of the multiple antennas is connected to the sensor body via wire or wirelessly so that they can communicate with the sensor body.

In this embodiment, the radio wave sensor 1 has one transmitting antenna and three or more (e.g., three) receiving antennas. The positions of the one transmitting antenna and three or more receiving antennas included in the radio wave sensor 1 are known, and four or more pieces of position information corresponding to the four or more antennas (hereinafter referred to as "antenna position information group") are pre-stored, for example, in the memory of the control device 2.

However, the number of receiving antennas that make up the radio wave sensor 1 may be two or one (two antennas allow for point placement on a plane, while one antenna allows for point placement along a line).

(1-1-3) Transmission and Reception Operation The radio wave sensor 1 performs a transmission and reception operation. The transmission and reception operation is an operation of transmitting a transmission wave Tr toward a real space in which a group of objects may exist, receiving a reflected wave Re from the real space, and outputting a signal (e.g., an IF signal) based on the transmission wave Tr and the reflected wave Re. The real space in which a group of objects may exist is, for example, a space (indoor space) surrounded by the floor, ceiling, and side walls of a room 400, and may hereinafter be referred to as the "real space (400)." Note that the real space in which a group of objects may exist is not limited to an indoor space, but may also be an outdoor space such as a corridor or terrace.

An object group is a collection of one or more objects. In this embodiment, the object group includes one or more of a human body 301, a moving object other than a human body (for example, electric blinds 200c, hereinafter sometimes referred to as "moving object other than a human body (200c)"), and a stationary object 302. In other words, each of the one or more objects that make up the object group is either a human body 301, a moving object other than a human body (200c), or a stationary object 302.

The radio wave sensor 1 performs the following transmission and reception operation at a predetermined cycle (e.g., once every 20 ms). This transmission and reception operation involves transmitting a transmission wave Tr from one transmitting antenna to a real space (400) in which a group of objects may exist, and receiving reflected waves Re from the real space (400) with each of three or more receiving antennas.

For example, if a human body 301, a moving object other than a human body (200c), and a stationary object 302 exist in the real space (400), the reflected waves Re from the group of objects (301, 200c, 302) are received by each of three or more receiving antennas. Note that in the example of Figure 2, the stationary object 302 is, for example, a desk 302a and a bed 302b placed on the floor of the room 400, but it may also be the floor.

The transmission wave Tr is a radio wave modulated using a predetermined method. The predetermined method is, for example, the FMCW (Frequency Modulated Continuous Wave) method, but is not limited to this. In this embodiment, the transmission wave Tr is a radio wave modulated using the FMCW method.

(1-1-4) FMCW Method The FMCW method is a method in which, as shown in FIG. 3, the frequency f of a transmission wave Tr (transmission signal) having a predetermined time length (chirp length Tc of the chirp signal) is linearly increased (or decreased) from a starting frequency f0 at a predetermined slope S as time t passes.

Note that the acquisition of location-identifiable information as described above does not have to be performed for all transmission and reception operations that are repeated at a predetermined cycle. For example, as described below, if a predetermined time T (e.g., T = 200 ms) is defined as one frame (described below) and N transmission and reception operations (N is a natural number, e.g., 10) occur in one frame, location-identifiable information may be acquired for each of the N transmission and reception operations that belong to one frame, or location-identifiable information may be acquired for only one of the N transmission and reception operations (e.g., the first transmission and reception operation).

The predetermined period is, for example, once every 20 ms. In this embodiment, one frame is 200 ms, and the frequency is 10 times per frame (N=10). However, the predetermined period may also be, for example, 20 times per frame (N=20), or 5 times per frame (N=5).

(1-1-5) IF Signal, FFT Result, and FFT Result Group Each time the radio wave sensor 1 performs a transmission/reception operation, it generates an IF signal for each of three or more receiving antennas, performs an FFT (Fast Fourier Transform) on the IF signal to obtain an FFT result, and outputs an FFT result group. In other words, each time a transmission/reception operation is performed, the radio wave sensor 1 outputs an FFT result group consisting of three or more FFT results corresponding to the three or more receiving antennas. However, a Fourier transform other than FFT may be performed on the IF signal, in which case the radio wave sensor 1 outputs a Fourier transform result group consisting of three or more Fourier transform results corresponding to the three or more receiving antennas.

As shown in Figure 3, the IF signal is a signal that indicates the frequency difference Δf between the transmitted wave Tr and the reflected wave Re. The IF signal is a signal that indicates the difference between the frequency of the transmitted wave Tr and the frequency of the reflected wave Re at time t, and is a function of time t, Δf(t), but indicates a constant value when the group of objects (301, 302) is stationary.

The IF signal is generated by mixing the transmitted wave Tr and the reflected wave Re. The IF signal is generated over the period during which the transmitted wave Tr is being transmitted and the reflected wave Re is being received (i.e., the period from the start of reception of the reflected wave Re to the end of transmission of the transmitted wave Tr).

The FFT result is the result of applying FFT to the IF signal. The FFT result is information that indicates the frequency spectrum (the relationship between frequency f and reflection intensity amp), as shown in Figures 8A and 8B, for example.

An FFT result group is information composed of three or more FFT results corresponding to three or more receiving antennas, obtained for one transmission/reception operation. Such an FFT result group makes it possible to identify the three-dimensional position of an object. Furthermore, by taking the difference between multiple FFT result groups, frequency components corresponding to stationary objects 302 are removed, and only frequency components corresponding to moving objects such as human bodies 301 (frequency f where amp exceeds a threshold, and the amp value corresponding to that frequency f) are obtained (see Figures 8A to 8C). Based on the frequency components obtained in this way, it is possible to obtain information regarding the three-dimensional position and movement of moving objects such as human bodies 301.

More specifically, in this embodiment, the radio wave sensor 1 outputs multiple FFT result sets corresponding to a series of multiple transmission and reception operations, and the output multiple FFT result sets are stored in chronological order in the memory of the control device 2. Meanwhile, the memory also stores the antenna position information set described above. The control device 2 then calculates the time difference (the time difference between the FFT results for each of the three receiving antennas) between the multiple FFT result sets (e.g., two adjacent FFT result sets) stored in chronological order in the memory, and acquires a ranging result set (two ranging results corresponding to the three receiving antennas) based on the calculated difference. The control device 2 performs three-point positioning using the ranging result set thus acquired and the antenna position information set previously stored. This makes it possible to acquire information specifying the three-dimensional position of the human body 301, for example, to calculate three-dimensional coordinates.

(1-2) Attitude Estimation Function of Device Control System As shown in FIG. 1, the device control system 100 includes a distance measurement unit 221, a point placement unit 222, and an attitude estimation unit 223.

(1-2-1) Distance Measuring Unit: One-to-One or One-to-Many Difference The distance measuring unit 221 measures the distance from the radio wave sensor 1 to the human body 301 based on the FFT results output by the radio wave sensor 1 .

In more detail, the ranging unit 221 retains the FFT result group output by the radio wave sensor 1 for a period equal to or greater than the predetermined cycle (preferably, a period equal to or greater than twice the predetermined cycle). The ranging unit 221 then performs ranging processing based on the difference between one FFT result group, a reference FFT result group (in this embodiment, a current FFT result group), and at least one target FFT result group (in this embodiment, at least one preceding FFT result group), among the multiple FFT result groups it retains, and acquires at least one ranging result group.

(1-2-1a) Reference FFT result group and target FFT result group The reference FFT result group is one FFT result group that serves as a starting point for obtaining a difference among the multiple FFT result groups held by the distance measuring unit 221. The target FFT result group is each of one or more FFT result groups that are the targets for obtaining a difference between the reference FFT result group and the multiple FFT result groups held by the distance measuring unit 221. The one or more target FFT result groups are located temporally before or after (usually before) the reference FFT group.

In this embodiment, one FFT result group is a current FFT result group (described below), and one or more target FFT result groups are one or more previous FFT result groups (described below).

By taking the difference between the reference FFT result group and at least one target FFT result group, the reflected components of the reflected waves Re from stationary objects 302 are removed (the reflection intensity, which is the received intensity of the reflected waves Re, is below the threshold), and only the reflected components from the human body 301 (moving object) remain (at least one pair, preferably two or more pairs, of reflection intensities amp exceeding the threshold and frequencies f corresponding to said reflection intensities are detected).

Note that the number of "at least one" preceding FFT result group, i.e., the number of preceding FFT result groups from which the difference with the current FFT result group is obtained, is preferably two or more (one-to-many inter-frame difference: one-to-five inter-frame difference in the illustrated example) as shown in Figures 9 and 11 in terms of improving resolution, but it can also be just one. In other words, even when one-to-one differences (one-to-one inter-frame differences) are taken as shown in Figures 7 and 8A-8C, attitude estimation is possible, and increasing the number of receiving antennas (hereinafter "number of antennas") can also improve resolution.

(1-2-1b) Current FFT result group and preceding FFT result group The current FFT result group is the most recent FFT result group among the multiple FFT result groups that are stored. The preceding FFT result group is the FFT result group that precedes the current FFT result group among the multiple FFT result groups that are stored. By setting the reference FFT result group as the current FFT result group and the target FFT result group as the preceding FFT result group, posture estimation can be performed in real time.

(1-2-1c) Distance Measurement Processing Distance measurement processing is processing for measuring (ranging) the distance to an object such as the human body 301 using the radio wave sensor 1. In the distance measurement processing of this embodiment, the distance from each of three or more receiving antennas (hereinafter referred to as each antenna) constituting the radio wave sensor 1 to the human body 301 is measured, and three or more distance measurement results (ranging result groups: described later) corresponding to the three or more receiving antennas are obtained.

In the ranging process, for example, based on various parameters shown in Figure 3, namely, the difference (frequency difference Δf) between the transmitted wave Tr and the reflected wave Re, the slope S of the linear change in the frequency f of the transmitted wave Tr, and the speed of light c, the distance d from each antenna to the human body 301 is calculated using the following equation 1.
d=(c/2S)×Δf...(Formula 1)

(1-2-1d) Distance Measurement Result Group The distance measurement result group is information consisting of three or more distance measurement results corresponding to three or more receiving antennas, obtained by the above formula 1 for one transmission/reception operation.

Three or more ranging results corresponding to three or more receiving antennas are, for example, three ranging results, numbered first through third, corresponding to three receiving antennas, numbered first through third. The first ranging result corresponds to approximately half the propagation distance from the radio wave radiated from the transmitting antenna to the first receiving antenna after being reflected by the human body 301 (i.e., the distance from the human body 301 to the first receiving antenna). The second ranging result corresponds to approximately half the propagation distance from the radio wave radiated from the transmitting antenna to the second receiving antenna after being reflected by the human body 301 (i.e., the distance from the human body 301 to the second receiving antenna). The third ranging result corresponds to approximately half the propagation distance from the radio wave radiated from the transmitting antenna to the third receiving antenna after being reflected by the human body 301 (i.e., the distance from the human body 301 to the third receiving antenna).

Note that "at least one" distance measurement result group, i.e., the number of distance measurement result groups obtained, is preferably two or more, but may be one.

(1-2-2) Point Arrangement Unit: Multi-Point Arrangement The point arrangement unit 222 performs coordinate calculation processing each time the distance measurement unit 221 performs distance measurement processing, and arranges at least one point corresponding to the current FFT result group in three-dimensional space 500, for example, as shown in Fig. 12A. However, the point may be arranged in two-dimensional space (plane) or one-dimensional space (line).

Note that "at least one" point, i.e., the number of points placed at one time corresponding to the current FFT result set, is preferably two or more (multi-point placement), but may also be one (single-point placement).

(1-2-2a) Three-dimensional space The three-dimensional space 500 is a virtual space corresponding to the real space (e.g., room 400: see FIG. 2) in which the radio wave sensor 1 and the group of objects (301, 302) exist. Arrangement in the three-dimensional space 500 may be a virtual operation or an operation of simply storing three-dimensional coordinates.

(1-2-2b) Coordinate Calculation Processing and Point Cloud The coordinate calculation processing is processing for calculating the three-dimensional coordinates of the human body 301 based on at least one acquired group of distance measurement results.

(1-2-3) Posture Estimation Unit The posture estimation unit 223 estimates the posture of the human body 301 (in this embodiment, whether the posture is standing, sitting, or lying down) based on, for example, a point cloud 501 as shown in Figures 12A to 12C. The point cloud 501 is a collection of one or more points arranged in a three-dimensional space 500 by the point arrangement unit 222.

In this way, according to this embodiment, the posture of the human body 301 can be estimated using the radio wave sensor 1 based on the FMCW method.

(2) Details of the Distance Measuring Unit and Point Arrangement Unit (2-1) One-to-Many Difference The distance measuring unit 221 preferably holds the FFT results output by the radio wave sensor 1 for a period of at least twice the predetermined period. The period of at least twice the predetermined period may be, for example, a period of at least twice one frame (2×T).

The distance measurement unit 221 then calculates the difference between the current FFT result group and each of the two or more previous FFT result groups among the three or more FFT result groups currently held, and performs distance measurement processing based on each of the two or more calculated differences to obtain two or more distance measurement result groups.

The point placement unit 222 performs coordinate calculation processing based on each of the two or more groups of distance measurement results acquired by the distance measurement unit 221, thereby placing two or more points corresponding to the current group of FFT results in three-dimensional space 500.

In this way, by calculating the difference between the current FFT result group and each of two or more previous FFT result groups, it is possible to improve the accuracy of estimating the posture of the human body 301 without increasing the number of antennas in the radio wave sensor 1.

(2-2) One-to-Many Difference Suitable for Detecting Various Movements of the Human Body The ranging unit 221 more preferably holds the FFT result groups output by the radio wave sensor 1 for a period of at least three times the predetermined cycle. The ranging unit 221 then calculates the difference between the current FFT result group and each of three or more previous FFT result groups among the four or more FFT result groups currently held, and selects two or more differences from the three or more calculated differences that correspond to various movements of the human body 301 (e.g., body movement, slight breathing movement, hand and foot movement, etc.). The ranging unit 221 performs the ranging process based on each of the two or more differences selected in this manner, thereby acquiring two or more ranging result groups.

The point placement unit 222 performs the coordinate calculation process based on each of the two or more groups of distance measurement results acquired by the distance measurement unit 221, thereby placing two or more points corresponding to the current group of FFT results in the three-dimensional space 500.

In this way, by calculating the difference between the current FFT result and each of three or more previous FFT results and selecting two or more differences corresponding to various movements of the human body 301, it is possible to improve the accuracy of estimating the posture of the human body 301 without increasing the number of antennas and while reducing the number of differences used in the ranging process.

(2-3) Frames and Inter-Frame Differences The above-described transmission and reception operations are performed N times (N is an integer equal to or greater than 2) per frame, with a predetermined time being one frame. In this embodiment, the predetermined time is 200 ms, and N=10.

N sets of FFT results per frame are stored in the memory across multiple frames. Specifically, the radio wave sensor 1 performs, for example, 10 transmission and reception operations per frame, and the memory of the control device 2 stores 10 sets of FFT results per frame across multiple frames (e.g., 6 frames). The distance measurement unit 221 obtains inter-frame differences between the multiple frames stored in the memory.

Inter-frame differences are the differences between a set of FFT results belonging to one frame (e.g., the reference frame Fr0 shown in Figure 7) and a set of FFT results belonging to one or more other frames (e.g., the subsequent frame Fr1, etc. shown in Figure 7).

(2-3-1) One-to-One Interframe Difference The interframe difference is, for example, a one-to-one interframe difference. The one-to-one interframe difference is the difference between a group of FFT results belonging to one frame (e.g., reference frame Fr0) and a group of FFT results belonging to another frame (e.g., subsequent frame Fr1).

(2-3-2) Representative value of each frame when calculating the difference between frames Each of the two FFT result groups corresponding to the two frames from which the difference is to be obtained (e.g., the FFT result group belonging to the reference frame Fr0 and the FFT result group belonging to the subsequent frame Fr1) is a representative value among the N (e.g., 10) FFT result groups in the frame to which it belongs.

The representative value is, for example, the average value of N FFT result groups (specifically, information consisting of the average value of 10 FFT results corresponding to the first receiving antenna, the average value of 10 FFT results corresponding to the second receiving antenna, and the average value of 10 FFT results corresponding to the third receiving antenna, etc.). In this case, the inter-frame difference is the difference between the average value of the 10 FFT result group belonging to the reference frame Fr0 and the average value of the 10 FFT result group belonging to the subsequent frame Fr1.

Alternatively, the representative value may be one of the N FFT result groups determined according to a predetermined rule (e.g., the kth FFT result group: k is an integer between 1 and N). In this case, the inter-frame difference is the difference between the kth (e.g., the first) FFT result group of the 10 FFT result groups belonging to the reference frame Fr0 and the kth (e.g., the first) FFT result group of the 10 FFT result groups belonging to the subsequent frame Fr1.

Note that the above points regarding inter-frame differences apply not only to one-to-one inter-frame differences, but also to one-to-many inter-frame differences.

For example, to calculate one-to-many frame differences (described below), the ranging unit 221 holds the FFT results output by the radio wave sensor 1 for a period of (K+1) frames (K is an integer greater than or equal to 2). In this embodiment, K=5, and the FFT results are held (for example, stored in the memory of the control device 2) for a period of (5+1) frames, or 6 x 200 ms = 1200 ms.

(2-3-3) One-to-Many Frame Difference The distance measurement unit 221 calculates the one-to-many frame difference using, for example, a group of FFT results spanning a (K+1) frame period stored in memory. The one-to-many frame difference is the difference (first time difference T, second time difference 2×T, ...) between the reference frame Fr0 and each of multiple target frames (for example, subsequent frames Fr1, Fr2, ... as shown in FIG. 9 ).

Note that Figure 9 illustrates multiple subsequent frames Fr1, Fr2, etc. that follow the reference frame Fr0 as an example of multiple target frames, but in this embodiment, the multiple target frames are, for example, multiple preceding frames Fr-1, Fr-2, etc. that precede the reference frame Fr0, as shown in Figure 11.

The difference between the FFT result groups is, for example, as shown in Figure 7, the difference between the FFT result group corresponding to the first transmission wave Tr1 of the reference frame Fr0 and the FFT result group corresponding to the first transmission wave Tr1 of the target frame (the subsequent frame Fr1 in the example of Figure 7) that has a time difference T from the reference frame Fr0.

Alternatively, the difference between the FFT result groups may be, for example, the difference between the FFT result group corresponding to the second transmission wave Tr2 of the reference frame Fr0 and the FFT result group corresponding to the second transmission wave Tr2 of the target frame (subsequent frame Fr1), or the difference between the FFT result group corresponding to the Nth transmission wave TrN of the reference frame Fr0 and the FFT result group corresponding to the Nth transmission wave Tr2 of the target frame (subsequent frame Fr1).

Alternatively, the difference in the FFT result group may be the sum of the N differences described above. In this embodiment, the difference in the FFT result group is the sum of the N differences described above, that is, the sum of the differences for each of the N transmission waves Tr1 to TrN.

In this embodiment, the one-to-many frame differences are a set of inter-frame differences between the current frame FFT result group and each of K or more (e.g., 5) previous frame FFT result groups out of the (N x (K + 1)) or more (e.g., 10 x (5 + 1) = 60) FFT result groups currently held.

(2-3-4) Current Frame and Current Frame FFT Result Group The current frame Fr0 is the current frame, and the current frame is the frame Fr0 that includes the most recent FFT result group. The current frame FFT result group is a set of N FFT result groups that belong to the current frame Fr0.

(2-3-5) Previous Frame and Previous Frame FFT Result Group Previous frames (Fr-1, Fr-2, ... Fr-K) are frames that precede the current frame Fr0. The previous frame FFT result group is a set of N FFT result groups that belong to K previous frames (Fr-1, Fr-2, ... Fr-K), respectively.

(2-3-6) Parameter K
The larger the value of the parameter K, the easier it is to detect various movements of the human body 301 (for example, body movements, slight respiratory movements, and hand and foot movements).

(2-3-6a) Body Movement, Respiratory Fluctuations, and Limb Movement Body movement is the movement of the entire body (trunk). As shown in Figures 10 and 11, body movement is irregular, with a long movement time and a large amount of movement. Limb movement is the movement of the limbs. As shown in Figures 10 and 11, limb movement is irregular, with a short movement time and a large amount of movement. Respiratory Fluctuations is the movement of the trunk associated with breathing. Respiratory Fluctuations is periodic, with a small amount of movement and a somewhat short movement time.

(2-3-6b) Specific examples of parameter K In this embodiment, as shown in FIG. 11, K=5, and of the five consecutive preceding frames (first preceding frame Fr-1, second preceding frame Fr-2, ... fifth preceding frame Fr-5), three (second, third and fifth preceding frames Fr-2, Fr-3 and Fr-5) that are suitable for detecting body movement, limb movement and slight respiratory movement are selected, thereby improving the estimation accuracy while reducing the processing load for difference calculation, but all five preceding frames Fr-1 to Fr-5 may also be used.

(2-3-7) Posture Estimation Based on One-to-Many Frame Differences The distance measurement unit 221 acquires K or more distance measurement result groups corresponding to K or more inter-frame differences that constitute the one-to-many frame differences calculated for the current frame Fr0 in this way.

The point placement unit 222 performs coordinate calculation processing based on each of the K or more distance measurement result groups obtained by the distance measurement unit 221 for the current frame Fr0, thereby placing K or more points corresponding to the current FFT result group in three-dimensional space 500.

In this way, by calculating the one-to-many frame difference between the current FFT result group and each of two or more previous FFT result groups, it is possible to further improve the accuracy of estimating the posture of the human body 301 without increasing the number of antennas.

(2-3-8) Preferred Value of Parameter K The value of K is preferably an integer equal to or greater than 3. The distance measurement unit 221 selects two or more inter-frame differences corresponding to various movements of the human body 301 (for example, body movement, slight respiratory movement, hand and foot movement, etc.) from among the three or more inter-frame differences that constitute the one-to-many inter-frame difference calculated for the current frame Fr0, and obtains two or more groups of distance measurement results corresponding to the two or more selected inter-frame differences.

The point placement unit 222 performs coordinate calculation processing based on each of the two or more groups of distance measurement results obtained by the distance measurement unit 221 for the current frame Fr0, thereby placing two or more points corresponding to the current group of FFT results in three-dimensional space 500.

In this way, by calculating three or more inter-frame differences (one-to-many inter-frame differences) for each of three or more preceding FFT result groups of the current FFT result group, and then calculating two or more of the three or more inter-frame differences corresponding to various movements of the human body 301, it is possible to further improve the accuracy of estimating the posture of the human body 301 without increasing the number of antennas and while suppressing the number of inter-frame differences used in the ranging process.

A more preferable value for K is 5. The distance measurement unit 221 selects three inter-frame differences corresponding to the body movement, respiratory micromovement, and limb movement of the human body 301 from the five inter-frame differences that make up the one-to-many inter-frame difference calculated for the current frame Fr0, and obtains three groups of distance measurement results corresponding to the selected three inter-frame differences.

The three inter-frame differences corresponding to the body movement, respiratory micromovement, and limb movement of the human body 301 are, for example, the second, third, and fifth inter-frame differences ΔA2, ΔA3, and ΔA5 of the first to fifth inter-frame differences ΔA1 to ΔA5, as shown in Figure 11.

The point placement unit 222 performs coordinate calculation processing based on each of the three groups of distance measurement results obtained by the distance measurement unit 221 for the current frame, thereby placing three points corresponding to the current group of FFT results in three-dimensional space 500.

In this way, by calculating five inter-frame differences (one-to-many inter-frame differences) for each of the five preceding FFT result groups of the current FFT result group, and then calculating three of the five inter-frame differences corresponding to the body movement, respiratory micromovement, and limb movement of the human body 301, it is possible to further improve the accuracy of estimating the posture of the human body 301 without increasing the number of antennas and while suppressing the number of inter-frame differences used in the ranging process.

Note that the above is merely an example; for example, the number of inter-frame differences calculated and the time difference can be changed as appropriate to improve estimation accuracy.

(2-3-9) Details of the Posture Estimation Unit (2-3-9a) Clustering and Distribution of Cluster Groups The posture estimation unit 223, for example, performs clustering on the point cloud 501 to obtain a cluster group (CL, CL1, CL2) which is a collection of one or more clusters (CL, CL1, CL2).

The cluster group (CL, CL1, CL2) may be, for example, a first cluster CL1 corresponding to the upper body (head, torso, and arms) of the human body 301, and a second cluster CL2 corresponding to the lower body (legs), as shown in Figures 12A and 12B. Alternatively, the cluster group (CL, CL1, CL2) may be, for example, a single cluster CL corresponding to the entire body (head, torso, arms, and legs), as shown in Figure 12C.

In addition, as the number of points constituting the point cloud 501 increases, it is expected that the cluster group (CL, CL1, CL2) will have a resolution sufficient to distinguish the outline (silhouette) of the human body 301 and, ultimately, each part constituting the human body 301.

The posture estimation unit 223 estimates the posture of the human body 301 based on the distribution (hereinafter simply referred to as "distribution") of the thus acquired cluster group (CL, CL1, CL2) in three-dimensional space 500.

The distribution may be, for example, the number of clusters (CL, CL1, CL2) obtained, the direction of expansion of one cluster (CL, CL1, CL2), or the distance between multiple clusters.

(2-3-9b) Posture Estimation Based on Distribution Conditions The posture estimation unit 223 estimates whether the posture of the human body 301 is lying down, sitting, or standing, based on distribution conditions, which are conditions related to distribution, for example.

The distribution conditions include, for example, a lying-down condition. A lying-down condition (first lying-down condition) that constitutes the distribution conditions is, for example, "only a single cluster CL is acquired, and the height of that single cluster CL from the floor surface is low enough to be below a threshold value" (see Figure 12C).

The distribution condition further includes, for example, a locus condition. A locus condition (first locus condition) that constitutes the distribution condition is, for example, "two clusters, first and second, CL1 and CL2, are obtained, and the distance between the two clusters CL1 and CL2 is so close that it is below a threshold value" (see Figure 12B).

The posture estimation unit 223 may, for example, determine that the posture of the human body 301 is lying down if the distribution satisfies the above-mentioned lying down condition (first lying down condition), determine that the posture is sitting down if the distribution satisfies the above-mentioned sitting down condition (first sitting down condition), and determine that the posture is standing up if the distribution satisfies neither the above-mentioned lying down condition nor the above-mentioned sitting down condition.

In this way, the posture of the human body 301 can be estimated with high accuracy based on the distribution of the cluster group (CL, CL1, CL2).

(2-3-9c) Posture estimation based on size ratio conditions Alternatively, the posture estimation unit 223 may estimate the posture of the human body 301 based on size ratio conditions regarding the ratios (hereinafter referred to as "size ratios") of the length D, width W, and height H of the three-dimensional figure 502 surrounding the cluster group (CL, CL1, CL2).

In this embodiment, the three-dimensional figure 502 is, for example, a rectangular parallelepiped with length D, width W, and height H, as shown in Figures 13A to 13C, and the dimensional ratio is, for example, the ratio between length D or width W and height H.

However, the three-dimensional figure 502 may also be, for example, an elliptical cylinder (not shown). In an elliptical cylinder, the minor axis corresponds to the vertical axis D and the major axis corresponds to the horizontal axis W.

The dimension ratio condition includes a lying-down condition. A lying-down condition (second lying-down condition) that constitutes the dimension ratio condition may be, for example, "the length D or width W of the three-dimensional figure 502 must be so large as to exceed a predetermined ratio (e.g., three times) of the height H."

The size ratio condition further includes, for example, an upright position condition. The upright position condition constituting the size ratio condition may be, for example, "the height H of the three-dimensional figure 502 must be greater than a predetermined ratio (e.g., three times) of the length D or width W."

The posture estimation unit 223 may, for example, determine that the posture of the human body 301 is lying down if the dimension ratio satisfies the above-mentioned lying down condition (second lying down condition), determine that the posture is standing if the dimension ratio satisfies the above-mentioned standing down condition, and determine that the posture is sitting if the dimension ratio satisfies neither the lying down condition nor the above-mentioned standing down condition.

In this way, the posture of the human body 301 can be accurately estimated based on the ratios of the length D, width W, and height H of the three-dimensional figure 502 that surrounds the cluster group (CL, CL1, CL2).

(2-3-9d) Posture estimation based on distribution size ratio conditions The posture estimation unit 223 may estimate whether the posture of the human body 301 is lying down, sitting, or standing, based on distribution size ratio conditions, which are conditions related to distribution and size ratios, for example.

The distribution size ratio condition includes a lying-down condition. The lying-down condition (third lying-down condition) that constitutes the distribution size ratio condition may be, for example, "only one cluster CL is acquired, and the length D or width W of the three-dimensional figure 502 is so large as to exceed a predetermined ratio (e.g., three times) of the height H."

The distribution size ratio condition further includes, for example, a locus condition. The locus condition (second locus condition) constituting the distribution size ratio condition may be, for example, "two clusters, first and second, CL1 and CL2, are obtained, and the distance between the two clusters CL1 and CL2 is so close that it is below a threshold value."

The posture estimation unit 223 may, for example, determine that the posture of the human body 301 is lying down if the distribution and size ratio satisfy the above-mentioned lying down condition (third lying down condition), determine that the posture is sitting down if the distribution and size ratio satisfy the above-mentioned sitting down condition (second sitting down condition), and determine that the posture is standing up if the distribution and size ratio satisfy neither the above-mentioned lying down condition nor the above-mentioned sitting down condition.

(2-4) Device Control Function of Device Control System As shown in FIG. 1, the device control system 100 further includes a behavior detection unit 224 and a device control unit 225.

(2-4-1) Behavior Detection Unit The behavior detection unit 224 detects the behavior of the person corresponding to the human body 301 based on the estimation result of the posture estimation unit 223 for the human body 301 .

In this embodiment, the behavior detection unit 224 detects non-operation behavior based at least in particular on changes in the estimation results of the posture estimation unit 223.

In this embodiment, the change in the estimation result is a change between a standing position, a sitting position, and a lying position. Specifically, for example, this includes a change from a standing position to a sitting position, a change from a sitting position to a standing position, a change from a standing or sitting position to a lying position, a change from a lying position to a standing or sitting position, etc.

The behavior detection unit 224 may also detect a non-operation behavior when an unchanged state in which no change in the estimation result of the posture estimation unit 223 is detected continues for a predetermined time or more. The behavior detection unit 224 detects a non-operation behavior, for example, when an unchanged state continues for a predetermined time or more after a change in the estimation result of the posture estimation unit 223 is detected.

Specifically, the behavior detection unit 224 detects a non-operation behavior of continuing a lying state (for example, falling asleep) when, for example, after the estimation result of the posture estimation unit 223 has changed from a standing or sitting position to a lying position, a no-change state in which no change from the lying position to another position is detected has continued for a predetermined time or more. Furthermore, the behavior detection unit 224 detects a non-operation behavior of continuing a lying state (for example, falling asleep) when, for example, after the estimation result of the posture estimation unit 223 has changed from a standing or sitting position to a lying position, a no-change state in which no change from the sitting position to another position is detected has continued for a predetermined time or more.

(2-4-1a) Non-operational behavior and operational behavior Non-operational behavior is a behavior that is different from operational behavior.

Operational behavior is an action intended to operate device 200. Device 200 is, for example, a lighting fixture 200a attached to the ceiling of room 400, a television (TV) 200b placed on the floor, and electric blinds 200c attached to window 402, as shown in FIG. 2. Operational behavior is, for example, an on/off operation of lighting fixture 200a via a wall switch, an on/off operation or channel change operation of TV 200b via a remote control, or an open/close operation of electric blinds 200c via a remote control.

Non-operational behavior is, for example, a daily behavior that does not intend to operate the device 200. Daily behavior is, for example, the behavior of entering the room 400 (entering through the entrance/exit 401 of the room 400), the behavior of lying down on the bed 302b (going to sleep), the behavior of getting up from the bed 302b (getting up), the behavior of leaving the room 400 (leaving through the entrance/exit 401), etc. Daily behavior may also be, for example, the behavior of sitting at the desk 302a (sitting) (desk work or relaxation), the behavior of leaving the desk 302a (leaving one's seat), etc.

Seating may be distinguished, for example, into sitting to work at desk 302a (desk work) and sitting to relax and watch TV 200b (relaxation). Whether it is desk work or relaxation may be determined, for example, based on the time of day (daytime or nighttime) when sitting is detected.

The behavior detection unit 224 may also detect non-operation behavior by taking into consideration environmental information, history information, etc. (described below). For example, waking up may be distinguished between waking up in the morning and waking up in the middle of the night. Whether it is waking up in the morning or waking up in the middle of the night may be determined based on the time of day when a change from a lying down position to a sitting or standing position is detected.

(2-4-2) Device Control Unit The device control unit 225 controls the device 200 based at least on the non-operation behavior detected by the behavior detection unit 224 .

In addition to control in response to non-operational behavior (automatic control), the device control unit 225 also performs control in response to operations on the device 200 (manual control). The operations may be, for example, operations on an operation device (touch panel, operation button, etc.) possessed by the control device 2, but may also be operations on an operation device (wall switch, remote control, etc.) possessed by the device 200, or gestures for operating the device 200. The operations on the operation device possessed by the device 200 are detected, for example, by intercepting control signals sent to the device 200 from an operation device such as a wall switch. Gestures are detected, for example, by analyzing images of the human body 301 captured by a camera (not shown), but may also be detected by the radio wave sensor 1, distinguishing them from non-operational behavior.

In this way, by detecting the behavior of the human body 301 (particularly non-operational behavior) based on the estimated posture changes and performing device control according to the detection results, it is possible to control the device 200 without operating it (operation-less device control).

(2-4-3) Information Acquisition Unit As shown in Fig. 1, the device control system 100 further includes an information acquisition unit 226. The information acquisition unit 226 acquires various types of information, such as environmental information and a person identifier (both of which will be described later).

The information acquisition unit 226 acquires, for example, environmental information. Environmental information is information about the environment in which the human body 301 is located. The environment may be, for example, the time of day (daytime/nighttime, early morning/late night, etc.), weather (sunny/rainy, etc.), temperature, humidity, etc., but may also include the type of room 400 in which the human body 301 is located (bedroom/living room) or direction (south-facing/north-facing), etc.

Note that if multiple people (two or more human bodies 301) may be present in the same environment, the information acquisition unit 226 may acquire a person identifier. In more detail, for example, a collection of group information (group of group information) that pairs a person identifier that identifies a person with feature information that indicates the characteristics of that person is stored in memory in advance. The person identifier is, for example, an ID such as "1" or "2", but it can be any information that can identify a person, such as a name or email address. The feature information paired with the person identifier is, for example, height information that indicates the height of the person. However, the feature information may also be speed information that indicates the walking speed of the person, or any information that indicates a feature that can distinguish the person from others. The feature information itself may be used as the person identifier.

When the posture estimation unit 223 detects a human body 301, the information acquisition unit 226 may acquire feature information of the human body 301 from the posture estimation unit 223, and read from memory a person identifier paired with the acquired feature information.

Note that the automatic control information (see Figure 14) described below may be customized for each of multiple person identifiers.

The behavior detection unit 224 detects non-operation behavior based further on the environmental information acquired by the information acquisition unit 226.

In this way, by using environmental information in addition to posture changes, the accuracy of detecting non-operational behavior can be improved.

(2-4-3a) Automatic Control and Automatic Control Information The device control unit 225 controls the device 200 based on the non-operation behavior, for example, by using one or more (six in this case) pieces of automatic control information (automatic control information group) as shown in FIG. 14. The automatic control information group is stored in advance in the memory of the control device 2, for example. In this embodiment, control based on the automatic control information group is referred to as "automatic control."

As shown in FIG. 14, each of the one or more pieces of automatic control information that make up the automatic control information group includes information regarding status changes, posture changes, environment, non-operational actions, control content, and control ID. Specifically, of the six pieces of automatic control information shown in FIG. 14, the first piece of automatic control information includes a status change of "absent to present," a posture change of "-," an environment of "-," a non-operational action of "entering the room," control content of "lights normally on (brightness 5)," and a control ID of "1." Note that "-" indicates that the relevant information is not included (the same applies below).

Similarly, the second automatic control information includes a status change of "-", a posture change of "standing to sitting", an environment of "daytime", a non-operational action of "desk work", a control content of "brighten the lights (+2)", and a control ID of "2". The third automatic control information includes a status change of "-", a posture change of "standing to sitting", an environment of "nighttime", a non-operational action of "relaxing", a control content of "TV on", and a control ID of "3". The fourth automatic control information includes a status change of "-", a posture change of "standing or sitting to lying down", an environment of "nighttime", a non-operational action of "sleeping", a control content of "lights off", and a control ID of "4". The fifth automatic control information includes a status change of "-", a posture change of "lying down to standing or sitting", an environment of "late night", a non-operational action of "waking up mid-evening", a control content of "lights dimmed (brightness 2)", and a control ID of "5". The sixth automatic control information includes a status change of "-", a posture change of "lying down → standing or sitting", an environment of "morning", a non-operational action of "getting up", a control content of "open blinds", and a control ID of "6".

(2-4-3b) Manual Control The device control unit 225 can also perform control (automatic control) using the automatic control information in response to an operation (hereinafter simply referred to as an “operation”) on the device 200. Control based on an operation is referred to as “manual control” in this embodiment.

(2-4-4) History Storage and Learning The device control system 100 further includes a history storage unit 227 and a learning unit 228.

(2-4-4a) History Accumulation Unit The history accumulation unit 227 accumulates control history information. The control history information is information relating to the control history of the device 200 performed by the device control unit 225. The control history information is, for example, automatic control history information, but may also be manual control history information. Alternatively, the control history information may include automatic control history information and manual control history information.

Automatic control history information is information about the control history of the device 200 in response to behavior (for example, non-operational behavior, but it can also be operational behavior). The automatic control history information associates the control content based on the behavior with the time the control was performed, environmental information, and a person identifier.

Manual control history information is information related to the control history of the device 200 in response to an operation. The manual control history information associates the control content of the device 200 based on an operation with one or more pieces of information including the time the operation was performed, environmental information, and a person identifier.

When the device control unit 225 controls the device 200, the history storage unit 227 stores control history information, such as that shown in FIG. 15, in the memory of the control device 2. Note that the control history information may also be stored in an external memory.

For example, if the device control unit 225 has already performed three control operations on lighting fixture 200a at time 9:02, three pieces of control history information, as shown in FIG. 15, are stored in the memory of the control device 2. Each of the three pieces of control history information includes information related to the history ID, time, control ID, and operation.

Of the three pieces of control history information shown in Figure 15, the first piece of control history information includes history ID "1", time "9:00", control ID "1", and operation "-". The second piece of control history information includes history ID "2", time "9:01", control ID "2", and operation "-". The third piece of control history information includes history ID "3", time "9:02", control ID "-", and operation "operation to dim the lights slightly (-1)".

The aforementioned automatic control history information has operation information marked with "-", which corresponds to the first and second pieces of control history information among the three pieces of control history information mentioned above. The aforementioned manual control history information has a control ID marked with "-", which corresponds to the third piece of control history information among the three pieces of control history information mentioned above.

(2-4-5) Learning Unit The learning unit 228 updates the automatic control information when the time and control content indicated by the automatic control history information and the time and operation indicated by the manual control history information satisfy predetermined update conditions.

(2-4-5a) Update conditions and updating of information for automatic control based thereon An update condition is, for example, "after automatic control based on non-operational behavior is performed on a device, an operation to cancel the control content of the automatic control is accepted within a predetermined time (for example, 2 minutes)."

In the example of Figure 15, the time "9:01" and control content (i.e., the control content corresponding to control ID "2" in Figure 14) "brighten the lights (+2)" indicated by the automatic control information, which is the second control history information, and the time "9:03" and operation "slightly dim the lights (-1)" indicated by the manual control information, which is the third control history information, satisfy the above update condition.

In other words, after the automatic control "brighten the lights (+2)" based on the non-operational behavior "desk work" was performed on one device "lighting fixture 200a," an operation to cancel the automatic control "dimming the lights slightly (-1)" was received within two minutes, and therefore the learning unit 228 determines that the update condition has been met. Then, of the six pieces of automatic update information in FIG. 14, the learning unit 228 updates the second piece of automatic control information corresponding to control ID "2" to, for example, "dimming the lights slightly (+1)."

Alternatively, the update condition may be, for example, "an operation different from the automatic control content has been received a specified number of times or more within a specified period (e.g., "more than twice a day," "more than three times a week," etc.)."

In this way, by accumulating history information regarding automatic control in response to non-operational behavior (automatic control history information) and history information regarding manual control in response to operation, and updating the automatic control information based on these two pieces of history information when the update conditions are met, it is possible to achieve a learning function.

(3) Specific Example of Device Control System As shown in Fig. 1, the device control system 100 of this example includes a radio wave sensor 1 and a control device 2. The radio wave sensor 1 and the control device 2 each have a communication module and are connected to each other so that they can communicate with each other via wired or wireless communication. The control device 2 also has a processor and memory. Programs and various information are stored in the memory, and the processor operates based on the programs and the like in the memory (and cooperates with the communication module), thereby realizing the operation of the control device 2 as described below. Note that the radio wave sensor 1 may also have a processor and memory, and transmission and reception operations, communication with the control device 2, and the like may be performed under the control of a program.

The radio wave sensor 1 is installed on the ceiling of the room 400 as shown in FIG. 2. The control device 2 is, for example, a multi-remote control, and is installed on a side wall of the room 400 as shown in FIG. 2.

The control device 2 estimates the posture (standing, sitting, or lying down) of the human body 301 present in the room 400 based on the output of the radio wave sensor 1. The control device 2 then detects non-operational behavior of the human body 301 based on changes in the estimated posture, and automatically controls multiple devices 200 (lighting fixtures 200a, TVs 200b, and electric blinds 200c) according to the detection results. The control device 2 also has an operation device such as a touch panel, and can manually control the multiple devices 200 according to operation.

As shown in FIG. 2, the control device 2 includes a reception unit 21, a processing unit 22, and an output unit 23. The processing unit 22 includes a distance measurement unit 221, a point placement unit 222, a posture estimation unit 223, a behavior detection unit 224, a device control unit 225, an information acquisition unit 226, a history accumulation unit 227, and a learning unit 228.

The reception unit 21 receives various types of information. Examples of such information include operation information for the device 200. The reception unit 21 may also receive information other than operation information, such as personal information such as the height of the human body 301.

The processing unit 22 performs various types of processing. These types of processing include, for example, processing by the distance measurement unit 221, processing by the point placement unit 222, processing by the posture estimation unit 223, processing by the behavior detection unit 224, processing by the device control unit 225, processing by the information acquisition unit 226, processing by the history accumulation unit 227, and processing by the learning unit 228. The processing unit 22 also performs various judgments, which will be explained in the flowcharts.

In more detail, the processing unit 22 may execute, for example, a holding process, an acquisition process, a position identification process, a human body estimation process, and a placement process. The processing unit 22 executes this series of processes, for example, every time position identification information is output from the radio wave sensor 1, but may also execute these processes every time the number of position identification information corresponding to one frame is output.

The retention process is a process of retaining the location-identifiable information output by the radio wave sensor 1. The acquisition process is a process of acquiring the difference between multiple pieces of retained location-identifiable information. The location determination process is a process of identifying the position of a moving object (301, 200c) among one or more objects (301, 302, 200c) constituting a group of objects (301, 302, 200c) based on the difference, and acquiring location information indicating the identified position. The human body estimation process is a process of estimating whether the acquired location information corresponds to a human body 301 or a moving object other than a human body 301 (e.g., electric blinds 200c) based at least on a change in the difference. Note that, in this embodiment, the change typically refers to a change in a value, such as a difference, over time, and may be referred to as a "temporal change." However, the change may also be, for example, a change in a value according to a position in space (spatial change).

The placement process is a process in which, when it is estimated that the acquired position information corresponds to the human body 301, a point corresponding to the acquired position information is virtually placed in space (three-dimensional space 500). The placement process is performed by the point placement unit 222 that constitutes the processing unit 22.

In this embodiment in particular, the storage process stores three or more pieces of location-identifiable information. The acquisition process acquires two or more differences with different time differences for the three or more pieces of location-identifiable information stored. The location identification process acquires two or more pieces of location information corresponding to the two or more acquired differences. The human body estimation process estimates whether each of the two or more pieces of acquired location information corresponds to a human body 301 or a moving object (200c) other than a human body 301. The placement process virtually places in space (500) a point corresponding to the location information estimated to correspond to the human body 301 from the two or more pieces of acquired location information.

In addition, the placement process may receive the estimation results of the human body estimation process and display, via the output unit 23, points corresponding to objects estimated to be human bodies 301 and points corresponding to objects estimated to be moving objects (200c) other than human bodies 301 in a manner that allows them to be distinguished from one another (for example, by displaying them in different colors, different sizes, etc.).

In addition, the processing unit 22 in this embodiment further executes device control processing. The device control processing is processing for controlling the device 200 based at least on a point cloud 501, which is a collection of multiple points arranged in a space (500). The human body estimation processing estimates the posture or posture change of the human body 301 based on the distribution of the point cloud 501.

The processing unit 22 may further execute a behavior detection process. The behavior detection process is a process for detecting the behavior of a person corresponding to the human body 301 based on the estimated posture or posture change. The device control process, for example, controls the device 200 based on the detected behavior.

The processing unit 22 may further execute information acquisition processing. The information acquisition processing is processing for acquiring environmental information related to the environment in which the human body 301 exists. The human body estimation processing detects behavior based on the environmental information acquired by the information acquisition processing.

The device control process controls the device 200 based on behavior, for example, using pre-stored automatic control information.

The processing unit 22 may further execute a history accumulation process. The history accumulation process is a process of accumulating history information related to the control history of the device 200. The history information may be, for example, automatic control history information. Alternatively, the history information may be manual control history information. For example, the history accumulation process preferably accumulates both automatic control history information and manual control history information, but may also accumulate only one of them.

The processing unit 22 may further execute a learning process. The learning process is a process of updating the automatic control information when the time and operation indicated by the manual control history information satisfy predetermined update conditions. Alternatively, the learning process may be a process of updating the automatic control information when the time and control content indicated by the automatic control history information and the time and operation indicated by the manual control history information satisfy predetermined update conditions.

The radio wave sensor 1 may perform the conversion process described in "(10) Second Modified Example of Device Control System," and the processing unit 22 may perform the storage process, difference acquisition process, and moving object estimation process described in "(10) Second Modified Example of Device Control System." In other words, the radio wave sensor 1 may include the conversion unit 11 shown in FIG. 16, and the processing unit 22 may include the conversion unit 11, storage unit 12, difference acquisition unit 13, and moving object estimation unit 14 shown in FIG. 16.

The output unit 23 outputs various types of information. For example, the various types of information are control information (e.g., remote control signals) for the device 200. The output here is typically transmission to the device 200, but may also include display on a display.

(4) Example of Operation of Device Control System The control device 2 constituting the device control system 100 operates, for example, according to the flowcharts of FIGS.

(4-1) Overall Processing The processing in FIG. 4 is started when the device control system 100 is started, and is ended when the operation is stopped.

First, the processing unit 22 of the control device 2 determines whether an FFT result group has been output from the radio wave sensor 1 (step S1). If it is determined that an FFT result group has not been output, processing proceeds to step S13.

If it is determined in step S1 that the FFT result group has been output, the distance measurement unit 221 constituting the processing unit 22 stores the FFT result group (step S2).

Next, the distance measurement unit 221 determines whether a set of FFT results for at least (K+1) frame periods has been stored (step S3). If it is determined that a set of FFT results for at least (K+1) frame periods has not yet been stored, processing returns to step S1.

If it is determined in step S3 that a group of FFT results for at least (K+1) frame periods has been retained, the distance measurement unit 221 performs one-to-many frame difference acquisition processing (step S4). Note that the one-to-many frame difference acquisition processing will be explained using the flowchart in Figure 5.

Next, the distance measurement unit 221 acquires L (or K) distance measurement results (distance measurement result group) corresponding to the L (or K) differences (difference group) (step S5).

Next, the point placement unit 222 calculates L (or K) three-dimensional coordinates (a group of three-dimensional coordinates) based on the group of distance measurement results acquired in step S5 (step S6).

Next, the point placement unit 222 places L (or K) points (point group 501) in three-dimensional space 500 based on the three-dimensional coordinate group acquired in step S6 (step S7).

It is determined whether the judgment start condition is met (step S8). The judgment start condition is, for example, a condition such that a predetermined number or more points are arranged in the three-dimensional space 500. If it is determined that the judgment start condition has not yet been met, processing returns to step S1.

If it is determined in step S8 that the determination start condition is met, the posture estimation unit 223 performs posture estimation processing (step S9). The posture estimation processing will be explained using the flowchart in Figure 6.

Next, the behavior detection unit 224 uses the automatic control information (see FIG. 14) to detect non-operation behavior based on changes in the estimation results from step S9 (e.g., changes in status, posture, environment) (step S10). For example, in response to a status change from "absent to present," the non-operation behavior "entering" is detected. In addition, in response to a posture change from "standing to sitting," it is determined whether the environment (current time zone) is daytime or nighttime, and in the case of daytime, the non-operation behavior "desk work" is detected, and in the case of nighttime, the non-operation behavior "relaxing" is detected.

Next, device control unit 225 performs device control (automatic control) in response to the detection results of the non-operation behavior, etc., in step S10 (step S11). For example, in response to the detection of the non-operation behavior "entering a room," automatic control (control with control ID "1") is performed to turn on lighting device 200a at normal brightness (brightness 5). In addition, in response to the detection of the non-operation behavior "desk work," automatic control (control with control ID "2") is performed to brighten lighting device 200a (brightness +2). In addition, in response to the detection of the non-operation behavior "relaxing," automatic control (control with control ID "3") is performed to turn on TV 200b.

Next, the history accumulation unit 227 accumulates automatic control history information related to the automatic control performed in step S11 (step S12). After that, processing returns to step S1.

If it is determined in step S1 that the FFT results have not yet been output, the processing unit 22 determines whether the reception unit 21 has received an operation on the device 200 (step S13). If it is determined that the reception unit 21 has not yet received an operation on the device 200, the processing returns to step S1.

If it is determined in step S13 that the reception unit 21 has received an operation on the device 200, the device control unit 225 performs device control (manual control) in accordance with the operation (step S14). For example, if an operation to slightly dim the lighting device 200a (brightness -1) is performed, manual control to slightly dim the lighting device 200a is performed.

Next, the history accumulation unit 227 accumulates manual control history information related to the manual control performed in step S14 (step S15).

Next, the learning unit 228 determines whether the update conditions have been met (step S16). If it is determined that the update conditions have not yet been met, processing returns to step S1.

If it is determined that the update conditions are met, the history accumulation unit 227 updates the automatic control information (step S17). For example, as shown in FIG. 15, one minute after the automatic control of control ID "2", manual control is performed to slightly dim lighting fixture 200a, and it is determined that the update conditions are met, and the control content of the automatic control of control ID "2" is updated to "brighten the lights (+1)." Processing then returns to step S1.

In the processing of FIG. 4, the radio wave sensor 1 may output three or more IF signals (output signals) corresponding to three or more receiving antennas instead of a group of FFT results (location-identifying information), and the processing unit 22 may obtain an FFT transformation result based on the three or more output IF signals. In this case, in step S1 above, the processing unit 22 may perform an FFT transformation on each of the three or more IF signals and determine whether or not an FFT transformation result has been obtained.

(4-2) One-to-Many Frame Difference Acquisition Processing The one-to-many frame difference acquisition processing in step S4 is executed, for example, according to the flowchart of FIG.

The processing unit 22 sets the initial value "1" to the variable i (step S41).

Next, the processing unit 22 determines whether the variable i is greater than K (step S42). If it is determined that the variable i is greater than K, the processing proceeds to step S45.

If it is determined in step S42 that the variable i is not greater than K (i.e., is equal to or less than K), the distance measurement unit 221 obtains the difference between the current FFT result group and the i-th preceding FFT result group, which is the i-th time difference (= i x T seconds) before the current FFT result group, and sets the obtained difference to the variable "i-th difference" (step S43).

Next, the processing unit 22 increments the variable i (step S44). After that, the processing returns to step S42.

If it is determined in step S42 that the variable i is greater than K, the distance measurement unit 221 selects L differences (e.g., three differences ΔA1, ΔA2, and ΔA3: see Figure 11) corresponding to various movements of the human body 301 from the first to Kth K differences (e.g., five differences ΔA1 to ΔA5: only some are shown in Figure 11) (step S45). Then, the process returns to the upper flowchart (see Figure 4).

(4-3) Attitude Estimation Processing The attitude estimation processing in step S9 is executed, for example, according to the flowchart of FIG.

The posture estimation unit 223 performs clustering on the point cloud 501 arranged in the three-dimensional space 500 to obtain a cluster group (CL, CL1, CL2: see Figures 12A to 12C) (step S91).

Next, the posture estimation unit 223 places a three-dimensional figure (rectangular parallelepiped) 502 (see Figures 13A to 13C) surrounding the cluster group (CL, CL1, CL2) acquired in step S91 in three-dimensional space 500 (step S92).

Next, the posture estimation unit 223 obtains the ratios (dimension ratios) of the length D, width W, and height K of the three-dimensional figure 502 placed in step S92 (step S93).

Next, the posture estimation unit 223 determines whether the distribution (cluster distribution) of the cluster group (CL, CL1, CL2) obtained in step S91 and the dimensional ratio obtained in step S93 satisfy the supine position condition (step S94).

If it is determined that the cluster distribution and size ratio satisfy the lying-down condition, the posture estimation unit 223 sets the variable "posture" to the value "lying-down" (step S95). The process then returns to the upper-level flowchart (see Figure 4).

If it is determined that the cluster distribution and size ratio do not satisfy the lying position condition, the posture estimation unit 223 further determines whether the cluster distribution and size ratio satisfy the sitting position condition (step S96).

If it is determined that the cluster distribution and size ratio satisfy the sitting position conditions, the posture estimation unit 223 sets the variable "posture" to the value "sitting position" (step S97). Then, the process returns to the upper-level flowchart (see Figure 4).

If it is determined that the cluster distribution and size ratio do not satisfy the sitting condition either, the posture estimation unit 223 sets the variable "posture" to the value "standing" (step S98). Then, the process returns to the upper-level flowchart (see Figure 4).

(5) Human Body Pose Estimation Method, Device Control Method, and Program The human body posture estimation function of the present disclosure may be realized by a human body posture estimation method or a program. The human body posture estimation method includes, among the various steps described above, at least steps S2 to S5 (ranging steps), steps S6 and S7 (placement steps), and step S9 (posture estimation step). The program is a program for causing one or more processors to execute the human body posture estimation method.

The device control function of the present disclosure may be realized by a device control method or a program. The device control method further includes step S10 (behavior detection step) and step S11 (device control step) in addition to the ranging step, placement step, and posture estimation step. The program is a program for causing one or more processors to execute the device control method.

(6) Modified Examples of Non-Operational Behavior In addition to going to bed and waking up, non-operational behavior may further include, for example, falling asleep after going to bed and waking up before waking up. Falling asleep and waking up can be detected, for example, based on changes (stopping and starting) in body movement. Stopping body movement may be, for example, a state in which no body movement exceeding a threshold is detected continuing for a predetermined period of time or more. Starting body movement may be, for example, a state in which body movement exceeding a threshold is detected continuing for a predetermined period of time or more.

(7) Modified Examples of Arrangement of Each Unit In this embodiment, the control device 2 includes the distance measurement unit 221, point placement unit 222, posture estimation unit 223, behavior detection unit 224, device control unit 225, information acquisition unit 226, history storage unit 227, and learning unit 228, but all of these elements may be included in the radio wave sensor 1. Alternatively, the radio wave sensor 1 may include only the distance measurement unit 221 out of these elements, and the other elements may be included in the control device 2. Alternatively, the control device 2 may be built into the radio wave sensor 1.

(8) First Modification of Device Control System: Human Body Pose Estimation System In this modification, the elements related to the device control function, i.e., the behavior detection unit 224, the device control unit 225, the information acquisition unit 226, the history accumulation unit 227, and the learning unit 228, are omitted from the device control system 100 shown in Fig. 1. This device control system 100 includes at least a distance measurement unit 221, a point placement unit 222, and a posture estimation unit 223, and may be referred to as a "human body posture estimation system 100."

Furthermore, the human body posture estimation system 100 may further include, in addition to the distance measurement unit 221, point placement unit 222, and posture estimation unit 223, a behavior detection unit 224, a device control unit 225, an information acquisition unit 226, a history storage unit 227, and a learning unit 228.

(9) Variation of Difference Acquisition: Intra-Frame Difference The radio wave sensor 1 may perform transmission and reception operations N times (where N is an integer greater than or equal to 4) per frame, with one frame being a predetermined time, and output N pieces of location identifiable information per frame. The processing unit 22 holds at least N pieces of location identifiable information corresponding to one frame. The processing unit 22 then selects three or more pieces of location identifiable information (representative location identifiable information) representing one frame from the N or more pieces of location identifiable information held corresponding to one or more frames, and acquires two or more differences for the selected three or more representative location identifiable information. The processing unit 22 may then acquire two or more pieces of location information based on two or more differences acquired for the three or more representative location identifiable information representing one frame.

In this example, the processing unit 22 further stores N pieces of location identifiable information corresponding to the frame following the first frame. When two or more pieces of location information acquired for three or more pieces of representative location identifiable information all correspond to a moving object (200c) other than a human body 301, the processing unit 22 treats the first frame and the next frame as a new frame. Then, from the 2 x N pieces of location identifiable information corresponding to the new frame, the processing unit 22 selects three or more representative location identifiable information representing the new frame, and obtains two or more differences for the selected three or more pieces of representative location identifiable information. The processing unit 22 may obtain two or more pieces of location information based on the two or more differences obtained for the three or more representative location identifiable information representing such a new frame.

(10) Second Modification of Device Control System: Device Control System Including Moving Object Estimation System In this modification, the explanation of the matters already mentioned in the embodiment may be omitted or simplified.

As shown in FIG. 16, the device control system 100 in this modified example includes a moving object estimation system 100A and a device control device 2B. The moving object estimation system 100A includes a radio wave sensor 1 and a moving object estimation device 2A. The moving object estimation device 2A includes a conversion unit 11, a storage unit 12, a difference acquisition unit 13, a moving object estimation unit 14, and a point placement unit 222. The moving object estimation unit 14 includes a distance measurement unit 221, a posture estimation unit 223, and a behavior detection unit 224. The device control device 2B includes a device control unit 225, an information acquisition unit 226, a history accumulation unit 227, and a learning unit 228.

In the example of FIG. 16, the moving object estimation device 2A includes a conversion unit 11, but the conversion unit 11 may be built into the radio wave sensor 1 or may be located between the radio wave sensor 1 and the moving object estimation device 2A. In the embodiment described above, although not shown, the conversion unit 11 is built into the radio wave sensor 1.

In the example of Figure 16, the moving object estimation device 2A and the equipment control device 2B are separate entities, but they may also be configured as an integrated unit. Furthermore, although omitted from Figure 16, the moving object estimation system 100A typically further includes a reception unit 21, a processing unit 22 (e.g., a first processing unit on the moving object estimation device 2A side and a second processing unit on the equipment control device 2B side), and an output unit 23 (see Figure 1).

(10-1) Radio wave sensor The radio wave sensor 1 constituting the moving object estimation system 100A transmits a transmission wave Tr, which is a radio wave modulated in a predetermined manner, toward a real space (for example, the inside of the room 400 shown in FIG. 2 ; hereinafter, referred to as “real space (400)”), receives a reflected wave Re from the real space (400), and performs a transmission/reception operation to output an output signal based on the transmission wave Tr and the reflected wave Re.

As in the embodiment, the transmission and reception operations are performed repeatedly at a predetermined cycle (i.e., periodically), but may also be performed irregularly.

Note that while the radio wave sensor 1 in the embodiment outputs position-identifying information such as FFT results, the radio wave sensor 1 in this modified example outputs an output signal such as an IF signal. Conversion of the output signal into position-identifying information is performed outside the radio wave sensor 1 (for example, by a conversion unit 11 constituting the moving object estimation device 2A).

(10-1-1) Output Signal In the case of radio waves modulated by the FMCW method, for example, the output signal is a signal (IF signal: see FIG. 3) indicating the frequency difference Δf between the transmitted wave Tr and the reflected wave Re at the same time. However, the output signal may also be a signal indicating the time difference Δt between the transmitted wave Tr and the reflected wave Re, which is calculated from the frequency difference Δf. Note that in the case of radio waves modulated by the pulse modulation method, the output signal may also be a signal indicating the time difference Δt between the transmitted wave Tr and the reflected wave Re.

(10-1-2) Modulation Method The predetermined method, i.e., the radio wave modulation method, is the FMCW method in this example, but it may also be, for example, a pulse modulation method, or any method that results in the acquisition of location-identifiable information.

(10-2) Moving Object Estimation Device The conversion unit 11 constituting the moving object estimation device 2A performs conversion processing. The conversion processing is processing for converting the output signal output by the radio wave sensor 1 into position identifiable information. The conversion processing is, for example, processing for performing a Fourier transform on an IF signal, which is one type of output signal. The Fourier transform is, for example, an FFT, but may also be an STFT (short-time Fourier transform) or the like.

Location-identifiable information is information that can identify the position of one or more objects (human body 301, electric blinds 200c, stationary object 302) present in real space (400), such as, but not limited to, the FFT results described in the embodiment.

The storage unit 12 performs a storage process. In this example, the storage process is a process of storing the position identifiable information converted by the conversion process. The position identifiable information is written, for example, to a memory possessed by the moving object estimation device 2A, and stored in the memory for a predetermined period (for example, one frame period, three frame periods, etc.).

The difference acquisition unit 13 performs difference acquisition processing. This processing acquires the differences between multiple pieces of location-identifiable information stored in the storage processing.

The moving object estimation unit 14 performs moving object estimation processing. Moving object estimation processing is processing for making estimations regarding moving objects. For example, the moving object estimation processing is processing for estimating whether each of one or more objects is a moving object or a non-moving object (stationary object 302) based on the differences acquired by the difference acquisition processing, identifying the position of an object estimated to be a moving object (human body 301, electric blinds 200c), and acquiring moving object position information indicating the identified position.

The point placement unit 222 performs point placement processing. Point placement processing is processing in which points corresponding to the moving object position information acquired by the moving object estimation processing are virtually placed in a virtual space (e.g., three-dimensional space 500 in the embodiment) corresponding to the real space (400). The point placement processing may be, for example, processing such as multi-point placement described as the operation of the point placement unit 222 in the embodiment.

The difference acquisition process in this example acquires two or more differences with different time differences for the three or more pieces of position identifiable information stored in the storage process. The moving object estimation process acquires two or more pieces of moving object position information corresponding to the two or more differences acquired by the difference acquisition process. The point placement process virtually places two or more points in virtual space (500) corresponding to the two or more pieces of moving object position information acquired by the moving object estimation process.

Here, "virtually placing two or more points in virtual space (500)" includes the case where "all points are placed" and the case where "only some points are placed" (in other words, at least one point is placed) of the two or more points corresponding to the two or more pieces of moving object position information acquired by the moving object estimation process, and may also include the case where "not a single point is placed."

However, cases in which "not a single point is placed" may be excluded, and such point placement processing virtually places in the virtual space (500) "all or part" of two or more points corresponding to two or more pieces of moving object position information acquired by the moving object estimation processing. In other words, even if all of the two or more points are determined to be moving objects other than a human body 301, for example, one representative point of the two or more points (such as any one point or a point corresponding to the average value of the two or more pieces of moving object position information corresponding to the two or more points) may be placed.

Note that if all of the two or more points corresponding to the two or more pieces of moving body position information acquired by the moving body estimation process are always positioned, there is no need to determine whether the moving body is a human body 301 or a moving object other than a human body 301 (whether the moving body position information corresponds to a human body 301 or a moving object other than a human body 301), as is done in this example.

In this example, the above judgment is made for each of the two or more pieces of moving object position information acquired, and if it is judged that the information corresponds to a human body 301, a point corresponding to that moving object position information is placed. Therefore, depending on the individual judgment results, there may be cases where all of the two or more points corresponding to the two or more pieces of moving object position information are placed, where only some of the points are placed (at least one point is placed), or where no points are placed.

In addition, for example, if the moving object position information to be judged is information that is unlikely to yield a valid judgment result, it may be excluded from the judgment, resulting in only some points or no points being placed. In this example, for example, based on information such as the reception strength of the reflected wave Re corresponding to the moving object position information, it may be determined whether or not each of two or more moving object position information acquired by the moving object estimation process should be subject to the above judgment, and the above judgment may be made only for the moving object position information that has been judged to be subject.

(10-2-1) Details of Point Placement In the point placement process, for example, only the point corresponding to the human body 301 among the moving objects (i.e., some of the two or more points) may be placed. In other words, points corresponding to moving objects other than the human body 301 may not be placed. However, points corresponding to moving objects other than the human body 301 may also be placed (i.e., all of the two or more points).

Note that the point placement process may, for example, place points corresponding to the human body 301 and living bodies other than the human body 301 (such as the bodies of pets: not shown) among the moving bodies. In other words, points corresponding to moving bodies other than living bodies may be excluded from placement targets.

In this way, by obtaining two or more differences with different time differences for three or more pieces of location-identifying information, it is possible to expect an increase in the number of points placed in the virtual space (500) (increasing the number of points in the point cloud 501). As a result, it becomes possible to more accurately determine whether the point cloud 501 corresponds to a moving object such as a human body 301 or a stationary object 302, thereby improving the accuracy of estimations regarding moving objects using the radio wave sensor 1.

Furthermore, as a result of improving the accuracy of estimations regarding moving objects in this way, the feasibility of controlling the device 200 using the radio wave sensor 1 (for example, the control accuracy of the device control device 2B described below) is improved.

(10-2-2) Estimation of Moving Object Position Information: Is the Moving Object a Human Body or a Moving Object Other Than a Human Body? The moving object is either the human body 301 or a moving object other than the human body 301. The moving object other than the human body 301 is, for example, the electric blinds 200c that open and close electrically, but it may be any moving object, such as a cleaning robot that cleans while moving. The moving object other than the human body 301 may also include the body of a living organism other than a human (for example, a pet kept by a person).

The moving object estimation process further estimates whether each of the two or more pieces of moving object position information acquired corresponds to a human body 301 or a moving object (200c) other than a human body 301, based at least on the change in the difference. The point placement process virtually places in the virtual space (500) points corresponding to moving object position information that the moving object estimation process estimates to correspond to a human body 301, out of the two or more pieces of moving object position information acquired by the moving object estimation process. In other words, points corresponding to moving object position information that the human body estimation process estimates to correspond to a moving object other than a human body 301 are excluded from placement targets.

In this way, when making an estimation regarding the human body 301, the moving object estimation system 100A obtains two or more differences with different time differences for three or more pieces of position-identifying information, making it possible to detect various movements of the human body 301 (e.g., body movement, slight breathing movements, hand and foot movements, etc.: see Figure 11), thereby improving the accuracy of estimation regarding the human body 301.

(10-2-3) Variant of estimation regarding moving object position information: whether the moving object other than a human body is a living organism or a moving object other than a living organism. The moving object other than the human body 301 may be, for example, either a living organism other than the human body 301 (for example, the body of a living creature such as a pet kept by a person: not shown) or a moving object other than a living organism (such as electric blinds 200c).

In this case, the point placement process places points corresponding to living bodies in the virtual space (500). In other words, points corresponding to moving objects other than living bodies are excluded from placement targets.

In this example, the device control process (described below) may perform control such as turning on lighting fixtures or turning on the air conditioner when a point cloud 501 corresponding to a living organism is detected (for example, when a person or pet enters a room).

Furthermore, the moving object estimation process may distinguish the multiple points in the virtual space (500) into a point cloud 501 corresponding to the human body 301 and a point cloud 501 corresponding to a non-human living body based on the distribution of the multiple points in the virtual space (500) (such as the shape of the cluster CL).

The device control process may perform different device control depending on whether the point cloud 501 corresponds to a human body 301 or a living organism other than a human body 301 (for example, a pet). In more detail, the device control process may perform control such that the television is turned on when point cloud 501 corresponding to a human body 301 is detected, but the television is not turned on even when point cloud 501 corresponding to a living organism other than a human body 301 (a pet) is detected. In this way, by performing different device control depending on whether the target object is a human body 301 or a living organism other than a human body 301, it is possible to create a more comfortable environment for people and pets to coexist, and even to achieve both comfort and energy savings.

(10-2-4) Estimation taking into account the reception strength of reflected waves The moving object estimation process estimates whether each of the two or more acquired moving object position information corresponds to a human body 301 or a moving object (200c) other than a human body 301 based on at least one of the reception strength of the reflected wave Re corresponding to the moving object position information and the change in the reception strength, in addition to the change in the difference.

In this way, by utilizing at least one of the received intensity and the change in received intensity of the reflected wave Re in addition to the change in the difference, it is possible to improve the accuracy of estimating whether the moving object (301, 200c) is a human body 301 or a moving object (200c) other than a human body 301.

(10-2-5) Estimation of Point Clouds The moving object estimation process estimates whether the point cloud 501 as a whole corresponds to the human body 301 or to a moving object (200c) other than the human body 301, further based on at least one of the distribution and a change in the distribution of the point cloud 501. In this way, by also using at least one of the distribution and a change in the distribution of the point cloud 501 arranged in the virtual space (500), it is possible to improve the accuracy of estimation as to whether the moving object (301, 200c) is the human body 301 or a moving object (200c) other than the human body 301.

(10-2-6) Inter-Frame Difference The radio wave sensor 1 performs transmission and reception operations N times (N is an integer equal to or greater than 1) per frame, with a predetermined time being one frame. As a result, N pieces of location identifiable information are output from the radio wave sensor 1 per frame. The storage process stores 3×N or more pieces of location identifiable information corresponding to three or more frames. The difference acquisition process selects one piece of representative location identifiable information representing each of the three or more frames from the 3×N or more pieces of location identifiable information corresponding to three or more frames stored in the storage process. The difference acquisition process then acquires two or more differences for the selected three or more pieces of representative location identifiable information. The moving object estimation process acquires two or more pieces of moving object location information based on two or more differences acquired for three or more pieces of representative location identifiable information representing each of the three or more frames.

This allows three or more pieces of representative location-identifying information to be selected over three or more frames, and two or more differences to be obtained for the three or more selected pieces of representative location-identifying information.

(10-2-7) Intra-Frame Difference The radio wave sensor 1 performs transmission and reception operations N times (N is an integer equal to or greater than 3) per frame, with a predetermined time being one frame, and outputs N or more output signals per frame. The conversion process converts each of the N or more output signals output per frame into location-identifiable information. The retention process retains at least the N pieces of location-identifiable information corresponding to one frame.

The difference acquisition process selects three or more representative pieces of location identifiable information representing one frame from the N or more pieces of location identifiable information corresponding to one or more frames held by the holding process. The difference acquisition process then acquires two or more differences for the three or more selected pieces of representative location identifiable information. The moving object estimation process acquires two or more pieces of moving object location information based on the two or more differences acquired for the three or more pieces of representative location identifiable information representing one frame.

This allows three or more pieces of representative location-identifying information to be selected within one frame, and two or more differences to be obtained for the three or more selected pieces of representative location-identifying information.

(10-2-7a) Frame Combination The holding process further holds N pieces of location identifiable information corresponding to the frame following the first frame. When two or more pieces of moving object location information acquired by the moving object estimation process for three or more pieces of representative location identifiable information all correspond to moving objects (200c) other than the human body 301, the difference acquisition process treats the first frame and the next frame as a new frame. Then, the difference acquisition process selects three or more pieces of representative location identifiable information representing the new frame from the 2×N pieces of location identifiable information corresponding to the new frame, and acquires two or more differences for the selected three or more pieces of representative location identifiable information. The moving object estimation process acquires two or more pieces of moving object location information based on the two or more differences acquired by the difference acquisition process for the three or more pieces of representative location identifiable information representing the new frame.

As a result, if two or more differences cannot be obtained within one frame, a new frame can be created with the next frame, and two or more differences can be obtained within the new frame.

(10-2-8) Specific Examples of Moving Object Estimation (10-2-8a) Identifying One-Dimensional Position Using One Antenna The radio wave sensor 1 has at least one antenna that receives reflected waves Re. The output signal is an IF signal generated based on the transmitted waves Tr and the reflected waves Re received by the one antenna for one transmission/reception operation. The IF signal is a signal that indicates the frequency difference between the transmitted waves Tr and the reflected waves Re, and is generated by mixing the transmitted waves Tr and the reflected waves Re.

The transform process is a Fourier transform process. The Fourier transform process is a process in which a Fourier transform is performed on an IF signal. The location-identifying information is the Fourier transform result of the Fourier transform process. The storage process stores multiple Fourier transform results corresponding to one antenna.

The moving object estimation process includes a ranging process. In this example, the ranging process is a process of measuring the distance from one antenna to an object estimated to be a moving object (301, 200c) based on the difference between multiple Fourier transform results corresponding to one antenna. The moving object position information is one-dimensional position information based on the ranging results of the ranging process.

This allows for improved accuracy when making estimates (moving object estimation processing) regarding moving objects (301, 200c) using the radio wave sensor 1, which uses radio waves modulated using the FMCW method. Furthermore, even if there is only one antenna for receiving the reflected wave Re, it is possible to at least improve the accuracy of measuring the distance to the moving object (301, 200c) (determining its one-dimensional position).

(10-2-8b) Identifying Two-Dimensional or Three-Dimensional Position Using Multiple Antennas The radio wave sensor 1 has multiple antennas that receive reflected waves Re. Multiple IF signals corresponding to the multiple antennas are output from the radio wave sensor 1 for each transmission/reception operation. For each of these multiple antennas, the converter 11 performs a conversion process, the holder 12 performs a storage process, the difference acquirer 13 performs a difference acquisition process, and the moving object estimator 14 performs a moving object estimation process including a ranging process.

The location-identifiable information is a group of Fourier transform results consisting of multiple Fourier transform results corresponding to multiple antennas. The storage process stores the multiple Fourier transform result groups. The moving object estimation process includes multiple ranging processes that measure the distance from each of the multiple antennas to an object estimated to be a moving object (301, 200c) based on the differences between the multiple Fourier transform result groups for each of the multiple antennas. The moving object position information is two-dimensional or three-dimensional position information based on multiple ranging results corresponding to the multiple ranging processes.

This improves the accuracy of the moving object estimation process, which identifies the two-dimensional position (e.g., two-dimensional coordinates in the two-dimensional virtual space 500, or a set of direction and distance, etc.) or three-dimensional position (e.g., three-dimensional coordinates in the three-dimensional virtual space 500, or a set of direction and distance, etc.) of the moving object (301, 200c).

(10-2-9) Moving object estimation method and program The moving object estimation method in this example includes at least step S1 (conversion step), step S2 (storage step), step S4 (difference acquisition step), steps S5 and S6 (moving object estimation step), and step S7 (point placement step) among the various steps described in the embodiment. The program causes one or more processors to execute this moving object estimation method.

(10-3) Device Control Device The device control unit 225 constituting the device control device 2B performs device control processing to control the device 200 based at least on a point cloud 501, which is a collection of multiple points arranged in a virtual space (500). In this way, the radio wave sensor 1 acquires the point cloud 501 corresponding to the moving object (301, 200c), and the device 200 is controlled based on the acquired point cloud 501, thereby realizing control of the device 200 using the radio wave sensor 1.

In particular, by acquiring a point cloud 501 corresponding to the human body 301 and controlling the device 200 based on the acquired point cloud 501, it is possible to achieve control of the device 200 based on the position and movement of the human body 301.

(10-3-1) Posture Estimation and Behavior Detection The moving object estimation process by the moving object estimation device 2A includes a posture estimation process. The posture estimation process is a process for estimating the posture of the human body 301. The posture estimation process estimates the posture of the human body 301 based on, for example, the distribution of the point cloud 501 (FIGS. 12A to 13C). Specifically, the posture estimation process may be, for example, the process described in the embodiment using FIG. 6.

The behavior detection unit 224 executes behavior detection processing. In this example, the behavior detection processing is processing for detecting the behavior of a person corresponding to the human body 301 based on the posture or posture change estimated by the posture estimation processing. In this example, the device control processing controls the device 200 based on the behavior detected by the behavior detection processing.

In this way, the posture or posture changes of the human body 301 are estimated based on the distribution of the point cloud 501, the human behavior corresponding to the human body 301 is detected based on the estimation results, and the device 200 is controlled based on the detection results, thereby achieving control of the device 200 based on the posture of the human body 301 and, ultimately, the human behavior.

(10-3-2) Acquisition of Environmental Information The information acquisition unit 226 executes information acquisition processing. As described above, the information acquisition processing is processing for acquiring environmental information related to the environment in which the human body 301 exists. The moving object estimation processing detects behavior based on the environmental information acquired by the information acquisition processing.

In this way, by using environmental information in addition to posture or posture changes, the accuracy of behavior detection and, ultimately, the accuracy of control of the device 200 can be improved.

(10-3-3) Device Control Using Automatic Control Information The device control process controls the device 200 based on behavior using pre-stored automatic control information. This allows the device 200 to be automatically controlled based on behavior using the automatic control information.

(10-3-4) Accumulation of Automatic Control History Information The history accumulation unit 227 in this example executes a history accumulation process. The history accumulation process in this example accumulates automatic control history information. Accumulating automatic control history information in this manner enables learning processing based on the automatic control history information, and ultimately makes it possible to improve control accuracy using the automatic control history information. Furthermore, as described above, the automatic control history information associates the behavior-based control content of the device 200 with one or more pieces of information from the time, environmental information, and person identifier. This allows control suited to each individual person.

(10-3-5) Further Accumulation of Manual Control History Information The history accumulation process further accumulates manual control history information. The manual control history information is information about the control history of the device 200 based on operations on the device 200. The operations are, for example, operations to adjust the brightness of lighting, as described above, and are accepted via an operating device such as a remote control. In this way, by further accumulating manual control history information, it becomes possible to improve control accuracy by utilizing two types of control history information, automatic and manual.

For example, based on the accumulated automatic control history information and manual control history information, if an operation to cancel the automatic control (e.g., dim the lights) is not performed within a certain time (e.g., within one minute) after automatic control to brighten the lights in response to the detection of seating is executed (hereinafter referred to as "tacit inaction") a predetermined number of times or more within a predetermined period (e.g., three or more times in one week), the system will learn that the control is appropriate, thereby improving control accuracy.

However, the above-described learning process can be realized by accumulating only automatic control history information, without accumulating manual control history information. For example, while accumulating automatic control history information, a determination can be made based on real-time operation information as to whether tacit inaction has been detected a predetermined number of times or more in a predetermined period (e.g., three or more times per week), and if tacit inaction has been detected a predetermined number of times or more in a predetermined period, the learning process can be repeated to learn that the control content is appropriate.

Alternatively, only manual control history information may be stored without storing automatic control history information. For example, based on the stored manual control history information, if an operation to brighten the lights is received within a certain time period (e.g., within one minute) after detecting that someone is seated is detected a predetermined number of times or more within a predetermined period (e.g., three or more times per week), the control content "brighten the lights in response to the detection of someone seated" is registered in the automatic control information. By repeating this process, it is possible to improve the control accuracy of automatic control of lighting brightness and diversify the content of automatic control.

As described above, the manual control history information associates the control content of the device 200 based on operations with one or more pieces of information from the time, environmental information, and person identifier. The learning unit 228 executes a learning process to update the automatic control information, for example, when the time and operation indicated by the control history information satisfy predetermined update conditions. This learning process using automatic control history information makes it possible to achieve control that is suited to the behavior of each individual person.

(10-3-6) Updating of Automatic Control Information Alternatively, the learning unit 228 may execute a learning process to update the automatic control information when a predetermined update condition is satisfied between the time and control content indicated by the information and the time and operation indicated by manual control history information regarding the control history of the device 200 based on operations on the device 200. Such a learning process using two types of control history information, automatic and manual, makes it possible to achieve control suited to the behavior and operation of each individual person.

(10-3-7) Clustering The virtual space (500) in this example is a three-dimensional space 500, similar to that in the embodiment. The moving object estimation process performs clustering on the point cloud 501 to obtain a cluster group (CL, CL1, CL2), which is a set of one or more clusters (CL, CL1, CL2), and estimates the posture of the human body 301 based on the distribution of the cluster group (CL, CL1, CL2) in the three-dimensional space 500. This makes it possible to accurately estimate the posture of the human body 301 based on the distribution of the cluster group (CL, CL1, CL2).

(10-3-8) Posture Estimation The moving object estimation process estimates the posture of the human body 301 based on the ratios of the length, width, and height of the three-dimensional figure 502 that surrounds the cluster group (CL, CL1, CL2). This makes it possible to accurately and easily estimate the posture of the human body 301 based on the ratios of the length D, width W, and height H of the three-dimensional figure 502 that surrounds the cluster group (CL, CL1, CL2).

(10-3-9) Detection of Behavior The behavior detection process detects a non-operation behavior, which is a behavior different from an operation of the human body 301 on the device 200, based at least on the posture or a change in posture estimated by the posture estimation process. The device control process controls the device 200 based at least on the detected non-operation behavior. In this way, by controlling the device 200 based on the non-operation behavior (for example, a behavior that does not intend to operate the device 200, such as a daily activity), it is possible to control the device 200 without intentionally operating the device 200 (operation-less device control).

(10-3-10) Device Control Method and Program The device control method uses a radio wave sensor 1 to make an estimation regarding a moving object (a human body 301, an electric blind 200c), and controls the device 200 based on the estimation results. The radio wave sensor 1 performs a transmission/reception operation of transmitting a transmission wave Tr, which is a radio wave modulated using a predetermined method, toward a real space (the interior of a room 400), receiving a reflected wave Re from the real space (400), and outputting an output signal based on the transmission wave Tr and the reflected wave Re. The device control method includes a conversion step (S1), a holding step (S2), a difference acquisition step (S4), a moving object estimation step (S5, S6), a point arrangement step (S7), and a device control step (S11). The conversion step (S1) performs a conversion process to convert the output signal output by the radio wave sensor 1 into position-identifiable information that can identify the position of each of one or more objects (301, 200c, stationary object 302) present in the real space (400). The storage step (S2) performs a storage process to store the position-identifiable information converted by the conversion process. The difference acquisition step (S4) performs a difference acquisition process to acquire differences between the multiple pieces of position-identifiable information stored in the storage process. The moving object estimation steps (S5, S6) determine whether each of the one or more objects is a moving object or a stationary object based on the differences acquired in the difference acquisition process. The method performs a moving object estimation process to estimate whether the object is a moving object (302), identify the position of the object estimated to be a moving object (301, 200c), and acquire moving object position information indicating the identified position. The point arrangement step (S7) performs a point arrangement process to virtually arrange points corresponding to the moving object position information acquired by the moving object estimation process in a virtual space (500) corresponding to the real space (400). The device control step (S11) controls the device 200 based at least on a point cloud 501, which is a collection of multiple points arranged in the virtual space (500). The program causes one or more processors to execute this device control method.

(11) Summary A moving object estimation system (100, 100A) according to a first aspect of the present disclosure is a moving object estimation system (100, 100A) that uses a radio wave sensor (1) to perform estimation of a moving object (a human body 301, an electric blind 200c). The moving object estimation system (100, 100A) includes a radio wave sensor (1), a conversion unit (11), a storage unit (12), a difference acquisition unit (13), a moving object estimation unit (14), and a point placement unit (222). The radio wave sensor (1) transmits a transmission wave (Tr), which is a radio wave modulated using a predetermined method, toward a real space (inside a room 400), receives a reflected wave (Re) from the real space (400), and performs a transmission/reception operation to output an output signal based on the transmission wave (Tr) and the reflected wave (Re). The conversion unit (11) performs a conversion process to convert an output signal output by the radio wave sensor (1) into position-identifiable information capable of identifying the position of each of one or more objects (301, 200c, stationary object 302) present in the real space (400). The storage unit (12) performs a storage process to store the position-identifiable information converted by the conversion process. The difference acquisition unit (13) performs a difference acquisition process to acquire differences between the plurality of position-identifiable information stored by the storage process. The moving object estimation unit (14) performs a moving object estimation process to estimate whether each of the one or more objects is a moving object or a stationary object (302) based on the differences acquired by the difference acquisition process, identify the positions of objects estimated to be moving objects (301, 200c), and acquire moving object position information indicating the identified positions. The point placement unit (222) performs a point placement process to virtually place points corresponding to the moving object position information acquired by the moving object estimation process in a virtual space (500) corresponding to the real space (400). The difference acquisition process acquires two or more differences with different time differences for three or more pieces of position identifiable information at different times stored in the storage process. The moving object estimation process acquires two or more pieces of moving object position information corresponding to the two or more differences acquired by the difference acquisition process. The point placement process virtually places two or more points in a virtual space (500) corresponding to the two or more pieces of moving object position information with different time differences acquired by the moving object estimation process.

According to this aspect, by obtaining two or more differences with different time differences for three or more pieces of location-identifying information, it is possible to expect an increase in the number of points to be placed in the virtual space (500). As a result, it is possible to more accurately determine, for example, whether a point cloud (501) consisting of multiple points placed in the virtual space (500) corresponds to a moving object (301, 200c) or a non-moving object (stationary object 302). Therefore, it is possible to improve the accuracy of estimations regarding moving objects (301, 200c) using the radio wave sensor (1).

In the moving object estimation system (100, 100A) according to the second aspect, in the first aspect, the moving object (301, 200c) is either a human body (301) or a moving object (200c) other than a human body (301). The moving object estimation process further estimates whether each of the two or more pieces of acquired moving object position information corresponds to a human body (301) or a moving object (200c) other than a human body (301) based on at least a change in the difference. The point placement process virtually places in the virtual space (500) points corresponding to the moving object position information that the moving object estimation process further estimates as corresponding to a human body (301) out of the two or more pieces of moving object position information acquired by the moving object estimation process.

According to this aspect, when making an estimation regarding a human body (301), such as determining whether a moving body (301, 200c) is a human body (301) or a moving object (200c) other than a human body (301), by obtaining two or more differences with different time differences for three or more pieces of location-identifying information, it becomes possible to detect various movements of the human body (301) (e.g., body movement, slight breathing movements, hand and foot movements, etc.). As a result, the accuracy of estimation regarding the human body (301) can be improved.

In the moving object estimation system (100, 100A) according to the third aspect, in the second aspect, the moving object estimation process further estimates whether each of the two or more acquired pieces of moving object position information corresponds to a human body (301) or a moving object (200c) other than a human body (301) based on at least one of the reception strength of the reflected wave (Re) corresponding to the moving object position information and the change in the reception strength.

According to this aspect, by utilizing at least one of the received strength and the change in received strength of the reflected wave (Re) in addition to the change in the difference, it is possible to improve the accuracy of estimating whether the moving object (301, 200c) is a human body (301) or a moving object (200c) other than a human body (301).

In the moving object estimation system (100, 100A) according to the fourth aspect, in the second or third aspect, the moving object estimation process further estimates whether a point cloud (501), which is a collection of multiple points arranged in a virtual space (500), corresponds to a human body (301) or a moving object (200c) other than a human body (301), based on at least one of the distribution and a change in the distribution of the point cloud (501).

According to this aspect, by utilizing at least one of the distribution and changes in distribution of the point cloud (501) arranged in the virtual space (500), it is possible to improve the accuracy of estimation as to whether the moving object (301, 200c) is a human body (301) or a moving object (200c) other than a human body (301).

In the moving object estimation system (100, 100A) according to the fifth aspect, in the first aspect, the radio wave sensor (1) performs transmission and reception operations N times (N is an integer greater than or equal to 1) per frame, with a predetermined time period being one frame, and outputs N pieces of location identifiable information per frame. The storage process stores 3 x N or more pieces of location identifiable information corresponding to three or more frames. The difference acquisition process selects one representative piece of location identifiable information representing each of the three or more frames from the 3 x N or more pieces of location identifiable information corresponding to three or more frames stored in the storage process, and acquires two or more differences for the selected three or more pieces of representative location identifiable information. The moving object estimation process acquires two or more pieces of moving object location information based on two or more differences acquired for the three or more representative location identifiable information representing each of the three or more frames.

According to this aspect, one representative piece of location identifiable information is selected from N pieces of location identifiable information (N is an integer greater than or equal to 1) corresponding to each of three or more frames, and two or more differences (inter-frame differences) are obtained for the three or more selected pieces of representative location identifiable information. In this way, two or more pieces of location information can be obtained based on the two or more differences obtained between frames.

In the moving object estimation system (100, 100A) according to the sixth aspect, in the first aspect, the radio wave sensor (1) performs transmission and reception operations N times (N is an integer greater than or equal to 3) per frame, with a predetermined time period being one frame, and outputs N or more output signals per frame. The storage process stores at least N pieces of location identifiable information corresponding to one frame. The difference acquisition process selects three or more representative location identifiable information representing one frame from the N or more pieces of location identifiable information corresponding to one or more frames stored in the storage process, and acquires two or more differences for the selected three or more representative location identifiable information. The moving object estimation process acquires two or more pieces of moving object location information based on two or more differences acquired for the three or more representative location identifiable information representing one frame.

According to this aspect, three or more representative pieces of location identifiable information are selected from N pieces of location identifiable information (N is an integer greater than or equal to three) corresponding to one frame, and two or more differences (intra-frame differences) are obtained for the three or more selected representative pieces of location identifiable information. In this way, two or more pieces of location information can be obtained based on the two or more differences obtained within the frame.

In the sixth aspect, the storage process may further store N pieces of location identifiable information corresponding to the frame following the one frame. When two or more pieces of moving object location information acquired by the moving object estimation process for three or more pieces of representative location identifiable information all correspond to moving objects (200c) other than a human body (301), the difference acquisition process sets the one frame and the next frame as a new frame, selects three or more representative location identifiable information representing the new frame from the 2 x N pieces of location identifiable information corresponding to the new frame, and acquires two or more differences for the selected three or more representative location identifiable information. The moving object estimation process acquires two or more pieces of moving object location information based on the two or more differences acquired by the difference acquisition process for the three or more representative location identifiable information representing the new frame.

In this way, if the difference cannot be obtained within a frame, it is possible to obtain the difference by combining it with the next frame.

In the moving object estimation system (100, 100A) according to the seventh aspect, in the first aspect, the predetermined method is the FMCW method. The radio wave sensor (1) has at least one antenna that receives reflected waves (Re). The output signal is an IF signal generated based on the transmitted wave (Tr) and the reflected wave (Re) received by one antenna for one transmission/reception operation. The conversion process is a Fourier transform process that performs a Fourier transform on the IF signal. The location identifiable information is the Fourier transform result of the Fourier transform process. The storage process stores multiple Fourier transform results. The moving object estimation process includes a ranging process that measures the distance from one antenna to an object estimated to be a moving object (301, 200c) based on the difference between multiple Fourier transform results corresponding to one antenna. The moving object location information is one-dimensional location information based on the ranging result of the ranging process.

This aspect makes it possible to improve the accuracy of estimations (moving object estimation processing) regarding moving objects (301, 200c) using a radio wave sensor 1 that uses radio waves modulated by the FMCW method. Furthermore, even if there is only one antenna that receives reflected waves (Re), it is possible to at least improve the accuracy of measuring the distance to the moving object (301, 200c) (determining its one-dimensional position).

In the seventh aspect, the Fourier transform may be an FFT. In this case, the location identifiable information is the FFT result obtained by performing an FFT on the IF signal.

In this way, by using the FFT results obtained by performing FFT on the IF signal, it is possible to speed up the Fourier transform and, ultimately, the moving object estimation process, while improving estimation accuracy.

In the moving object estimation system (100, 100A) according to the eighth aspect, in the seventh aspect, the radio wave sensor (1) has multiple antennas that receive reflected waves (Re) and outputs multiple IF signals corresponding to the multiple antennas in one transmission/reception operation. For each of the multiple antennas, a conversion unit (11) performs conversion processing, a storage unit (12) performs storage processing, a difference acquisition unit (13) performs difference acquisition processing, and a moving object estimation unit (14) performs moving object estimation processing including a ranging processing. The location-identifiable information is a Fourier transform result group consisting of multiple Fourier transform results corresponding to the multiple antennas. The storage processing stores the multiple Fourier transform result groups. The moving object estimation processing includes multiple ranging processing that measures the distance from each of the multiple antennas to an object estimated to be a moving object (301, 200c) based on the differences between the multiple Fourier transform result groups for each of the multiple antennas. The moving object position information is two-dimensional or three-dimensional position information based on multiple ranging results corresponding to the multiple ranging processing.

This aspect improves the accuracy of the moving object estimation process that identifies the two-dimensional position (e.g., two-dimensional coordinates in the two-dimensional virtual space 500, or a set of direction and distance, etc.) or three-dimensional position (e.g., three-dimensional coordinates in the three-dimensional virtual space 500, or a set of direction and distance, etc.) of the moving object (301, 200c).

The moving object estimation method according to the ninth aspect is a moving object estimation method that uses a radio wave sensor (1) to make an estimation regarding a moving object (human body 301, electric blinds 200c). The moving object estimation method includes a transmission/reception step (not shown), a conversion step (S1), a retention step (S2), a difference acquisition step (S4), a moving object estimation step (S5, S6), and a point placement step (S7). In the transmission/reception step (not shown), the radio wave sensor (1) transmits a transmission wave (Tr), which is a radio wave modulated using a predetermined method, toward a real space (inside a room 400), receives a reflected wave (Re) from the real space (400), and performs a transmission/reception operation to output an output signal based on the transmission wave (Tr) and the reflected wave (Re). The conversion step (S1) performs a conversion process to convert the output signal output by the radio wave sensor (1) into location-identifiable information capable of identifying the respective positions of one or more objects (301, 200c, stationary object 302) present in the real space (400). The storage step (S2) performs a storage process to store the location-identifiable information converted by the conversion process. The difference acquisition step (S4) performs a difference acquisition process to acquire differences between the multiple location-identifiable information stored in the storage process. The moving object estimation steps (S5, S6) perform a moving object estimation process to estimate whether each of the one or more objects is a moving object or a stationary object (302) based on the differences acquired by the difference acquisition process, identify the positions of the objects estimated to be moving objects (301, 200c), and acquire moving object position information indicating the identified positions. The point placement step (S7) performs a point placement process to virtually place points corresponding to the moving object position information acquired by the moving object estimation process in a virtual space (500) corresponding to the real space (400). The difference acquisition process acquires two or more differences with different time differences for three or more pieces of position identifiable information at different times that are stored in the storage process. The moving object estimation process acquires two or more pieces of moving object position information corresponding to the two or more differences acquired by the difference acquisition process. The point placement process virtually places two or more points in virtual space (500) that correspond to the two or more pieces of moving object position information with different time differences that are acquired by the moving object estimation process.

According to this aspect, as with the first aspect, it is possible to improve the accuracy of estimations regarding moving objects (301, 200c) using the radio wave sensor (1).

A program according to an eleventh aspect causes one or more processors to execute the moving object estimation method according to the tenth aspect.

According to this aspect, as with the first aspect, it is possible to improve the accuracy of estimations regarding moving objects (301, 200c) using the radio wave sensor (1).

100 Equipment control system (human body posture estimation system, moving body estimation system)
100A Moving object estimation system 1 Radio wave sensor 11 Conversion unit 12 Storage unit 13 Difference acquisition unit 14 Moving object estimation unit 2 Control device 2A Moving object estimation device 2B Equipment control device 21 Reception unit 22 Processing unit 221 Distance measurement unit 222 Point placement unit 223 Posture estimation unit 224 Behavior detection unit 225 Equipment control unit 226 Information acquisition unit 227 History accumulation unit 228 Learning unit 23 Output unit 200 Equipment 301 Human body 302 Stationary object 400 Room (real space)
500 Three-dimensional space (virtual space)
501 Point cloud 502 3D figure (rectangular parallelepiped)
CL, CL1, CL2 cluster group Tr transmitted wave Re reflected wave

Claims (10)

A moving object estimation system that estimates a moving object using a radio wave sensor,
a radio wave sensor that performs a transmission/reception operation of transmitting a transmission wave, which is a radio wave modulated by a predetermined method, toward a real space, receiving a reflected wave from the real space, and outputting an output signal based on the transmission wave and the reflected wave;
a conversion unit that performs a conversion process to convert the output signal output by the radio wave sensor into position identifiable information that can identify the position of each of one or more objects present in the real space;
a storage unit that performs a storage process to store the location identifiable information converted by the conversion process;
a difference acquisition unit that performs a difference acquisition process to acquire differences between the plurality of pieces of location identifiable information that are stored in the storage process;
a moving object estimation unit that performs a moving object estimation process of estimating whether each of the one or more objects is a moving object or a stationary object based on the difference acquired by the difference acquisition process, specifying a position of the object that is estimated to be a moving object, and acquiring moving object position information that indicates the specified position;
a point placement unit that performs a point placement process to virtually place points corresponding to the moving object position information acquired by the moving object estimation process in a virtual space corresponding to the real space,
the difference acquisition process acquires two or more differences with different time differences for three or more pieces of the location identifiable information at different times that are stored in the storage process,
the moving object estimation process acquires two or more pieces of moving object position information corresponding to the two or more differences acquired in the difference acquisition process,
the point arrangement process virtually arranges, in the virtual space, two or more points corresponding to two or more pieces of moving object position information with different time differences acquired in the moving object estimation process;
Motion estimation system. the moving object is either a human body or a moving object other than the human body,
the moving object estimation process further estimates whether each of the acquired two or more pieces of moving object position information corresponds to the human body or a moving object other than the human body, based on at least the change in the difference;
the point arrangement process virtually arranges, in the virtual space, points corresponding to the moving object position information that the moving object estimation process further estimates as corresponding to the human body, among the two or more pieces of moving object position information acquired by the moving object estimation process;
The moving object estimation system according to claim 1 . the moving object estimation process further estimates whether each of the acquired two or more pieces of moving object position information corresponds to the human body or a moving object other than the human body based on at least one of the reception intensity of the reflected wave corresponding to the moving object position information and a change in the reception intensity.
The moving object estimation system according to claim 2 . the moving object estimation process further estimates whether a point cloud, which is a collection of the plurality of points arranged in the virtual space, corresponds to the human body or a moving object other than the human body based on at least one of a distribution of the point cloud and a change in the distribution.
The moving object estimation system according to claim 2 or 3. the radio wave sensor performs the transmission and reception operation N times (N is an integer equal to or greater than 1) per frame, with a predetermined time being one frame, and outputs N pieces of the location identifiable information per frame;
the storing process stores 3×N or more pieces of the location identifiable information corresponding to three or more frames;
the difference acquisition process selects one representative piece of location identifiable information representing each of the three or more frames from 3×N or more pieces of location identifiable information corresponding to three or more frames held in the holding process, and acquires two or more differences for the selected three or more pieces of representative location identifiable information;
the moving object estimation process acquires two or more pieces of moving object position information based on two or more differences acquired for three or more pieces of representative position identifiable information representing the three or more frames, respectively;
The moving object estimation system according to claim 1 . the radio wave sensor performs the transmission and reception operation N times (N is an integer of 3 or more) per frame, with a predetermined time being one frame, and outputs N or more of the output signals per frame;
The storing process stores at least N pieces of the location identifiable information corresponding to one frame,
the difference acquisition process selects three or more pieces of representative location identifiable information representing the frame from one frame among the N or more pieces of location identifiable information corresponding to one or more frames held in the holding process, and acquires two or more of the differences for the selected three or more pieces of representative location identifiable information;
the moving object estimation process acquires two or more pieces of moving object position information based on two or more differences acquired for three or more pieces of representative position identifiable information representing one frame;
The moving object estimation system according to claim 1 . the predetermined method is an FMCW method,
the radio wave sensor has at least one antenna that receives the reflected wave;
the output signal is an IF signal generated based on the transmission wave and the reflected wave received by the one antenna for one transmission/reception operation,
the conversion processing is a Fourier transform processing of performing a Fourier transform on the IF signal,
the location identifiable information is a Fourier transform result of the Fourier transform processing,
the storing process stores a plurality of the Fourier transform results corresponding to the one antenna;
the moving object estimation process includes a ranging process of measuring a distance from the one antenna to the object estimated to be the moving object based on the difference between the plurality of Fourier transform results corresponding to the one antenna,
the moving object position information is one-dimensional position information based on a distance measurement result of the distance measurement process;
The moving object estimation system according to claim 1 . the radio wave sensor has a plurality of antennas for receiving the reflected waves, and outputs a plurality of the IF signals corresponding to the plurality of antennas in one transmission/reception operation;
for each of the plurality of antennas, the conversion unit performs the conversion process, the storage unit performs the storage process, the difference acquisition unit performs the difference acquisition process, and the moving object estimation unit performs the moving object estimation process including the ranging process,
the location identifiable information is a group of Fourier transform results consisting of a plurality of the Fourier transform results corresponding to the plurality of antennas,
The storage process stores a plurality of groups of the Fourier transform results,
the moving object estimation process includes a plurality of distance measurement processes for measuring a distance from each of the plurality of antennas to the object estimated to be the moving object based on the difference between the plurality of Fourier transform result groups for each of the plurality of antennas,
the moving object position information is two-dimensional or three-dimensional position information based on a plurality of the distance measurement results corresponding to a plurality of the distance measurement processes;
The moving object estimation system according to claim 7 . A moving object estimation method for performing estimation regarding a moving object, which is a moving object, using a radio wave sensor, comprising:
a transmitting/receiving step in which the radio wave sensor transmits a transmission wave, which is a radio wave modulated in a predetermined manner, toward a real space, receives a reflected wave from the real space, and outputs an output signal based on the transmission wave and the reflected wave;
a conversion step of performing a conversion process to convert the output signal output by the radio wave sensor into position identifiable information that can identify the position of each of one or more objects present in the real space;
a holding step of performing a holding process to hold the location identifiable information converted by the conversion process;
a difference acquisition step of performing a difference acquisition process of acquiring differences between the plurality of pieces of location identifiable information stored in the storage process;
a moving object estimation step of performing a moving object estimation process of estimating whether each of the one or more objects is a moving object or a non-moving object based on the difference acquired by the difference acquisition process, specifying the position of the object estimated to be a moving object, and acquiring moving object position information indicating the specified position;
a point arrangement step of performing a point arrangement process of virtually arranging points corresponding to the moving object position information acquired by the moving object estimation process in a virtual space corresponding to the real space,
the difference acquisition process acquires two or more differences with different time differences for three or more pieces of the location identifiable information at different times that are stored in the storage process,
the moving object estimation process acquires two or more pieces of moving object position information corresponding to the two or more differences acquired in the difference acquisition process,
the point arrangement process virtually arranges, in the virtual space, two or more points corresponding to two or more pieces of moving object position information with different time differences acquired in the moving object estimation process;
Motion estimation method. A program for causing one or more processors to execute the moving object estimation method according to claim 9.