JP7705253B2 - Electronic device, electronic device control method, and program - Google Patents

Description

This disclosure relates to an electronic device, a control method for an electronic device, and a program.

For example, devices have been proposed for monitoring the behavior of monitored individuals, such as those who require nursing care or long-term care, in settings such as nursing homes. For example, Patent Document 1 discloses a monitoring system that detects a specific behavior of a monitored individual based on images obtained by an imaging device. Patent Document 2 discloses a sensor that can estimate the position and posture of a living body using wireless signals. Patent Document 3 discloses a processing device that detects when a target changes posture from a sleeping position to an upright position based on fluctuations in the reception level of wireless radio waves.

JP 2017-91552 A JP 2018-8021 A JP 2019-220140 A

The applicant has conducted extensive research into electronic devices that monitor monitored subjects using images captured by an imaging unit such as a digital video camera, and has already filed several patent applications (e.g., Patent Application No. 2020-145012). Therefore, if it were possible to estimate the location of a monitored subject's body, such as the position of the head or legs, based on the results of detecting an object based on the reflected waves of a transmitted wave, as with radar, it would be beneficial for monitoring the monitored subject.

The objective of this disclosure is to provide an electronic device, a control method for an electronic device, and a program that are useful for monitoring a monitored subject.

The electronic device according to an embodiment includes:
Point cloud information obtained by a sensor detecting an object reflecting the transmission wave based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave of the transmission wave; and
and information on an image obtained by imaging the object using an imaging unit.
A predetermined portion of the object is machine-learned.
The point cloud information obtained by the detection by the sensor after recognizing a predetermined portion of the object is
If the maximum value of the coordinates of the extracted point group is Xmax, the minimum value of the extracted coordinates is Xmin, and the maximum value of the coordinates after normalization is X'max, this is information obtained by converting the coordinates (X) before normalization into the coordinates after normalization (X') using the following equation (1).
X'=((X-Xmin)/(Xmax-Xmin))・X'max (1)

Moreover, the electronic device according to an embodiment includes:
A predetermined portion of an object is machine-learned using point cloud information obtained by detecting an object reflecting the transmission wave based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave of the transmission wave,
Based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave of the transmission wave, a sensor detects a monitored object reflecting the transmission wave, and the information of the point cloud is used to obtain the detected point cloud information.
A predetermined portion of the monitored object is estimated.
The point cloud information obtained by the detection by the sensor after recognizing a predetermined portion of the object is
If the maximum value of the coordinates of the extracted point group is Xmax, the minimum value of the extracted coordinates is Xmin, and the maximum value of the coordinates after normalization is X'max, this is information obtained by converting the coordinates (X) before normalization into the coordinates after normalization (X') using the following equation (1).
X'=((X-Xmin)/(Xmax-Xmin))・X'max (1)

A method for controlling an electronic device according to an embodiment includes:
A step of acquiring point cloud information by detecting an object reflecting the transmission wave using a sensor based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave of the transmission wave;
acquiring image information by capturing an image of the object with an imaging unit;
A step of machine learning a predetermined portion of the object;
Includes.
The point cloud information obtained by the detection by the sensor after recognizing a predetermined portion of the object is
If the maximum value of the coordinates of the extracted point group is Xmax, the minimum value of the extracted coordinates is Xmin, and the maximum value of the coordinates after normalization is X'max, this is information obtained by converting the coordinates (X) before normalization into the coordinates after normalization (X') using the following equation (1).
X'=((X-Xmin)/(Xmax-Xmin))・X'max (1)

Moreover, a method for controlling an electronic device according to an embodiment includes:
A step of performing machine learning of a predetermined portion of an object using point cloud information obtained by detecting an object reflecting the transmission wave based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave of the transmission wave;
A step of estimating a predetermined part of the monitored object using point cloud information obtained by a sensor detecting the monitored object reflecting the transmission wave based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave of the transmission wave;
Includes.
The point cloud information obtained by the detection by the sensor after recognizing a predetermined portion of the object is
If the maximum value of the coordinates of the extracted point group is Xmax, the minimum value of the extracted coordinates is Xmin, and the maximum value of the coordinates after normalization is X'max, this is information obtained by converting the coordinates (X) before normalization into the coordinates after normalization (X') using the following equation (1).
X'=((X-Xmin)/(Xmax-Xmin))・X'max (1)

A program according to an embodiment includes:
For electronic devices,
A step of acquiring point cloud information by detecting an object reflecting the transmission wave using a sensor based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave of the transmission wave;
acquiring image information by capturing an image of the object with an imaging unit;
A step of machine learning a predetermined portion of the object;
Execute the command.
The point cloud information obtained by the detection by the sensor after recognizing a predetermined portion of the object is
If the maximum value of the coordinates of the extracted point group is Xmax, the minimum value of the extracted coordinates is Xmin, and the maximum value of the coordinates after normalization is X'max, this is information obtained by converting the coordinates (X) before normalization into the coordinates after normalization (X') using the following equation (1).
X'=((X-Xmin)/(Xmax-Xmin))・X'max (1)

Moreover, a program according to an embodiment includes:
For electronic devices,
A step of performing machine learning of a predetermined portion of an object using point cloud information obtained by detecting an object reflecting the transmission wave based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave of the transmission wave;
A step of estimating a predetermined part of the monitored object using point cloud information obtained by a sensor detecting the monitored object reflecting the transmission wave based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave of the transmission wave;
Execute the command.
The point cloud information obtained by the detection by the sensor after recognizing a predetermined portion of the object is
If the maximum value of the coordinates of the extracted point group is Xmax, the minimum value of the extracted coordinates is Xmin, and the maximum value of the coordinates after normalization is X'max, this is information obtained by converting the coordinates (X) before normalization into the coordinates after normalization (X') using the following equation (1).
X'=((X-Xmin)/(Xmax-Xmin))・X'max (1)

According to one embodiment, it is possible to provide an electronic device, a method for controlling an electronic device, and a program that are useful for monitoring a monitored subject.

1 is a block diagram showing a functional configuration of an electronic device according to an embodiment; FIG. 2 is a block diagram showing a functional configuration of a sensor according to an embodiment. FIG. 4 illustrates an example of a transmission signal transmitted by a sensor according to an embodiment. 10 is a flowchart illustrating an operation of an electronic device according to an embodiment. 11 is a diagram illustrating the coordinates of a predetermined part extracted by an electronic device according to an embodiment. FIG. 5 is a flowchart illustrating in more detail a portion of the operation shown in FIG. 4. FIG. 10 is a diagram illustrating an example of a process performed by an electronic device according to an embodiment. FIG. 10 is a diagram illustrating an example of a process performed by an electronic device according to an embodiment. FIG. 10 is a diagram illustrating an example of a process performed by an electronic device according to an embodiment. FIG. 10 is a diagram illustrating an example of a process performed by an electronic device according to an embodiment. FIG. 10 is a diagram illustrating an example of a process performed by an electronic device according to an embodiment. 11A and 11B are diagrams illustrating an example of point cloud information processed by an electronic device according to an embodiment. 11A and 11B are diagrams illustrating an example of point cloud information processed by an electronic device according to an embodiment. FIG. 1 is a diagram illustrating an example of the configuration of a detection device that supplies information for machine learning to an electronic device according to an embodiment. FIG. 1 is a diagram illustrating an example of the configuration of a detection device that supplies information for machine learning to an electronic device according to an embodiment. FIG. 1 is a diagram illustrating an example of machine learning performed by an electronic device according to an embodiment. 1A to 1C are diagrams illustrating an example of designation of a predetermined portion and image recognition by an electronic device according to an embodiment. 11A and 11B are diagrams illustrating an example of estimation of a predetermined part by an electronic device according to an embodiment.

In the present disclosure, an "electronic device" may be a device that is powered by electricity. Furthermore, a "system" may include a device that is powered by electricity. Furthermore, a "user" may be a person (typically a human) who uses a system and/or an electronic device according to an embodiment. A user may include a person who monitors a monitored subject by using a system and/or an electronic device according to an embodiment. Furthermore, a "monitored subject" may be a person (e.g., a human or an animal) that is monitored by a system and/or an electronic device according to an embodiment. Furthermore, a user may include a monitored subject.

The electronic device according to one embodiment is used in a specific facility used by people engaged in social activities, such as a company, a hospital, a nursing home, a school, a sports gym, and a care facility. For example, in a company, it is extremely important to understand and/or manage the health status of employees. Similarly, in a hospital, it is extremely important to understand and/or manage the health status of patients and medical staff, and in a nursing home, it is extremely important to understand and/or manage the health status of residents and staff. The electronic device according to one embodiment is used in a specific facility, not limited to the above-mentioned facilities such as a company, a hospital, and a nursing home, but may be any facility where it is desired to understand and/or manage the health status of a monitored subject. Any facility may include, for example, a non-commercial facility such as a user's home. The electronic device according to one embodiment may be used in a moving object such as a train, a bus, or an airplane, as well as in a station or a boarding area.

The electronic device according to one embodiment may be used, for example, in a care facility or the like, to monitor the behavior of a monitored subject, such as a person requiring nursing care or a care recipient. The electronic device according to one embodiment may monitor, for example, a motion of a monitored subject, such as a person requiring nursing care or a care recipient, sliding off a chair in which they are sitting. Here, the motion of a monitored subject sliding off a chair in which they are sitting may be, for example, a motion of a monitored subject, who is sitting on the seat of a chair, falling off the chair by partially sliding their body on the seat.

In particular, the electronic device according to one embodiment can issue a predetermined warning when a monitored person, such as a person requiring nursing care or care, is seated in a chair or the like, before the person stops sliding off the chair, for example, before the person falls onto the floor or the like. Therefore, according to the electronic device according to one embodiment, staff at a care facility or the like can recognize that the monitored person, such as a person requiring nursing care or care, is about to slide off the chair before the monitored person actually slides off the chair.

The electronic device according to one embodiment can also monitor the motion of a monitored subject, such as a person requiring nursing care or a care recipient, falling from a standing state. In particular, the electronic device according to one embodiment can issue a predetermined warning when a monitored subject, such as a person requiring nursing care or a care recipient, is standing and before the motion of falling is completed, for example, before the monitored subject falls onto the floor. Therefore, according to the electronic device according to one embodiment, staff at a care facility or the like can recognize that a monitored subject, such as a person requiring nursing care or a care recipient, is about to fall before the monitored subject actually falls.

The electronic device according to one embodiment can also monitor whether a monitored subject, such as a person requiring nursing care or a care recipient, falls into various dangerous situations. In particular, the electronic device according to one embodiment can issue a predetermined warning before a monitored subject, such as a person requiring nursing care or a care recipient, actually falls into a dangerous situation. The electronic device according to one embodiment allows staff at a care facility, for example, to recognize various dangers that may affect a monitored subject, such as a person requiring nursing care or a care recipient, at an earlier stage.

The electronic device according to one embodiment described below generates point cloud information that can be processed two-dimensionally, like an image, from point cloud information in three-dimensional space detected by a sensor based on technology such as millimeter wave radar. By employing technology such as image recognition, the electronic device according to one embodiment can recognize, for example, body parts of a monitored subject based on the point cloud information that can be processed two-dimensionally. By, for example, constantly monitoring the movement of the body parts of the monitored subject recognized in this way, the electronic device according to one embodiment can determine whether or not the monitored subject will fall into various dangerous situations. The electronic device according to one embodiment will be described in detail below with reference to the drawings.

1 is a functional block diagram showing a schematic configuration of an electronic device 1 according to an embodiment. As shown in FIG. 1, the electronic device 1 according to an embodiment includes a controller 10. In an embodiment, the electronic device 1 may include at least one of a memory unit 20, a communication unit 30, a display unit 40, and a notification unit 50, as appropriate. The controller 10, the memory unit 20, the communication unit 30, the display unit 40, and the notification unit 50 described above may be disposed or built into any location in the electronic device 1. In addition, at least one of the controller 10, the memory unit 20, the communication unit 30, the display unit 40, and the notification unit 50 described above may be disposed outside the electronic device 1. The electronic device 1 according to an embodiment may omit at least some of the functional units shown in FIG. 1, or may include other functional units other than the functional units shown in FIG. 1 as appropriate.

The electronic device 1 according to an embodiment may be various devices. For example, the electronic device according to an embodiment may be any device, such as a dedicated terminal, a general-purpose smartphone, a tablet, a phablet, a notebook computer (notebook PC), a computer, or a server. The electronic device according to an embodiment may have a function of communicating with other electronic devices, such as a mobile phone or a smartphone. Here, the above-mentioned "other electronic device" may be, for example, an electronic device such as a mobile phone or a smartphone, or may be any device, such as a base station, a server, a dedicated terminal, or a computer. The "other electronic device" may be, for example, a sensor 100 and/or an imaging unit 300 described later. The "other electronic device" in the present disclosure may also be a device or apparatus driven by power. When the electronic device according to an embodiment communicates with the other electronic device, the communication may be wired and/or wireless.

As shown in FIG. 1, the electronic device 1 according to an embodiment may be connected to the sensor 100 by wire and/or wirelessly. With such a connection, the electronic device 1 according to an embodiment can acquire information on the results detected by the sensor 100. Furthermore, the electronic device 1 according to an embodiment may be connected to the imaging unit 300 by wire and/or wirelessly. With such a connection, the electronic device 1 according to an embodiment can acquire information on the image captured by the imaging unit 300. The sensor 100 and the imaging unit 300 will be described further below.

The controller 10 controls and/or manages the entire electronic device 1, including each functional unit constituting the electronic device 1. The controller 10 may include at least one processor, such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor), to provide control and processing power for executing various functions. The controller 10 may be realized as a single processor, as a number of processors, or as individual processors. The processor may be realized as a single integrated circuit. An integrated circuit is also called an IC (Integrated Circuit). The processor may be realized as multiple communicatively connected integrated circuits and discrete circuits. The processor may be realized based on various other known technologies.

In one embodiment, the controller 10 may be configured as, for example, a CPU or DSP and a program executed by the CPU or DSP. The program executed in the controller 10 and the results of the processing executed in the controller 10 may be stored in, for example, the storage unit 20. The controller 10 may include memory necessary for the operation of the controller 10 as appropriate.

In the electronic device 1 according to one embodiment, the controller 10 can perform various processes on information output as a result of detection by the sensor 100, for example. For this reason, in the electronic device 1, the controller 10 may be connected to the sensor 100 by wire and/or wirelessly. Also, in the electronic device 1 according to one embodiment, the controller 10 can perform various processes on information (images) output as a result of imaging by the imaging unit 300, for example. For this reason, in the electronic device 1, the controller 10 may be connected to the imaging unit 300 by wire and/or wirelessly. The operation of the controller 10 of the electronic device 1 according to one embodiment will be described further below.

The storage unit 20 may function as a memory that stores various information. The storage unit 20 may store, for example, a program executed in the controller 10 and the results of processing executed in the controller 10. The storage unit 20 may store or accumulate the detection results by the sensor 100 and/or the images captured by the imaging unit 300. The storage unit 20 may also function as a work memory for the controller 10. The storage unit 20 may be configured, for example, by a semiconductor memory or the like, but is not limited thereto, and may be any storage device. For example, the storage unit 20 may be a storage medium such as a memory card inserted into the electronic device 1 according to one embodiment. The storage unit 20 may also be configured to include, for example, a hard disk drive (HDD) and/or a solid state drive (SSD). The storage unit 20 may also be an internal memory of a CPU used as the controller 10 described later, or may be connected to the controller 10 as a separate unit.

The storage unit 20 may store, for example, machine learning data. Here, the machine learning data may be data generated by machine learning. Furthermore, machine learning may be based on AI (Artificial Intelligence) technology that enables specific tasks to be performed through training. More specifically, machine learning may be technology in which an information processing device such as a computer learns a large amount of data and automatically constructs an algorithm or model that performs tasks such as classification and/or prediction. In this specification, machine learning may be included as part of AI (Artificial Intelligence).

In this specification, machine learning may include supervised learning, which learns the characteristics or rules of input data based on correct answer data. Machine learning may also include unsupervised learning, which learns the characteristics or rules of input data in the absence of correct answer data. Machine learning may also include reinforcement learning, which learns the characteristics or rules of input data by giving rewards or punishments. In this specification, machine learning may also be any combination of supervised learning, unsupervised learning, and reinforcement learning. The concept of machine learning data in this embodiment may include an algorithm that outputs a predetermined inference (estimation) result using an algorithm learned for input data. In this embodiment, as this algorithm, for example, linear regression that predicts the relationship between a dependent variable and an independent variable, a neural network (NN) that mathematically models the neurons of the human brain nervous system, a least squares method that calculates by squaring the error, a decision tree that solves problems in a tree structure, and regularization that transforms data in a predetermined manner, and other appropriate algorithms can be used. This embodiment may use deep learning, which is a type of neural network. Deep learning is a type of neural network, and neural networks with deep network hierarchies are called deep learning.

In the technology disclosed herein, it may be assumed that there is generally a certain relationship between the motion a of the body of the monitored subject and the motion result A of the monitored subject resulting from this motion a. The motion result here may include the motion of the monitored subject, the start time of the motion of the monitored subject, and accidents, incidents, and other events resulting from the motion of the monitored subject. For example, it is assumed that the body of the monitored subject performs motion a, and motion result A of the monitored subject results from this motion a. It is also assumed that the body of the monitored subject performs motion b, and motion result B of the monitored subject results from this motion b. The technology disclosed herein accumulates the relationship between the motion a and motion result A, motion b and motion result B, and other motions and motion results as machine learning data (e.g., in the storage unit 20). Then, when motion x is extracted, the technology disclosed herein may estimate motion result X related to motion x using the machine learning data.

In particular, in one embodiment, the machine learning data may be data obtained by machine learning of the movement of feature points when the monitored subject performs a predetermined motion. The machine learning data may also be data obtained by machine learning of the movement of feature points when a specific person (e.g., a specific person requiring care) as the monitored subject performs a predetermined motion.

The communication unit 30 has an interface function for wired or wireless communication. The communication method performed by the communication unit 30 in one embodiment may be a wireless communication standard. For example, the wireless communication standard includes cellular phone communication standards such as 2G, 3G, 4G, and 5G. For example, the cellular phone communication standards include LTE (Long Term Evolution), W-CDMA (Wideband Code Division Multiple Access), CDMA2000, PDC (Personal Digital Cellular), GSM (registered trademark) (Global System for Mobile communications), and PHS (Personal Handy-phone System). For example, the wireless communication standards include WiMAX (Worldwide Interoperability for Microwave Access), IEEE 802.11, WiFi, Bluetooth (registered trademark), IrDA (Infrared Data Association), and NFC (Near Field Communication). The communication unit 30 may include, for example, a modem whose communication method is standardized by the International Telecommunication Union Telecommunication Standardization Sector (ITU-T). The communication unit 30 can support one or more of the above communication standards.

The communication unit 30 may be configured to include, for example, an antenna for transmitting and receiving radio waves and an appropriate RF unit. The communication unit 30 may perform wireless communication with, for example, a communication unit of another electronic device via, for example, an antenna. The communication unit 30 may also be configured as an interface such as a connector for wired connection to the outside. The communication unit 30 may be configured using known technology for performing wireless communication, and therefore a more detailed description of the hardware, etc. will be omitted.

The various information received by the communication unit 30 may be supplied to, for example, the storage unit 20 and/or the controller 10. The various information received by the communication unit 30 may be stored in, for example, a memory built into the controller 10. The communication unit 30 may also transmit, for example, the results of processing by the controller 10 and/or information stored in the storage unit 20 to the outside.

The display unit 40 may be any display device, such as a liquid crystal display (LCD), an organic electro-luminescence panel, or an inorganic electro-luminescence panel. The display unit 40 may display various information such as characters, figures, or symbols. The display unit 40 may also display objects and icon images constituting various GUIs, for example, to prompt the user to operate the electronic device 1. Various data necessary for displaying on the display unit 40 may be supplied from, for example, the controller 10 or the storage unit 20. In addition, when the display unit 40 includes, for example, an LCD, it may be configured to include a backlight, etc., as appropriate. In one embodiment, the display unit 40 may display information based on the results of detection by the sensor 100, for example. In one embodiment, the display unit 40 may display information based on the results of imaging by the imaging unit 300, for example.

The notification unit 50 may issue a predetermined warning to alert the user of the electronic device 1 based on a predetermined signal output from the controller 10. The notification unit 50 may be any functional unit that stimulates at least one of the user's hearing, vision, and touch, such as sound, voice, light, text, video, and vibration as the predetermined warning. Specifically, the notification unit 50 may be at least one of a sound output unit such as a buzzer or speaker, a light-emitting unit such as an LED, a display unit such as an LCD, and a tactile sensation providing unit such as a vibrator. In this way, the notification unit 50 may issue a predetermined warning based on a predetermined signal output from the controller 10. In one embodiment, the notification unit 50 may issue a predetermined alarm as information that acts on at least one of the hearing, vision, and touch.

In one embodiment, the notification unit 50 may issue a warning that the monitored subject is at risk of slipping off the chair before the monitored subject starts to slip off the chair on which he/she is sitting. In one embodiment, the notification unit 50 may issue a warning that the monitored subject is at risk of slipping off the chair before the monitored subject finishes slipping off the chair on which he/she is sitting. For example, in one embodiment, the notification unit 50 that outputs visual information may warn the user of the risk of the monitored subject slipping off the chair by emitting light or a predetermined display when it detects that the monitored subject is at risk of slipping off the chair. In one embodiment, the notification unit 50 that outputs auditory information may warn the user of the risk of the monitored subject slipping off the chair by a predetermined sound or voice when it detects that the monitored subject is at risk of slipping off the chair. In this embodiment, the warning may be a combination of light emission or a predetermined display and a predetermined sound or voice.

The electronic device 1 shown in FIG. 1 has a built-in notification unit 50. However, in one embodiment, the notification unit 50 may be provided outside the electronic device 1. In this case, the notification unit 50 and the electronic device 1 may be connected by wire or wirelessly, or by a combination of wire and wireless.

As shown in FIG. 1, at least a portion of the functional units constituting the electronic device 1 according to one embodiment may be constituted by specific means in which software and hardware resources work together.

The sensor 100 shown in FIG. 1 is configured to detect a human body, such as a monitored object, as point cloud information in a three-dimensional space. The sensor 100 according to one embodiment will be described in more detail below.

FIG. 2 is a functional block diagram that shows a schematic configuration of a sensor 100 according to an embodiment. The sensor 100 shown in FIG. 2 is, as an example, a millimeter wave radar sensor based on millimeter wave radar (RADAR (Radio Detecting and Ranging)) technology. However, the sensor 100 according to an embodiment is not limited to a millimeter wave radar sensor. The sensor 100 according to an embodiment may be, for example, a quasi-millimeter wave radar sensor. The sensor 100 according to an embodiment is not limited to a millimeter wave radar sensor or a quasi-millimeter wave radar sensor, and may be any type of radar sensor that transmits and receives radio waves. The sensor 100 according to an embodiment may be, for example, a microwave sensor, an ultrasonic sensor, or a sensor based on technology such as LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging).

When measuring distances and the like using a millimeter wave radar, a frequency modulated continuous wave radar (hereinafter, FMCW radar) may be used. In an FMCW radar, the frequency of the radio waves to be transmitted is swept to generate a transmission signal. Therefore, in a millimeter wave FMCW radar using radio waves in the 79 GHz frequency band, for example, the frequency of the radio waves used has a frequency bandwidth of 4 GHz, such as 77 GHz to 81 GHz. A radar in the 79 GHz frequency band has a wider usable frequency bandwidth than other millimeter wave/quasi-millimeter wave radars, such as those in the 24 GHz, 60 GHz, and 76 GHz frequency bands. Below, as an example, a case where such an FMCW radar is used will be described. The FMCW radar system used in this disclosure may include the FCM (Fast-Chirp Modulation) system, which transmits a chirp signal at a shorter period than normal. The signal generated by the sensor 100 is not limited to an FMCW signal. The signal generated by the sensor 100 may be a signal of various types other than the FMCW type. The transmission signal sequence stored as the signal to be transmitted may differ depending on the various types. For example, in the case of the radar signal of the FMCW type described above, a signal whose frequency increases and decreases for each time sample may be used. Since the various types described above can be appropriately applied using publicly known technologies, a more detailed description will be omitted as appropriate.

As shown in FIG. 2, the sensor 100 according to one embodiment may include a radar control unit 110, a transmission unit 120, and a reception unit 130. The above-mentioned radar control unit 110, transmission unit 120, reception unit 130, etc. may be disposed or built into any location in the sensor 100. In addition, at least one of the above-mentioned radar control unit 110, transmission unit 120, reception unit 130, etc. may be disposed outside the sensor 100. The sensor 100 according to one embodiment may omit at least some of the functional units shown in FIG. 2, or may include other functional units as appropriate in addition to the functional units shown in FIG. 2.

The radar control unit 110 controls and/or manages the entire sensor 100, including each functional unit constituting the sensor 100. The radar control unit 110 may include at least one processor, such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor), to provide control and processing power for executing various functions. The radar control unit 110 may be realized as a single processor, as a number of processors, or as individual processors. The processor may be realized as a single integrated circuit. An integrated circuit is also called an IC (Integrated Circuit). The processor may be realized as multiple communicatively connected integrated circuits and discrete circuits. The processor may be realized based on various other known technologies.

In one embodiment, the radar control unit 110 may be configured as, for example, a CPU or DSP and a program executed by the CPU or DSP. The program executed in the radar control unit 110 and the results of the processing executed in the radar control unit 110 may be stored in, for example, any storage unit built into the radar control unit 110. The radar control unit 110 may include memory necessary for the operation of the radar control unit 110 as appropriate.

In the sensor 100 according to one embodiment, the radar control unit 110 may appropriately perform various processes such as distance FFT (Fast Fourier Transform) processing, speed FFT processing, arrival angle estimation processing, and clustering processing. Each of these processes performed by the radar control unit 110 is known as general radar technology, and therefore a detailed description is omitted.

2, the transmitting unit 120 may include a signal generating unit 121, a synthesizer 122, a phase control unit 123, an amplifier 124, and a transmitting antenna 125. The sensor 100 according to one embodiment may include a plurality of transmitting antennas 125. In this case, the sensor 100 may also include a plurality of phase control units 123 and amplifiers 124 corresponding to each of the plurality of transmitting antennas 125.

As shown in FIG. 2, the receiving unit 130 may include a receiving antenna 131, an LNA 132, a mixer 133, an IF unit 134, and an AD conversion unit 135. The sensor 100 according to one embodiment may include multiple receiving units 130 corresponding to the multiple transmitting antennas 125.

In the sensor 100 according to one embodiment, the radar control unit 110 can control at least one of the transmitting unit 120 and the receiving unit 130. In this case, the radar control unit 110 may control at least one of the transmitting unit 120 and the receiving unit 130 based on various information stored in an arbitrary storage unit. For example, an arbitrary storage unit built into the radar control unit 110 may store various parameters for setting the range in which an object is detected by the transmission wave transmitted from the transmitting antenna 125 and the reflected wave received from the receiving antenna 131. In addition, in the sensor 100 according to one embodiment, the radar control unit 110 may instruct the signal generating unit 121 to generate a signal, or control the signal generating unit 121 to generate a signal.

The signal generating unit 121 generates a signal (transmission signal) to be transmitted as a transmission wave from the transmitting antenna 125 under the control of the radar control unit 110. When generating the transmission signal, the signal generating unit 121 may assign the frequency of the transmission signal, for example, based on the control by the radar control unit 110. Specifically, the signal generating unit 121 may assign the frequency of the transmission signal, for example, according to parameters set by the radar control unit 110. For example, the signal generating unit 121 receives frequency information from the radar control unit 110 or an arbitrary storage unit, and generates a signal of a predetermined frequency in a frequency band such as 77 to 81 GHz. The signal generating unit 121 may be configured to include a functional unit such as a voltage controlled oscillator (VCO).

The signal generating unit 121 may be configured as hardware having the relevant function, or may be configured as, for example, a microcomputer, or may be configured as, for example, a processor such as a CPU or DSP and a program executed by the processor. Each of the functional units described below may also be configured as hardware having the relevant function, or, if possible, may be configured as, for example, a microcomputer, or may be configured as, for example, a processor such as a CPU or DSP and a program executed by the processor.

In the sensor 100 according to one embodiment, the signal generating unit 121 may generate a transmission signal (transmission chirp signal) such as a chirp signal. In particular, the signal generating unit 121 may generate a signal (linear chirp signal) whose frequency changes periodically and linearly. For example, the signal generating unit 121 may generate a chirp signal whose frequency increases periodically and linearly from 77 GHz to 81 GHz over time. Also, for example, the signal generating unit 121 may generate a signal whose frequency periodically repeats a linear increase (up chirp) and decrease (down chirp) from 77 GHz to 81 GHz over time. The signal generated by the signal generating unit 121 may be set in advance in, for example, the radar control unit 110. Also, the signal generated by the signal generating unit 121 may be stored in advance in, for example, an arbitrary storage unit. Since chirp signals used in technical fields such as radar are known, a more detailed description will be simplified or omitted as appropriate. The signal generated by the signal generating unit 121 is supplied to the synthesizer 122.

Figure 3 is a diagram illustrating an example of a chirp signal generated by the signal generating unit 121.

In FIG. 3, the horizontal axis represents the elapsed time, and the vertical axis represents the frequency. In the example shown in FIG. 3, the signal generating unit 121 generates a linear chirp signal whose frequency changes periodically and linearly. In FIG. 3, each chirp signal is shown as c1, c2, ..., c8. As shown in FIG. 3, in each chirp signal, the frequency increases linearly with the passage of time.

In the example shown in FIG. 3, eight chirp signals such as c1, c2, ..., c8 are included in one subframe. That is, subframe 1 and subframe 2 shown in FIG. 3 are each composed of eight chirp signals such as c1, c2, ..., c8. Also, in the example shown in FIG. 3, 16 subframes such as subframe 1 to subframe 16 are included in one frame. That is, frame 1 and frame 2 shown in FIG. 3 are each composed of 16 subframes. Also, as shown in FIG. 3, a frame interval of a predetermined length may be included between frames. One frame shown in FIG. 3 may be, for example, about 30 to 50 milliseconds long.

In FIG. 3, the same configuration may be used for frames 2 and onward. Also, in FIG. 3, the same configuration may be used for frames 3 and onward. In the sensor 100 according to one embodiment, the signal generating unit 121 may generate a transmission signal as any number of frames. Also, in FIG. 3, some chirp signals are omitted. In this way, the relationship between time and frequency of the transmission signal generated by the signal generating unit 121 may be stored in, for example, any storage unit.

In this way, the sensor 100 according to one embodiment may transmit a transmission signal consisting of subframes including multiple chirp signals. Also, the sensor 100 according to one embodiment may transmit a transmission signal consisting of a frame including a predetermined number of subframes.

Hereinafter, the sensor 100 will be described as transmitting a transmission signal having a frame structure as shown in FIG. 3. However, the frame structure as shown in FIG. 3 is an example, and the number of chirp signals included in one subframe is not limited to eight. In one embodiment, the signal generating unit 121 may generate a subframe including any number of chirp signals (for example, any multiple). The subframe structure as shown in FIG. 3 is also an example, and the number of subframes included in one frame is not limited to 16. In one embodiment, the signal generating unit 121 may generate a frame including any number of subframes (for example, any multiple). The signal generating unit 121 may generate signals of different frequencies. The signal generating unit 121 may generate multiple discrete signals having different bandwidths, each with a frequency f.

Returning to FIG. 2, the synthesizer 122 increases the frequency of the signal generated by the signal generating unit 121 to a frequency in a predetermined frequency band. The synthesizer 122 may increase the frequency of the signal generated by the signal generating unit 121 to a frequency selected as the frequency of the transmission wave to be transmitted from the transmitting antenna 125. The frequency selected as the frequency of the transmission wave to be transmitted from the transmitting antenna 125 may be set by, for example, the radar control unit 110. In addition, the frequency selected as the frequency of the transmission wave to be transmitted from the transmitting antenna 125 may be stored in, for example, an arbitrary storage unit. The signal whose frequency has been increased by the synthesizer 122 is supplied to the phase control unit 123 and the mixer 133. When there are multiple phase control units 123, the signal whose frequency has been increased by the synthesizer 122 may be supplied to each of the multiple phase control units 123. Furthermore, if there are multiple receiving units 130, the signal whose frequency has been increased by the synthesizer 122 may be supplied to each of the mixers 133 in the multiple receiving units 130.

The phase control unit 123 controls the phase of the transmission signal supplied from the synthesizer 122. Specifically, the phase control unit 123 may adjust the phase of the transmission signal by appropriately advancing or delaying the phase of the signal supplied from the synthesizer 122 based on, for example, the control by the radar control unit 110. In this case, the phase control unit 123 may adjust the phase of each transmission signal based on the path difference of each transmission wave transmitted from the multiple transmission antennas 125. By the phase control unit 123 appropriately adjusting the phase of each transmission signal, the transmission waves transmitted from the multiple transmission antennas 125 reinforce each other in a predetermined direction to form a beam (beamforming). In this case, the correlation between the direction of beamforming and the phase amount to be controlled of the transmission signal transmitted by each of the multiple transmission antennas 125 may be stored in, for example, an arbitrary storage unit. The transmission signal phase-controlled by the phase control unit 123 is supplied to the amplifier 124.

The amplifier 124 amplifies the power of the transmission signal supplied from the phase control unit 123, for example, based on the control by the radar control unit 110. If the sensor 100 has multiple transmission antennas 125, the multiple amplifiers 124 may each amplify the power of the transmission signal supplied from a corresponding one of the multiple phase control units 123, for example, based on the control by the radar control unit 110. Since the technology itself for amplifying the power of the transmission signal is already known, a more detailed description will be omitted. The amplifier 124 is connected to the transmission antenna 125.

The transmitting antenna 125 outputs (transmits) the transmission signal amplified by the amplifier 124 as a transmission wave. When the sensor 100 includes multiple transmitting antennas 125, the multiple transmitting antennas 125 may each output (transmit) the transmission signal amplified by a corresponding one of the multiple amplifiers 124 as a transmission wave. The transmitting antenna 125 can be configured in the same manner as a transmitting antenna used in known radar technology, and therefore a detailed description thereof will be omitted.

In this way, the sensor 100 according to one embodiment includes a transmitting antenna 125, and can transmit a transmission signal (e.g., a transmitting chirp signal) as a transmission wave from the transmitting antenna 125. Here, at least one of the functional units constituting the sensor 100 may be housed in a single housing. In addition, in this case, the single housing may have a structure that makes it difficult to open. For example, the transmitting antenna 125, the receiving antenna 131, and the amplifier 124 may be housed in a single housing, and this housing may have a structure that makes it difficult to open. Furthermore,

2 shows an example in which the sensor 100 has one transmitting antenna 125. However, in one embodiment, the sensor 100 may have any number of transmitting antennas 125. On the other hand, in one embodiment, the sensor 100 may have multiple transmitting antennas 125 when the transmission wave transmitted from the transmitting antenna 125 forms a beam in a predetermined direction. In one embodiment, the sensor 100 may have any number of transmitting antennas 125. In this case, the sensor 100 may also have multiple phase control units 123 and amplifiers 124 corresponding to the multiple transmitting antennas 125. The multiple phase control units 123 may each control the phase of the multiple transmitting waves supplied from the synthesizer 122 and transmitted from the multiple transmitting antennas 125. The multiple amplifiers 124 may each amplify the power of the multiple transmitting signals transmitted from the multiple transmitting antennas 125. In this case, the sensor 100 may be configured to include multiple transmitting antennas. In this way, when the sensor 100 has multiple transmission antennas 125, each of the multiple transmission antennas 125 may also be configured to include multiple functional units required to transmit transmission waves.

The receiving antenna 131 receives the reflected wave. The reflected wave may be a transmitted wave reflected by a predetermined object 200. The receiving antenna 131 may be configured to include multiple antennas. The receiving antenna 131 may be configured in the same manner as a receiving antenna used in known radar technology, and therefore a detailed description is omitted. The receiving antenna 131 is connected to the LNA 132. A received signal based on the reflected wave received by the receiving antenna 131 is supplied to the LNA 132.

The sensor 100 according to one embodiment can receive a reflected wave that is a result of a transmission wave transmitted as a transmission signal (transmission chirp signal), such as a chirp signal, from a plurality of receiving antennas 131 and reflected by a specific object 200. In this way, when a transmission chirp signal is transmitted as a transmission wave, a reception signal based on the received reflected wave is referred to as a reception chirp signal. In other words, the sensor 100 receives a reception signal (e.g., a reception chirp signal) as a reflected wave from the receiving antenna 131.

The LNA 132 amplifies, with low noise, the received signal based on the reflected wave received by the receiving antenna 131. The LNA 132 may be a low noise amplifier, and amplifies, with low noise, the received signal supplied from the receiving antenna 131. The received signal amplified by the LNA 132 is supplied to the mixer 133.

The mixer 133 generates a beat signal by mixing (multiplying) the RF frequency reception signal supplied from the LNA 132 with the transmission signal supplied from the synthesizer 122. The beat signal mixed by the mixer 133 is supplied to the IF unit 134.

The IF unit 134 performs frequency conversion on the beat signal supplied from the mixer 133, thereby lowering the frequency of the beat signal to an intermediate frequency (IF). The beat signal whose frequency has been lowered by the IF unit 134 is supplied to the AD conversion unit 135.

The AD conversion unit 135 digitizes the analog beat signal supplied from the IF unit 134. The AD conversion unit 135 may be configured with any analog-to-digital conversion circuit (Analog to Digital Converter (ADC)). The beat signal digitized by the AD conversion unit 135 is supplied to the radar control unit 110. When there are multiple receiving units 130, each of the beat signals digitized by the multiple AD conversion units 135 may be supplied to the radar control unit 110.

The radar control unit 110 may perform FFT processing (hereinafter, appropriately referred to as "distance FFT processing") on the beat signal digitized by the AD conversion unit 135. For example, the radar control unit 110 may perform FFT processing on the complex signal supplied from the AD conversion unit 135. The beat signal digitized by the AD conversion unit 135 can be expressed as a time change in signal strength (power). By performing FFT processing on such a beat signal, the radar control unit 110 can express it as a signal strength (power) corresponding to each frequency. By performing distance FFT processing in the radar control unit 110, a complex signal corresponding to distance can be obtained based on the beat signal digitized by the AD conversion unit 135.

When a peak is equal to or greater than a predetermined threshold in the results obtained by distance FFT processing, the radar control unit 110 may determine that a predetermined object 200 is present at the distance corresponding to the peak. For example, a method is known in which, when a peak value equal to or greater than a threshold is detected from the average power or amplitude of a disturbance signal, such as in a detection process based on a constant false alarm rate (CFAR), an object that reflects the transmitted wave (a reflecting object) is present.

In this way, the sensor 100 according to one embodiment can detect an object 200 reflecting a transmission wave as a target based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave.

The radar control unit 110 can estimate the distance to a predetermined object based on one chirp signal (e.g., c1 shown in FIG. 3). That is, the sensor 100 can measure (estimate) the distance between the sensor 100 and a predetermined object 200 by performing distance FFT processing. The technology itself for measuring (estimating) the distance to a predetermined object by performing FFT processing on a beat signal is well known, so a more detailed explanation will be simplified or omitted as appropriate.

The radar control unit 110 may further perform FFT processing (hereinafter referred to as "velocity FFT processing" as appropriate) on the beat signal that has been subjected to distance FFT processing. For example, the radar control unit 110 may perform FFT processing on the complex signal that has been subjected to distance FFT processing. The radar control unit 110 can estimate the relative velocity with respect to a specified object based on a subframe of the chirp signal (e.g., subframe 1 shown in FIG. 3). By performing velocity FFT processing on multiple chirp signals in the radar control unit 110, a complex signal corresponding to the relative velocity can be obtained based on the complex signal corresponding to the distance obtained by the distance FFT processing.

By performing distance FFT processing on the beat signal as described above, multiple vectors can be generated. By determining the phase of the peak in the result of performing velocity FFT processing on these multiple vectors, the relative velocity with respect to a specified object can be estimated. In other words, by performing velocity FFT processing, the electronic device 1 can measure (estimate) the relative velocity between the sensor 100 and the specified object 200. Since the technology itself of measuring (estimating) the relative velocity with respect to a specified object by performing velocity FFT processing on the result of distance FFT processing is well known, a more detailed description will be simplified or omitted as appropriate.

In general FMCW radar technology, the presence or absence of a target can be determined based on the results of performing fast Fourier transform processing on beat frequencies extracted from a received signal. Here, the results of extracting beat frequencies from a received signal and performing fast Fourier transform processing also contain noise components due to clutter (unwanted reflected components). Therefore, processing may be performed to remove noise components from the results of processing the received signal and extract only the target signal.

The radar control unit 110 may also estimate the direction (arrival angle) in which the reflected wave arrives from a specific object 200 based on the determination of whether or not a target exists. The radar control unit 110 may estimate the arrival angle for a point where it is determined that a target exists. The sensor 100 can estimate the direction in which the reflected wave arrives by receiving reflected waves from multiple receiving antennas 131. For example, the multiple receiving antennas 131 are assumed to be arranged at a predetermined interval. In this case, the transmission wave transmitted from the transmission antenna 125 is reflected by the specific object 200 to become a reflected wave, and the multiple receiving antennas 131 arranged at a predetermined interval each receive the reflected wave R. Then, the radar control unit 110 can estimate the direction in which the reflected wave arrives at the receiving antenna 131 based on the phase of the reflected wave received by each of the multiple receiving antennas 131 and the path difference of each reflected wave. That is, the sensor 100 can measure (estimate) the arrival angle θ indicating the direction in which the reflected wave reflected by the target arrives based on the result of the speed FFT processing.

Various techniques have been proposed for estimating the direction of arrival of the reflected wave R based on the results of velocity FFT processing. For example, known algorithms for estimating the direction of arrival include MUSIC (MUltiple SIgnal Classification) and ESPRIT (Estimation of Signal Parameters via Rotational Invariance Technique). Therefore, detailed descriptions of known techniques will be simplified or omitted as appropriate.

The radar control unit 110 detects an object present within the range where the transmission wave is transmitted based on at least one of distance FFT processing, speed FFT processing, and arrival angle estimation. The radar control unit 110 may perform object detection by, for example, performing clustering processing based on the supplied distance information, speed information, and angle information. Known algorithms used when clustering data include, for example, DBSCAN (Density-based spatial clustering of applications with noise). In the clustering processing, for example, the average power of the points that make up the detected object may be calculated.

As described above, the sensor 100 can detect an object reflecting the transmission wave in three-dimensional space as point cloud information. That is, in one embodiment, based on the detection result output from the sensor 100, it can determine (detect) whether or not an object reflecting the transmission wave exists at a certain coordinate in three-dimensional space. Also, in one embodiment, the sensor 100 can detect the signal strength and speed of each point in three-dimensional space. As described above, the sensor 100 according to one embodiment may detect an object reflecting the transmission wave as point cloud information in three-dimensional space based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave of the transmission wave.

As shown in FIG. 1, the electronic device 1 according to an embodiment may be configured to include an imaging unit 300. The electronic device 1 and the imaging unit 300 may be connected by wire or wirelessly, or by a combination of wire and wireless.

The imaging unit 300 may be configured to include an image sensor that electronically captures an image, such as a digital camera. The imaging unit 300 may be configured to include an imaging element that performs photoelectric conversion, such as a CCD (Charge Coupled Device Image Sensor) or a CMOS (Complementary Metal Oxide Semiconductor) sensor. The imaging unit 300 may capture an image of an object to be monitored, for example. Here, the monitored object may be, for example, a human being. The imaging unit 300 may convert the captured image into a signal and transmit it to the electronic device 1. For example, the imaging unit 300 may transmit a signal based on the captured image to the extraction unit 11, the storage unit 20, and/or the controller 10 of the electronic device 1. The imaging unit 300 is not limited to an imaging device such as a digital camera, and may be any device that captures an image of the monitored object.

In one embodiment, the imaging unit 300 may capture images of the monitored object as still images at predetermined intervals (e.g., 15 frames per second). In another embodiment, the imaging unit 300 may capture images of the monitored object as a continuous video.

Next, the operation of the electronic device 1 according to one embodiment will be described.

The operation of the electronic device 1 according to one embodiment can typically be divided into a "learning phase" and an "estimation phase." In the learning phase, when a human being, such as a monitored subject, performs a predetermined motion, an operation may be performed to machine-learn the relationship between the positions (coordinates) of each body part during the motion and the timing of the motion. In the estimation phase, an operation may be performed to estimate the start of the predetermined motion from the positions (coordinates) of each body part during the motion of the monitored subject, based on the results of the machine learning in the learning phase. Furthermore, in the estimation phase, when it is determined that a predetermined or higher risk exists, for example, when the motion described above is started, a predetermined alarm or the like may be output.

Furthermore, the operation of the electronic device 1 according to one embodiment may determine the presence or absence of risk according to a predetermined algorithm, rather than being divided into a "learning phase" and an "estimation phase." For example, in the electronic device 1 according to one embodiment, the factors of risk that may affect the monitored subject may be defined in advance as an algorithm. In this case, the electronic device 1 according to one embodiment may output a predetermined warning or the like when it is determined according to the predetermined algorithm that the risk is greater than or equal to a predetermined level.

Figure 4 is a flowchart showing an example of the operation performed by the electronic device 1 according to an embodiment. The operation shown in Figure 4 may be started, for example, when the electronic device 1 according to an embodiment performs an operation of monitoring a monitored object. In the following, a description will be given assuming that the monitored object monitored by the electronic device 1 is a human. However, the monitored object monitored by the electronic device 1 is not limited to a human, and may be an animal other than a human. Furthermore, the monitored object is not limited to a human or an animal, and may be a machine such as a work robot. The operation shown in Figure 4 may be based on a process executed by the controller 10 of the electronic device 1. That is, in the following description, the operation and/or process performed by the "electronic device 1" may be, for example, the operation and/or process executed by the controller 10 of the electronic device 1.

4 starts, the electronic device 1 acquires point cloud information output by the sensor 100 (step S1). Here, the point cloud information output by the sensor 100 may be information detected by the sensor 100. In other words, the information detected by the sensor 100 may be information in which an object reflecting a transmission wave is detected as point cloud information in a three-dimensional space based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave of the transmission wave.

Once the point cloud information is acquired in step S1, the electronic device 1 performs preprocessing of the acquired point cloud information (step S2). The preprocessing of the point cloud information performed in step S2 may involve generating point cloud information that can be processed two-dimensionally from the point cloud information in three-dimensional space. In other words, once the point cloud information is preprocessed in step S2, the point cloud information becomes information that can be processed two-dimensionally, and can be processed in the same way as image data, for example by performing image recognition. The preprocessing of the point cloud information performed in step S2 will be described further below.

After the pre-processing of the point cloud information is performed in step S2, the electronic device 1 recognizes, for example, a body part of the monitored subject based on the two-dimensionally processable point cloud information (step S3). In step S3, the electronic device 1 may recognize the body part of the monitored subject based on the two-dimensionally processable point cloud information by adopting a technology such as image recognition. In addition, in step S3, the electronic device 1 may extract, for example, the coordinates of the image-recognized predetermined part. As described above, by performing the processing in step S2, the point cloud information in the three-dimensional space becomes two-dimensionally processable point cloud information. Therefore, in step S3, the electronic device 1 can process the point cloud information as if it were an image. In step S3, the electronic device 1 may correspond each point of the two-dimensionally processable point cloud to a pixel in the image. In step S3, the electronic device 1 may determine each body part of the monitored subject, such as the head, neck, shoulders, arms, hands, torso, or legs, by image recognition based on the two-dimensionally processable point cloud information.

Once the body part is recognized in step S3, the electronic device 1 normalizes the coordinates of the body part (step S4). By normalizing the coordinates of the body part in step S4, the electronic device 1 can process point cloud information that can be processed two-dimensionally without being affected by differences due to, for example, the angle of view. By normalizing the coordinates of the body part in step S4, data for predicting the behavior of the monitored subject can be generated (step S5).

Here, we will explain the normalization of the coordinates of the body part performed in step S4 described above. It is expected that the coordinates of the specific body part extracted in step S3 will vary due to, for example, the body size of the monitored subject. It is also expected that the coordinates of the specific body part extracted in step S3 will vary due to, for example, the distance between the sensor 100 and the monitored subject, and the direction from the sensor 100 to the monitored subject. Therefore, in one embodiment, by normalizing the X-direction component and the Y-direction component of the coordinates extracted in step S3, respectively, the extracted coordinates can be used generically for machine learning.

In this case, the extracted X and Y coordinates may be normalized based on the respective maximum and minimum values of the X and Y coordinates extracted in, for example, 15 frames per second. Here, the maximum value of the X coordinate extracted in step S3 is Xmax, and the minimum value of the X coordinate extracted in step S3 is Xmin. Furthermore, the maximum value of the X coordinate after normalization is X'max. In this case, the X coordinate (X) before normalization can be converted to the X coordinate (X') after normalization using the following formula (1).

X'=((X-Xmin)/(Xmax-Xmin))・X'max (1)

Similarly, the maximum value of the Y coordinate extracted in step S3 is defined as Ymax, and the minimum value of the Y coordinate extracted in step S3 is defined as Ymin. Furthermore, the maximum value of the Y coordinate after normalization is defined as Y'max. In this case, the Y coordinate before normalization (Y) can be converted to the Y coordinate after normalization (Y') using the following equation (2).

Y'=((Y-Ymin)/(Ymax-Ymin))・Y'max (2)

By normalizing the X-direction and Y-direction components of the extracted coordinates according to the above formulas (1) and (2), it is expected that the influence on machine learning caused by individual differences in the monitored object and the environment in which the sensor 100 captures the monitored object can be reduced.

Thus, in one embodiment, the controller 10 may normalize each directional component of the coordinates of a predetermined number of joint points in the body of the monitored subject, which are extracted two-dimensionally, based on the maximum and minimum values of each directional component.

By normalizing the coordinates in step S4, each coordinate (X, Y) shown in FIG. 5 is normalized to the coordinate (X', Y').

Next, the data generation in step S5 will be further explained. As described above, data generation in step S5 may be understood to mean generating data for predicting the behavior of the monitored subject. When the sensor 100 captures images of a predetermined number of frames per second, the controller 10 may extract the predetermined number of frames per second as a predetermined part of the monitored subject's body. Also, when the controller 10 acquires images of a predetermined number of frames per second, the controller 10 may extract the predetermined number of frames per second as a predetermined part of the monitored subject's body. As an example, the controller 10 may extract the predetermined part of the monitored subject's body at 15 frames per second.

5 is a diagram showing the coordinates of the specific parts extracted from the body of the monitored subject in, for example, 15 frames per second. As shown in FIG. 5, in step S5, the controller 10 may arrange the coordinates of the specific parts extracted from the body of the monitored subject in each frame. As shown in FIG. 5, the controller 10 may extract the coordinates of the specific parts two-dimensionally (as X and Y coordinates) from the body of the monitored subject in each frame. In the table shown in FIG. 5, each row shows a schematic diagram of the specific parts of the body of the monitored subject extracted as X and Y coordinates in each frame. In addition, in the table shown in FIG. 5, the rows showing each frame are shown from top to bottom according to the passage of time. The coordinates of the 15 frames shown in FIG. 5 may be, for example, coordinates tracked for one second in an image (or video). In addition, after the 15 frames shown in FIG. 5, the coordinates of the specific parts of the body of the monitored subject may be extracted sequentially.

Thus, in one embodiment, the controller 10 may extract two-dimensional coordinates of a predetermined number of joint points in the body of the monitored subject for each image captured by the sensor 100 for a predetermined number of frames per second.

Once the data is generated in step S5, the electronic device 1 predicts the behavior of the monitored subject (step S6). Various methods can be envisioned for predicting the behavior of the monitored subject in step S6, such as using machine learning, or simply using an algorithm without using machine learning. In this disclosure, the method for predicting the behavior of the monitored subject is not a primary feature, and therefore will not be described in detail.

Once the behavior of the monitored subject is predicted in step S5, the electronic device 1 determines whether or not the behavior of the monitored subject poses a predetermined or greater risk (step S7). When performing the process of step S7, the electronic device 1 may set risk standards for various behaviors of the monitored subject. The electronic device 1 may then determine whether or not the behavior of the monitored subject exceeds the risk standard (i.e., whether or not there is a predetermined or greater risk).

If there is no risk equal to or greater than the predetermined level in step S7, the electronic device 1 may determine that there is no need to notify the user of the risk and may terminate the operation shown in FIG. 4. On the other hand, if there is a risk equal to or greater than the predetermined level in step S7, the electronic device 1 issues a predetermined warning from the notification unit 50 (step S8).

In this way, the electronic device 1 according to one embodiment can monitor whether the monitored person falls into various dangerous situations. In particular, the electronic device 1 according to one embodiment can issue a predetermined warning before the monitored person actually falls into a dangerous situation. The electronic device 1 according to one embodiment can recognize various dangers that may affect the monitored person at an earlier stage.

Next, we will further explain the preprocessing of point cloud information performed in step S2 of Figure 4.

Figure 6 is a flowchart showing in more detail the pre-processing of point cloud information performed by electronic device 1 in step S2 of Figure 4. The operations shown in Figure 6 may be based on processing executed by controller 10 of electronic device 1. That is, in the following description, operations and/or processing performed by "electronic device 1" may be operations and/or processing executed by, for example, controller 10 of electronic device 1.

6 starts, the electronic device 1 reduces the dimension of the point cloud information (step S11). More specifically, in step S11, the electronic device 1 generates point cloud information that can be processed two-dimensionally from the point cloud information in three-dimensional space output by the sensor 100.

Figure 7 is a diagram explaining point cloud information in a three-dimensional space output by the sensor 100. Figure 7 is a diagram showing an example of a situation in which a monitored object Tm standing at a certain location is three-dimensionally detected by the sensor 100 installed at the origin O.

As shown in FIG. 7, the position where the sensor 100 is installed is taken as the reference (origin O), the direction approaching the monitored object Tm is taken as the positive X-axis direction, and the direction moving away from the monitored object Tm is taken as the negative X-axis direction. Also, as shown in FIG. 7, the position where the sensor 100 is installed is taken as the reference (origin O), the right side of the sensor 100 is taken as the positive Y-axis direction, and the left side of the sensor 100 is taken as the negative Y-axis direction. Also, as shown in FIG. 7, the position where the sensor 100 is installed is taken as the reference (origin O), the upper side of the sensor 100 is taken as the positive Z-axis direction, and the lower side of the sensor 100 is taken as the negative Z-axis direction. That is, in FIG. 7, lx indicates the distance in the depth direction, ly indicates the distance in the horizontal direction, and lz indicates the distance in the vertical (height) direction. In the situation shown in FIG. 7, the output of the sensor 100 may be four-channel data of data elements (signal strength and speed) of positions (X, Y, Z) in three-dimensional space at a certain moment. In step S11 shown in FIG. 7, the electronic device 1 converts the information detected in this three-dimensional space into two-dimensional information.

Figure 8 is a diagram illustrating an example of generating point cloud information that can be processed two-dimensionally from point cloud information in three-dimensional space output by the sensor 100. Figure 8 is a diagram illustrating an example of a monitored object Tm standing at a certain location, detected three-dimensionally (spatially) by the sensor 100 installed at the origin O, and converted into two-dimensional (planar) data by the electronic device 1.

As shown in FIG. 8, with the position of origin O as the reference, the right side of origin O is the positive X-axis direction, and the bottom side of origin O is the positive Y-axis direction. That is, in FIG. 8, px indicates the horizontal coordinate, and py indicates the vertical coordinate. In the situation shown in FIG. 8, the output of sensor 100 is converted at a certain moment from the above-mentioned four-channel data into three-channel data of data elements (signal strength and speed) of position (X, Y) on a two-dimensional plane. In this way, in step S11 shown in FIG. 7, electronic device 1 converts information detected in three-dimensional space into information on a two-dimensional plane.

When converting the information detected in three-dimensional space as described above into information on a two-dimensional plane, the coordinates of the two-dimensional plane may be calculated, for example, based on the following formulas (3) and (4).

In the above formulas (3) and (4), lx _i , ly _i , and lz _i indicate outputs based on the results of detection by the sensor 100, i.e., point cloud information in three-dimensional space. In particular, lx _i indicates the x-direction distance of the i-th piece of information detected by the sensor 100. Furthermore, ly _i indicates the y-direction distance of the i-th piece of information detected by the sensor 100. Furthermore, lz _i indicates the z-direction distance of the i-th piece of information detected by the sensor 100.

In the above formulas (3) and (4), px _i and py _i indicate the coordinates of the point cloud converted into two-dimensional plane information by the electronic device 1. In particular, px _i indicates the x-coordinate of the i-th piece of information detected by the sensor 100. Furthermore, py _i indicates the y-coordinate of the i-th piece of information detected by the sensor 100.

Furthermore, in the above formula (3), M indicates the number of pixels in the horizontal direction when the image is assumed to be a two-dimensional plane, and αx indicates the horizontal angle of view when the image is assumed to be a two-dimensional plane. Furthermore, in the above formula (4), N indicates the number of pixels in the vertical direction when the image is assumed to be a two-dimensional plane, and αy indicates the vertical angle of view when the image is assumed to be a two-dimensional plane.

In the above formulas (3) and (4), px _i and py _i may be rounded off to integers so as to function as coordinate values. Also, in the above formulas (3) and (4), data that does not satisfy, for example, 0≦px _i ≦M or 0≦py _i ≦N may be discarded so that px _i and py _i fit within the size of the image after conversion onto a two-dimensional plane.

In this way, the electronic device 1 according to one embodiment generates point cloud information that can be processed two-dimensionally from the output of the sensor 100. In particular, the electronic device 1 according to one embodiment may convert the point cloud information detected by the sensor 100 into point cloud information that can be processed two-dimensionally based on at least one of a predetermined number of pixels in a two-dimensional image and a predetermined angle of view. The electronic device 1 according to one embodiment may use a conversion formula other than the above-mentioned formula (3) and formula (4) when generating point cloud information that can be processed two-dimensionally from the output of the sensor 100.

According to the electronic device 1 of one embodiment, the dimensionality of the point cloud information is reduced in step S11, so the amount of calculation can be significantly reduced. Therefore, according to the electronic device 1 of one embodiment, the processing load of the controller 10, for example, can be reduced.

After the dimensionality of the point cloud information is reduced in step S11, the electronic device 1 superimposes the frame information (step S12). In step S12, the electronic device 1 may execute a process of superimposing frames multiple times, such as that shown in FIG. 3.

Figure 9 is a diagram explaining the superimposition of frame information performed in step S12. As shown in Figure 9, the electronic device 1 may superimpose point cloud data generated in, for example, 50 frames. Here, the electronic device 1 may simply add up the values of the point cloud data generated in, for example, 50 frames. On the other hand, the electronic device 1 may average the values of the point cloud data generated in, for example, 50 frames according to the number of times it occurs in the 50 frames. For example, if information at a certain point is detected in 5 frames out of 50 frames, the electronic device 1 may average the values detected in those 5 frames.

In this way, the electronic device 1 according to one embodiment may generate point cloud information that can be processed two-dimensionally by aggregating the output of the sensor 100 for multiple frames. The electronic device 1 according to one embodiment may also generate point cloud information that can be processed two-dimensionally by averaging the point cloud information detected by the sensor 100 according to the number of detections for each of multiple frames.

According to the electronic device 1 of one embodiment, since frame information is superimposed in step S12, for example, the point cloud information tends to become dense in two dimensions. Therefore, according to the electronic device 1 of one embodiment, the recognition rate of body parts, for example, performed in step S3 of FIG. 4, can be improved.

After the frame information is superimposed in step S12, the electronic device 1 enlarges the points in the point cloud information (step S13). In step S13, the electronic device 1 may enlarge each point of the frame superimposed in step S12. Furthermore, when enlarging each point in step S13, the electronic device 1 may enlarge each point by any size in a predetermined direction.

Figure 10 is a diagram illustrating the enlargement of a point in the point cloud information performed in step S13. As shown on the left side of Figure 10, for example, after steps S11 and S12, a point indicated by a filled dot is detected in the point cloud information that can be processed two-dimensionally. In this case, as shown on the right side of Figure 10, the electronic device 1 may perform a process to enlarge the point, for example, by one dot (one pixel) in the left-right direction and by two dots (two pixels) in the up-down direction. Here, the electronic device 1 may perform a process to enlarge the point, for example, by two dots (two pixels) in the left-right direction and by one dot (one pixel) in the up-down direction. In this way, the electronic device 1 may perform a process to enlarge the point by any dot (pixel) in the left-right direction and/or up-down direction.

In this way, the electronic device 1 according to one embodiment may increase the size of each point constituting a point cloud in a specified direction in point cloud information that can be processed two-dimensionally.

According to the electronic device 1 of one embodiment, the points in the point cloud information are enlarged, so that, for example, the point cloud information tends to become denser in two dimensions. Therefore, according to the electronic device 1 of one embodiment, the recognition rate of body parts, for example, performed in step S3 of FIG. 4, can be improved.

In the above embodiment, the electronic device 1 has been described as enlarging each point of the point cloud information that has been reduced in dimension in step S11. However, in one embodiment, the order of processing is not limited to this. For example, in one embodiment, the electronic device 1 may enlarge each point of the point cloud information before it is reduced in dimension in step S11, and then reduce the dimension of the point cloud information in step S11. In this way, the electronic device 1 according to one embodiment may generate point cloud information that can be processed two-dimensionally by enlarging the size of each point constituting the point cloud in a predetermined direction in the point cloud information detected by the sensor 100.

After enlarging each point in the point cloud information in step S13, the electronic device 1 weights each point in the point cloud information (step S14). In step S14, the electronic device 1 may adopt information with a frequency equal to or higher than a predetermined frequency or exclude information with a frequency equal to or lower than a predetermined frequency, for example, depending on the detection frequency in a predetermined number of frames.

Figure 11 is a diagram for explaining the weighting of each point in the point cloud information performed in step S14. In Figure 11, Dr shown on the left side indicates distance information detected as a point cloud, and Di indicates intensity information detected as a point cloud. In this way, the left side of Figure 11 indicates two-channel information of detected distance and intensity. Also, in Figure 11, HM shown in the upper part may be information indicating the detection frequency of the detected point cloud for each pixel, for example, as in a heat map (hereinafter, also referred to as "detection frequency information"). In Figure 11, Dr' shown on the right side indicates the distance information Dr shown on the left side multiplied by the detection frequency information HM. Also, in Figure 11, Di' shown on the right side indicates the intensity information Di shown on the left side multiplied by the detection frequency information HM. In this way, the right side of Figure 11 indicates the two-channel information of detected distance and intensity multiplied by the detection frequency information HM.

As shown on the right side of FIG. 11, based on the detected distance and intensity information multiplied by the detection frequency information HM, the electronic device 1 may adopt information with a predetermined frequency or more, or exclude information with a predetermined frequency or less. For example, the electronic device 1 may adopt only distance and intensity information that is detected 10 frames or more out of 50 frames. Also, for example, the electronic device 1 may exclude distance and intensity information that is detected 5 frames or less out of 50 frames.

In this way, the electronic device 1 according to one embodiment may weight each of the points constituting the point cloud in the point cloud information that can be processed two-dimensionally according to the detection frequency in each of multiple frames. In particular, the electronic device 1 according to one embodiment may exclude points constituting the point cloud in the point cloud information that can be processed two-dimensionally that have a detection frequency in each of multiple frames that is equal to or lower than a predetermined frequency.

For example, when an object is detected by a sensor 100 such as a millimeter wave radar, the detection result includes noise. FIG. 12 is a diagram showing an example of point cloud information that has been processed from step S21 to step S23 in FIG. 6. FIG. 12 shows point cloud information based on the output of the sensor 100 superimposed on an image captured by the imaging unit 300. As shown in FIG. 12, most of the point cloud information based on the output of the sensor 100 overlaps with the monitored object Tm in the image captured by the imaging unit 300. On the other hand, some of the point cloud information based on the output of the sensor 100 does not overlap with the monitored object Tm in the image captured by the imaging unit 300. In this way, when the sensor 100 is, for example, a millimeter wave radar, the detection result includes noise when an object is detected by the sensor 100.

Figure 13 is a diagram showing an example of point cloud information that has undergone the processing of step S24 in Figure 6. As shown in Figure 13, by undergoing the processing of step S24, the point cloud information based on the output of the sensor 100 comes to nearly overlap with the monitored target Tm in the image captured by the imaging unit 300. In other words, by undergoing the processing of step S24, the influence of noise in the point cloud information based on the output of the sensor 100 is significantly reduced.

As described above, the electronic device 1 according to one embodiment can process (pre-process) the results of detecting an object based on the reflected waves of a transmitted wave, as in a millimeter wave radar, as if it were an image captured by an imaging unit. Therefore, the electronic device 1 according to one embodiment can be useful for monitoring a monitored object.

The processing from step S21 to step S24 shown in FIG. 6 can perform the processing of step S2 shown in FIG. 4, i.e., pre-processing of the point cloud data. The pre-processing of the point cloud data shown in step S2 of FIG. 4 generates point cloud information that can be processed two-dimensionally. Therefore, after this, the electronic device 1 may perform processing such as image recognition on the point cloud information that can be processed two-dimensionally as the processing from step S3 onward shown in FIG. 4. That is, the electronic device 1 may recognize the monitored object Tm as a two-dimensional object in the point cloud information that can be processed two-dimensionally. Therefore, the electronic device 1 may generate point cloud information that can be recognized as a two-dimensional object by pre-processing the point cloud data in step S2. That is, the electronic device 1 according to one embodiment may generate point cloud information that can be recognized as a two-dimensional object from the output of the sensor 100.

As described above, in the processing of step S3 shown in FIG. 4, the body parts of the monitored subject Tm are image-recognized from point cloud information that can be processed two-dimensionally. Therefore, the electronic device 1 may generate point cloud information that can recognize the body parts of the monitored subject Tm by pre-processing the point cloud data in step S2. In this way, the electronic device 1 according to one embodiment may generate point cloud information that can recognize the body parts of the monitored subject that are detected as objects from the output of the sensor 100.

Also, as shown in FIG. 13, the electronic device 1 may output point cloud information that can be processed two-dimensionally superimposed on two-dimensional image information of an object captured by the imaging unit 300. Point cloud information superimposed on two-dimensional image information of an object captured by the imaging unit 300 as shown in FIG. 13 is not essential when monitoring the behavior of the monitored subject Tm. However, for example, when recognizing a body part of the monitored subject Tm by image recognition or a human, it is convenient to have point cloud information superimposed on two-dimensional image information of an object captured by the imaging unit 300 as shown in FIG. 13. Therefore, the electronic device 1 according to one embodiment may superimpose point cloud information that can be processed two-dimensionally on two-dimensional image information of an object captured by the imaging unit 300.

Next, a further description will be given of an aspect in which the electronic device 1 according to one embodiment uses machine learning to recognize (or estimate) the body part of the monitored subject Tm shown in step S3 of FIG. 4. The electronic device 1 according to one embodiment uses machine learning to recognize the body part of the monitored subject Tm, and can estimate the body part of the monitored subject Tm from the output of the sensor 100, for example, as shown in FIG. 5.

First, a detection device for recognizing (estimating) the body part of a monitored subject Tm through machine learning in an electronic device 1 according to an embodiment will be described. Figs. 14 and 15 are diagrams showing an example of the configuration of a detection device that supplies information for machine learning to an electronic device 1 according to an embodiment. Fig. 14 is a front view showing an example of a detection device according to an embodiment as viewed from the front. Also, Fig. 15 is a side view showing an example of a detection device according to an embodiment as viewed from the side (left).

As shown in Figs. 14 and 15, the detection device 3 according to one embodiment includes a sensor 100 and an imaging unit 300. Also, as shown in Figs. 14 and 15, the detection device 3 according to one embodiment may include at least one of a stand unit 5 and a ground unit 7, as appropriate.

The sensor 100 shown in FIG. 14 and FIG. 15 may be the sensor 100 described in FIG. 1 and/or FIG. 2. As shown in FIG. 14 and FIG. 15, the sensor 100 may include a radio wave input unit 101 that receives a reflected wave of a transmitted wave reflected by an object. As shown in FIG. 15, the radio wave input unit 101 may be oriented along the optical axis Ra of the sensor 100. Here, the optical axis of the sensor 100 may be, for example, a direction perpendicular to a surface on which at least one of the transmitting antenna 125 and the receiving antenna 131 of the sensor 100 is installed. In addition, when at least one of the transmitting antenna 125 and the receiving antenna 131 is included in a plurality of antennas, the optical axis of the sensor 100 may be a direction perpendicular to a surface on which at least one of the plurality of antennas is installed. With this configuration, the sensor 100 can transmit a transmitted wave and/or receive a reflected wave with the optical axis Ra as the center. That is, the sensor 100 can detect an object as a point cloud within a range centered on the optical axis Ra.

The imaging unit 300 shown in FIG. 14 and FIG. 15 may be the imaging unit 300 described in FIG. 1. As shown in FIG. 14 and FIG. 15, the imaging unit 300 may include a light input unit 301 that receives light reflected by an object. As shown in FIG. 15, the light input unit 301 may face the optical axis La of the imaging unit 300. The light input unit 301 may also be a position where a lens is placed in the imaging unit 300. Here, the optical axis of the imaging unit 300 may be, for example, a direction perpendicular to a surface on which a light receiving element (or imaging element) used for imaging in the imaging unit 300 is installed. With this configuration, the imaging unit 300 can capture an image centered on the optical axis La.

As shown in Figures 14 and 15, the stand unit 5 maintains the sensor 100 and the imaging unit 300 at a predetermined height from the ground in the detection device 3. The stand unit 5 may maintain the sensor 100 at a height at which the sensor 100 can easily detect an object such as a monitored target. The stand unit 5 may also maintain the sensor 100 at a height at which the imaging unit 300 can easily capture an image of an object such as a monitored target. The stand unit 5 may be provided with a mechanism that can adjust the sensor 100 and the imaging unit 300 in the detection device 3, for example, in the height direction.

As shown in Figs. 14 and 15, the grounding section 7 fixes the sensor 100 and the imaging section 300 to the ground surface in the detection device 3. The grounding section 7 can have various configurations, such as a pedestal shape, to stabilize the detection device 3 including the sensor 100 and the imaging section 300.

As shown in Figs. 14 and 15, in a detection device 3 according to an embodiment, the sensor 100 and the imaging unit 300 may be disposed adjacent to each other in the vicinity. In the example shown in Figs. 14 and 15, the sensor 100 and the imaging unit 300 are disposed adjacent to each other in the up-down direction. In a detection device 3 according to an embodiment, the sensor 100 and the imaging unit 300 may be disposed adjacent to each other in the left-right direction or diagonally, for example.

Also, as shown in FIG. 15, in a detection device 3 according to an embodiment, the sensor 100 and the imaging unit 300 may be arranged so that their respective optical axes Ra and La are parallel. That is, in an electronic device 1 according to an embodiment, the optical axis La of the imaging unit 300 may be set parallel to the optical axis Ra of the sensor 100, and the point cloud information from the sensor 100 and the image information from the imaging unit 300 may be used.

Also, as shown in FIG. 15, in a detection device 3 according to one embodiment, the sensor 100 and the imaging unit 300 may be arranged so that the distance between their respective optical axes Ra and La is maintained at G. By arranging them in this manner, the point cloud information from the sensor 100 and the image information from the imaging unit 300 are shifted from each other by the distance G. For example, in the arrangement shown in FIG. 15, the image information from the imaging unit 300 is shifted upward by the distance G from the point cloud information from the sensor 100. Also, in the arrangement shown in FIG. 15, the point cloud information from the sensor 100 is shifted downward by the distance G from the image information from the imaging unit 300.

Therefore, in the arrangement shown in FIG. 15, for example, by correcting the point cloud information from the sensor 100 so as to shift it upward by a distance G, the position of the point cloud information from the sensor 100 can be made to correspond to the position of the image information from the imaging unit 300. Also, in the arrangement shown in FIG. 15, for example, by correcting the image information from the imaging unit 300 so as to shift it downward by a distance G, the position of the image information from the imaging unit 300 can be made to correspond to the position of the point cloud information from the sensor 100. In this way, the electronic device 1 may correct at least one of the point cloud information from the sensor 100 and the image information from the imaging unit 300 so that the point cloud information from the sensor 100 and the image information from the imaging unit 300 positionally correspond to each other.

That is, in the electronic device 1 according to one embodiment, at least one of the point cloud information obtained by the sensor 100 detecting an object (monitored target) and the image information obtained by the imaging unit 300 capturing an image of the object may be corrected. The electronic device 1 according to one embodiment may use information obtained by aligning the point cloud information from the sensor 100 and the image information from the imaging unit 300 through correction.

It is also assumed that the detection range (angle) of the point cloud by the sensor 100 and the imaging range (angle or angle of view) of the image by the imaging unit 300 will not be the same. In such a case, the electronic device 1 may adjust the wider range (angle) of the point cloud information by the sensor 100 and the image information by the imaging unit 300 to the narrower range (angle) so that they positionally correspond to each other. In other words, the electronic device 1 may use information only on the overlapping range between the imaging range of the imaging unit 300 and the detectable range of the sensor 100, and delete or ignore information on the non-overlapping range.

As described above, in the electronic device 1 according to one embodiment, the image information obtained by the imaging unit 300 capturing an image of an object (target to be monitored) may be information that positionally corresponds to the point cloud information obtained by the sensor 100 detecting the object (target to be monitored).

The following further describes the operation of the electronic device 1 according to one embodiment when the body part of the monitored subject Tm shown in step S3 of FIG. 4 is recognized (or estimated) by machine learning using the detection device 3.

The electronic device 1 according to an embodiment uses the point cloud information output from the sensor 100 to machine-learn a predetermined part (e.g., a body part) of an object such as a monitored object. Here, the electronic device 1 according to an embodiment may use image information obtained by capturing an image of the object (monitored object) by the imaging unit 300 to machine-learn a predetermined part of the object. Hereinafter, such a machine learning phase by the electronic device 1 according to an embodiment may be simply referred to as a "learning phase". After undergoing machine learning as described above, the electronic device 1 according to an embodiment can estimate a predetermined part (e.g., a body part) of an object such as a monitored object using the point cloud information output from the sensor 100. Hereinafter, such a part estimation phase by the electronic device 1 according to an embodiment may be simply referred to as an "estimation phase". Once a predetermined part (e.g., a body part) of an object such as a monitored object is estimated (recognized) in this way, the electronic device 1 according to an embodiment can perform the process from step S4 shown in FIG. 4 onward based on the information of the estimated (recognized) part.

(Learning Phase)
First, the learning phase will be described. The electronic device 1 according to an embodiment uses the output of the sensor 100 to perform machine learning of a predetermined part (e.g., a body part) of an object (a monitored target).

FIG. 16 is a diagram illustrating an example of machine learning by the electronic device 1 according to an embodiment. The electronic device 1 according to an embodiment performs machine learning on a predetermined part (e.g., a body part) of an object (e.g., a monitored subject) detected by the sensor 100, using point cloud data based on the output of the sensor 100, for example, as shown on the left side of FIG. 16. At this time, the electronic device 1 may perform machine learning by using coordinates of the part of the monitored subject based on an image captured by the imaging unit 300, for example, as shown on the upper side of FIG. 16. FIG. 16 conceptually illustrates an aspect in which the electronic device 1 according to an embodiment performs machine learning based on point cloud data based on the output of the sensor 100 and coordinates of the part based on an image captured by the imaging unit 300.

When performing machine learning as shown in FIG. 16, the electronic device 1 according to one embodiment may use data suggesting the correct answer (hereinafter, sometimes referred to as "correct answer data") as information indicating the coordinates of each body part based on the image captured by the imaging unit 300. That is, the coordinates of the body part of the monitored subject based on the image captured by the imaging unit 300 as shown in the upper part of FIG. 16 may be correct answer data indicating the coordinates of each body part of the monitored subject. Such correct answer data may be specified in advance by personnel, or may be image-recognized by an information processing device such as the electronic device 1.

Figure 17 is a diagram illustrating an example of designation of a specific part and image recognition by electronic device 1 according to an embodiment. As shown on the left side of Figure 17, for example, by having personnel designate or recognize an image of a monitored object captured by imaging unit 300, the coordinates of each part of the monitored object can be obtained as shown on the right side of Figure 17.

For example, the electronic device 1 may be configured so that the positions corresponding to each part in the image of the monitored subject captured by the imaging unit 300 are manually specified by a specific person or the like. For example, the electronic device 1 according to one embodiment may request a person such as a user of the electronic device 1 to specify the "neck" part in the image of the monitored subject captured by the imaging unit 300. In this case, the person such as a user of the electronic device 1 may perform an operation to specify the neck part in the image of the monitored subject captured by the imaging unit 300 in response to a request from the electronic device 1. For example, the person such as a user of the electronic device 1 may specify the neck part with a pointer in the image of the monitored subject displayed on the display unit or the like as shown on the left side of FIG. 17. Also, for example, the person such as a user of the electronic device 1 may specify the neck part by touching it with a finger or a stylus in the image of the monitored subject displayed on the display unit or the like as shown on the left side of FIG. 17.

When the neck area is identified in this manner, the electronic device 1 may identify the coordinates of the position corresponding to that area in the image of the monitored subject captured by the imaging unit 300, as shown on the right side of Figure 17. Similarly, the electronic device 1 may ask a person, such as a user of the electronic device 1, to specify the "right shoulder" area, then the "left shoulder" area, and so on, in the image of the monitored subject captured by the imaging unit 300.

In this way, in the electronic device 1 according to one embodiment, image information obtained by the imaging unit 300 capturing an image of an object (target to be monitored) may be information on an image in which a specific part of the object is designated in advance (e.g., by a human hand).

Also, for example, the electronic device 1 may be configured to perform image recognition (without manual intervention by a specified person or the like) of the positions corresponding to each part in the images of the monitored subject captured by the imaging unit 300. For example, the electronic device 1 according to one embodiment may identify the coordinates of the "neck" part by image recognition in the images of the monitored subject captured by the imaging unit 300. In this case, the electronic device 1 may use information learned by machine learning using correct answer data for each part previously designated by the human intervention of a specified person or the like, for example, as described above. Also, in this case, the electronic device 1 can improve the accuracy of image recognition of each part by machine learning the positions corresponding to each part in the images of the monitored subject captured by the imaging unit 300.

When the neck area has been image-recognized in this manner, the electronic device 1 may identify the coordinates of the position corresponding to that area in the image of the monitored subject captured by the imaging unit 300, as shown on the right side of Figure 17. Similarly, the electronic device 1 may then image-recognize the "right shoulder" area, then the "left shoulder" area, etc. in the image of the monitored subject captured by the imaging unit 300.

In this way, in the electronic device 1 according to one embodiment, information on the image obtained by the imaging unit 300 capturing an image of an object (target to be monitored) may be information on image recognition of a specific part of the object.

The right side of Figure 17 shows that a specific part is identified on an object, such as a monitored object, and the position of the specific part, i.e., the coordinates of the position of the specific part, is shown.

As described above, the electronic device 1 according to an embodiment may perform machine learning of a predetermined part (body part) of an object (monitored target) using the output of the sensor 100 and the output of the imaging unit 300. Here, the output of the sensor 100 may be point cloud information obtained by the sensor 100 detecting an object (monitored target) reflecting a transmission wave based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave of the transmission wave. Furthermore, the output of the imaging unit 300 may be image information obtained by the imaging unit 300 capturing an image of the object (monitored target). Furthermore, the electronic device 1 according to an embodiment may perform machine learning of the position of a predetermined part of the object (monitored target). Furthermore, the electronic device 1 according to an embodiment may perform machine learning of the coordinates of the position of a predetermined part of the object (monitored target).

(Estimation Phase)
Next, the estimation phase will be described. When the electronic device 1 according to an embodiment performs machine learning as described above, it can estimate a predetermined part (e.g., a body part) of an object such as a monitored target by using the point cloud information output from the sensor 100.

Figure 18 is a diagram illustrating an example of estimating a specific part by electronic device 1 according to an embodiment. Electronic device 1 according to an embodiment can estimate a specific part (e.g., a body part) of an object such as a monitored target, as shown on the right side of Figure 18, through machine learning shown in the center of Figure 18, using point cloud data based on the output of sensor 100 shown on the left side of Figure 18. The right side of Figure 18 shows that the position of the specific part, i.e., the coordinates of the position of the specific part, is shown as the specific part of an object such as a monitored target.

In this way, the electronic device 1 according to one embodiment may perform machine learning of a predetermined part of an object using point cloud information obtained by detecting the object using, for example, the sensor 100. By performing such machine learning, the electronic device 1 according to one embodiment can estimate a predetermined part of the monitored object using point cloud information obtained by detecting the monitored object using the sensor 100.

According to the electronic device 1 of one embodiment, each part of a specific object such as a monitored object can be recognized based on point cloud data output from a sensor 300 such as a millimeter wave radar sensor. Therefore, according to the electronic device 1 of one embodiment, even if a specific object such as a monitored object is performing complex movements, each part of the object can be recognized. Furthermore, according to the electronic device 1 of one embodiment, after a specific machine learning, each part of a specific object such as a monitored object can be recognized based on point cloud data output from a sensor 300 such as a millimeter wave radar sensor. Therefore, according to the electronic device 1 of one embodiment, each part can be recognized without using an image of the monitored object, for example. Therefore, it is possible to imagine a use case that takes into consideration the privacy of the monitored object.

Although the embodiments of the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can easily make various modifications or corrections based on the present disclosure. Therefore, it should be noted that these modifications or corrections are included in the scope of the present disclosure. For example, the functions included in each component or step can be rearranged so as not to cause logical inconsistencies, and multiple components or steps can be combined into one or divided. Although the embodiments of the present disclosure have been described mainly with respect to the device, the embodiments of the present disclosure can also be realized as a method including steps executed by each component of the device. The embodiments of the present disclosure can also be realized as a method, a program, or a storage medium on which a program is recorded, executed by a processor provided in the device. It should be understood that these are also included in the scope of the present disclosure.

The above-described embodiment is not limited to implementation only as the electronic device 1. For example, the above-described embodiment may be implemented as an electronic device 1 included in the electronic device 1. The above-described embodiment may also be implemented as, for example, a monitoring method using an apparatus such as the electronic device 1. Furthermore, the above-described embodiment may also be implemented as, for example, a program executed by an apparatus such as the electronic device 1 or an information processing apparatus (for example, a computer).

REFERENCE SIGNS LIST 1 Electronic device 3 Detector 5 Stand unit 7 Ground unit 10 Controller 20 Memory unit 30 Communication unit 40 Display unit 50 Notification unit 100 Sensor 101 Radio wave input unit 110 Radar control unit 120 Transmitter 121 Signal generator 122 Synthesizer 123 Phase controller 124 Amplifier 125 Transmitting antenna 130 Receiving unit 131 Receiving antenna 132 LNA
133 Mixer 134 IF section 135 AD conversion section 300 Imaging section 301 Optical input section

Claims (13)

Point cloud information obtained by a sensor detecting an object reflecting the transmission wave based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave of the transmission wave; and
and information on an image obtained by capturing an image of the object by an imaging unit.
Machine learning is performed on a predetermined portion of the object;
The point cloud information obtained by the detection by the sensor after recognizing a predetermined portion of the object is
An electronic device, in which the maximum value of the coordinates of the extracted point cloud is Xmax, the minimum value of the extracted coordinates is Xmin, and the maximum value of the coordinates after normalization is X'max, is information obtained by converting the coordinates (X) before normalization into the coordinates after normalization (X') using the following formula (1).
X'=((X-Xmin)/(Xmax-Xmin))・X'max (1) The electronic device according to claim 1, which performs machine learning to determine the position of a predetermined part of the object. The electronic device according to claim 2, which performs machine learning to determine the coordinates of the position of a predetermined part of the object. The electronic device according to any one of claims 1 to 3, which performs machine learning of a predetermined part of the object using information that corresponds in position to information of a point cloud obtained by detecting the object with the sensor as information of the image obtained by the imaging unit capturing an image of the object. The electronic device according to claim 4, wherein the optical axis of the imaging unit is set parallel to the optical axis of the sensor, and machine learning is performed on a predetermined part of the object using point cloud information obtained by the sensor detecting the object and image information obtained by the imaging unit capturing an image of the object. The electronic device according to any one of claims 1 to 5, which performs machine learning of a predetermined part of the object using information obtained by correcting at least one of the point cloud information obtained by detecting the object with the sensor and the image information obtained by capturing an image of the object with the imaging unit, thereby aligning the two. The electronic device according to any one of claims 1 to 6, which performs machine learning of a specific part of the object using image information in which a specific part of the object is specified in advance as image information obtained by the imaging unit capturing an image of the object. The electronic device according to any one of claims 1 to 6, which performs machine learning of a specific part of the object using image recognition information of a specific part of the object as information of an image obtained by the imaging unit capturing an image of the object. A predetermined portion of an object is machine-learned using point cloud information obtained by detecting an object reflecting the transmission wave based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave of the transmission wave,
Based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave of the transmission wave, a sensor detects a monitored object reflecting the transmission wave, and the information of the point cloud is used to obtain the detected point cloud information.
Estimating a predetermined part of the monitored object;
The point cloud information obtained by the detection by the sensor after recognizing a predetermined portion of the object is
An electronic device, in which the maximum value of the coordinates of the extracted point cloud is Xmax, the minimum value of the extracted coordinates is Xmin, and the maximum value of the coordinates after normalization is X'max, is information obtained by converting the coordinates (X) before normalization into the coordinates after normalization (X') using the following formula (1).
X'=((X-Xmin)/(Xmax-Xmin))・X'max (1) A step of acquiring point cloud information by detecting an object reflecting the transmission wave using a sensor based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave of the transmission wave;
acquiring image information by capturing an image of the object with an imaging unit;
A step of machine learning a predetermined portion of the object;
Including,
The point cloud information obtained by the detection by the sensor after recognizing a predetermined portion of the object is
A method for controlling an electronic device, the information being obtained by converting a coordinate (X) before normalization into a coordinate (X') after normalization using the following formula (1), where the maximum value of the coordinates of the extracted point cloud is Xmax, the minimum value of the extracted coordinates is Xmin, and the maximum value of the coordinates after normalization is X'max:
X'=((X-Xmin)/(Xmax-Xmin))・X'max (1) A step of performing machine learning of a predetermined portion of an object using point cloud information obtained by detecting an object reflecting the transmission wave based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave of the transmission wave;
A step of estimating a predetermined part of the monitored object using point cloud information obtained by a sensor detecting the monitored object reflecting the transmission wave based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave of the transmission wave;
Including,
The point cloud information obtained by the detection by the sensor after recognizing a predetermined portion of the object is
A method for controlling an electronic device, the information being obtained by converting a coordinate (X) before normalization into a coordinate (X') after normalization using the following formula (1), where the maximum value of the coordinates of the extracted point cloud is Xmax, the minimum value of the extracted coordinates is Xmin, and the maximum value of the coordinates after normalization is X'max:
X'=((X-Xmin)/(Xmax-Xmin))・X'max (1) For electronic devices,
A step of acquiring point cloud information by detecting an object reflecting the transmission wave using a sensor based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave of the transmission wave;
acquiring image information by capturing an image of the object with an imaging unit;
A step of machine learning a predetermined portion of the object;
Run the command,
The point cloud information obtained by the detection by the sensor after recognizing a predetermined portion of the object is
A program that converts pre-normalization coordinates (X) into normalized coordinates (X') using the following formula (1), where the maximum value of the coordinates of the extracted point cloud is Xmax, the minimum value of the extracted coordinates is Xmin, and the maximum value of the coordinates after normalization is X'max.
X'=((X-Xmin)/(Xmax-Xmin))・X'max (1) For electronic devices,
A step of performing machine learning of a predetermined portion of an object using point cloud information obtained by detecting an object reflecting the transmission wave based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave of the transmission wave;
A step of estimating a predetermined part of the monitored object using point cloud information obtained by a sensor detecting the monitored object reflecting the transmission wave based on a transmission signal transmitted as a transmission wave and a reception signal received as a reflected wave of the transmission wave;
Run the command,
The point cloud information obtained by the detection by the sensor after recognizing a predetermined portion of the object is
A program that converts pre-normalization coordinates (X) into normalized coordinates (X') using the following formula (1), where the maximum value of the coordinates of the extracted point cloud is Xmax, the minimum value of the extracted coordinates is Xmin, and the maximum value of the coordinates after normalization is X'max.
X'=((X-Xmin)/(Xmax-Xmin))・X'max (1)

Images