JP7530012B2 - Information output system, information output method, information output device, and program - Google Patents

Description

The present invention relates to an information output system, an information output method, an information output device, and a program.

The Internet of Things (IoT), in which various devices are connected to the Internet, is becoming a reality, and various devices such as automobiles, drones, and construction machinery vehicles are being connected wirelessly. Wireless communication standards include wireless LAN defined by the standard IEEE 802.11, Bluetooth (registered trademark), cellular communication using LTE and 5G, Low Power Wide Area (LPWA) communication for IoT, Electronic Toll Collection System (ETC), Vehicle Information and Communication System (VICS), and ARIB-STD-T109, and other supported wireless standards are also evolving and are expected to become more widespread in the future.

While wireless communication is used for various purposes, it is problematic that, depending on the service, the requirements for wireless communication quality may not always be met. For example, the movement of a wireless communication terminal or surrounding objects may change the direction of antenna directivity and the propagation environment, which may affect communication quality. Non-Patent Document 1 shows that it is possible to use a depth camera to predict that objects surrounding a wireless communication terminal will block the communication path, and the usefulness of video information obtained from cameras and sensors in wireless communication has also been attracting attention.

Nishio, Takayuki, et al. "Proactive received power prediction using machine learning and depth images for mmWave networks." IEEE Journal on Selected Areas in Communications 37.11 (2019): 2413-2427. J. Redmon and A. Farhadi, "Yolov3: An incremental improvement," CoRR, vol. abs/1804.02767, Apr. 2018.

However, in order to take into account the impact on communication conditions caused by changes in the antenna directivity and propagation environment due to the movement of the wireless communication terminal or surrounding objects, it is necessary to know the position and status of the wireless communication terminal within the environment.

The present invention has been made in consideration of the above, and aims to grasp the position and status of a wireless communication terminal performing wireless communication.

An information output system of one embodiment of the present invention is an information output system that outputs terminal information of a wireless communication terminal that performs wireless communication, and includes a collection device that collects environmental information sensed in a space in which the wireless communication terminal exists, a communication device that acquires actual measured values of wireless communication quality of the wireless communication terminal, a detection unit that detects candidate objects of the wireless communication terminal from the environmental information, an estimation unit that uses a prediction model that has been machine-learned to determine the relationship between the environmental information and wireless communication quality, inputs the environmental information into the prediction model, and outputs an estimated value of wireless communication quality for each of the candidate objects, and an extraction unit that compares the estimated value and the actual measured value for each of the candidate objects, and extracts terminal information for the candidate object with the highest estimated accuracy from the environmental information.

According to the present invention, it is possible to grasp the position and status of a wireless communication terminal performing wireless communication.

FIG. 1 is a diagram showing an example of the configuration of a terminal information output system according to the present embodiment. FIG. 2 is a flowchart showing an example of a process flow of the terminal information output system. FIG. 3 is a plan view of the indoor experimental area. FIG. 4 shows an image of the experimental area obtained from a camera. FIG. 5 is a diagram showing the evaluation results obtained by comparing the estimated RSSI values with the actually measured values. FIG. 6 is a diagram showing predicted and measured RSSI values. FIG. 7 is a diagram illustrating an example of a hardware configuration of a device included in the terminal information output system.

One embodiment of the present invention will be described with reference to the drawings. The embodiment described below is an example of the present invention, and the present invention is not limited to the following embodiment. Note that components with the same reference numerals in this specification and drawings are assumed to indicate the same components.

First, referring to FIG. 1, the terminal information output system 1 of this embodiment will be described. The terminal information output system 1 is a system that outputs terminal information of a wireless communication terminal that performs wireless communication. The terminal information is, for example, information such as the position, attitude, and speed of the wireless communication terminal. The terminal information may also include a classification of the owner and inclusion of the wireless communication terminal (for example, a person, a car, a drone, a robot, etc.).

1 includes a collection unit 11, a wireless communication unit 12, a detection unit 13, an estimation unit 14, an extraction unit 15, a model generation unit 16, a model selection unit 17, a prediction unit 18, and an anomaly detection unit 19. The detection unit 13, the estimation unit 14, and the extraction unit 15 may be configured as a terminal information output device 20, or each unit included in the terminal information output system 1 may be configured as an individual device.

The collection unit 11 collects environmental information from a sensor. The environmental information is sensing data of the environment in which the target wireless communication terminal exists, and is, for example, image information obtained from a camera such as an RGB camera or a depth camera, point cloud information obtained from a Light Detection and Ranging (LiDAR), or time-varying information including the position, orientation, and speed of an object obtained from a camera, LiDAR, and other sensors. The collection unit 11 may collect environmental information from multiple sensors, or may collect multiple types of environmental information. Environmental information is also referred to as real-world information or sensor information.

The wireless communication unit 12 collects wireless communication information of the target wireless communication terminal. The wireless communication information is information on the quality of wireless communication, such as radio wave propagation information of the wireless communication signal, channel information related to the radio wave propagation information, feedback information of the channel information, received signal power, signal-to-noise power ratio, signal-to-interference-plus-noise power ratio, Received Signal Strength Indication (RSSI), Received Signal Reference Quality (RSRQ), packet error rate, number of arriving bits, bit error rate, number of arriving bits per unit time, and other indices related to QoE (Quality of Experience), as well as differential information of these values, indices calculated from these values using a formula, and setting items of the communication system that affect these indices. The wireless communication unit 12 may be, for example, a base station that performs wireless communication with the target wireless communication terminal, or the wireless communication unit 12 may acquire wireless communication information from the target wireless communication terminal.

The detection unit 13 uses an object detection technique to detect objects that are candidates for the target wireless communication terminal from the environmental information, and outputs terminal candidate information including the class and position information of the detected candidate object. The class is defined by the object detection technique and indicates the type of object. When the wireless communication terminal is held by a person, or when the wireless communication terminal is mounted on a vehicle, drone, or robot, the class is defined as a person, a vehicle, a drone, or a robot. The position information of an object is information that indicates the position of the object in the environmental information, and is, for example, information such as the center position, width, height, contour, and distance (depth) of the object on the camera image. The detection unit 13 detects candidate objects from the image using, for example, deep learning such as CNN. Other machine learning algorithms may be used. Parameters used in machine learning are learned in advance. The detection unit 13 may detect all objects shown in the image, or if the class of the target wireless communication terminal is known in advance, may detect only objects that correspond to that class, or may detect all objects shown in the image and then extract and output only information on objects that correspond to the class of the target wireless communication terminal. By outputting only the class of the target wireless communication terminal, it is possible to reduce the processing time in the estimation unit 14 described below.

The estimation unit 14 inputs the terminal candidate information into an estimation model that has learned the relationship between environmental information and wireless communication information, estimates the communication state for each of the candidate objects included in the terminal candidate information, and outputs an estimated value of the communication state for each of the candidate objects. The estimation unit 14 can estimate the communication state for each of the candidate objects by machine learning such as a neural network or a statistical method.

The model generation unit 16 acquires in advance environmental information and wireless communication information when the wireless communication terminal moves within the environment, and models the relationship between the two by machine learning. The model generation unit 16 may acquire environmental information and wireless communication information for each type of terminal and environment to create an estimation model, or may create an estimation model that can be adapted to another terminal or environment by performing transfer learning or statistical processing on a model created for a certain terminal or environment.

The model selection unit 17 selects a model corresponding to the target wireless communication terminal from the estimation models created by the model generation unit 16 and inputs the selected model to the estimation unit 14. When it is desired to simultaneously detect multiple wireless communication terminals, a model corresponding to each of the wireless communication terminals may be selected and input to the estimation unit 14.

The extraction unit 15 compares the wireless communication information of the target wireless communication terminal acquired from the wireless communication unit 12 with the estimated values of the communication state of each candidate object estimated by the estimation unit 14, identifies the candidate object with the highest estimation accuracy as the target wireless communication terminal, and outputs terminal information such as the position, attitude, and speed of the candidate object. The terminal information of the candidate object can be extracted from the terminal candidate information.

The terminal information output by the extraction unit 15 can be used for the communication quality prediction described in Non-Patent Document 1, and can also be used for predicting future communication conditions by the prediction unit 18. In addition, the behavior of the wireless communication terminal or the owner of the wireless communication terminal can be extracted from the terminal information, and can be used to estimate characteristics such as preferences from the behavior history, or to predict behavior such as starting communication.

The prediction unit 18 predicts a future communication state using the terminal information extracted by the extraction unit 15. The prediction unit 18, for example, uses an estimation model that has learned the relationship between environmental information and wireless communication information, and inputs the terminal information into the estimation model to predict future wireless communication information (communication state).

The anomaly detection unit 19 detects an anomaly when the estimation accuracy is lower than normal, that is, when the estimation accuracy is lower than a predetermined value, during estimation accuracy evaluation by the extraction unit 15. The anomaly detection unit 19 can be used to detect malfunctions of the wireless communication terminal, changes in the internal structure of the terminal (such as the antenna position), changes in the environment, and the like.

In addition, the terminal information output system 1 does not need to be equipped with the prediction unit 18 and the anomaly detection unit 19.

Next, referring to the flowchart in Figure 2, the processing flow of the terminal information output system 1 will be explained.

In step S11, the collection unit 11 collects environmental information of the environment in which the target wireless communication terminal is located from a sensor or the like.

In step S12, the detection unit 13 detects a candidate object from the environmental information. The detection unit 13 may detect multiple candidate objects.

In step S13, the estimation unit 14 estimates the communication state for each candidate object detected in step S12.

In step S14, the wireless communication unit 12 acquires the communication status of the target wireless communication terminal. Note that the processing of step S14 may be performed before the processing of step S11, or may be performed in parallel with the processing of steps S11, S12, and S13.

In step S15, the extraction unit 15 compares the estimated communication state of each candidate object estimated in step S13 with the actual measured communication state of the target wireless communication terminal obtained in step S14, identifies the target wireless communication terminal from the candidate objects, and extracts terminal information of the identified object from the environmental information.

In step S16, the prediction unit 18 predicts the future communication state using terminal information of the target wireless communication terminal.

Next, we will explain the experiments conducted to demonstrate the effectiveness of this embodiment and their results.

The wireless communication terminal used in the experiment is an autonomous mobile robot, which moves randomly within the indoor experimental area shown in Figure 3. A camera 110, a base station 120, and a stationary robot 130 were installed within the experimental area. The experimental area is photographed by the camera 110, and the obtained video information is used as environmental information. The base station 120 performs wireless communication with the autonomous mobile robot. Wireless communication information is obtained from the base station 120. The stationary robot 130 installed in the center of the experimental area is detected as a candidate object, just like the target autonomous mobile robot. The stationary robot 130 may also perform wireless communication with the base station 120.

Figure 4 is an image of the experimental area obtained from the camera 110. The image in Figure 4 shows a stationary robot 130 and an autonomous mobile robot 140. The detection unit 13 detects each of the stationary robot 130 and the autonomous mobile robot 140 as candidate objects, and obtains a frame (bounding box) surrounding the object (robot) detected in the image as terminal candidate information. The bounding box information includes four pieces of information: the center coordinates (X, Y), width W, and height H of the frame. The bounding box information may also include the class and likelihood of the detected object.

A wireless LAN (IEEE802.11ac) was used for wireless communication of the autonomous mobile robot. The wireless communication unit 12 measured the median RSSI value for 0.1 seconds as wireless communication information. The wireless communication unit 12 may be provided in either the base station 120 or the autonomous mobile robot.

The estimation unit 14 uses an estimation model that has previously learned the relationship between bounding box information for the past second and RSSI two seconds later, inputs bounding box information for candidate objects detected from the video of the camera 110, and outputs an RSSI estimate for each candidate object. In this experiment, a random forest was used as the estimation model. The number of decision trees in the random forest was 500. By inputting the bounding box information detected from the video of Figure 4 into the estimation model, an RSSI estimate for the stationary robot 130 and an RSSI estimate for the autonomous mobile robot 140 can be obtained.

The extraction unit 15 compares the RSSI estimate value with the actual RSSI value measured by the wireless communication unit 12 for each candidate object, extracts a bounding box in which the autonomous mobile robot is detected, and outputs it as terminal information. If the candidate object is a target autonomous mobile robot, the RSSI estimate value output when the bounding box information is input to the estimation model more accurately represents the actual RSSI value measured by the wireless communication unit 12 than when the bounding box information of other candidate objects is input. In this experiment, the prediction accuracy was evaluated using the ^R2 score expressed by the following formula.

Here, n is the number of samples, Yi is the actual RSSI value for the i-th sample, Yi with a hat is the predicted RSSI value predicted by the random forest, and Yave is the average of the actual RSSI values. The closer the ^R2 score is to 1, the higher the prediction accuracy.

Fig. 5 shows the evaluation results when comparing the estimated RSSI value and the actual measured value for 10 seconds each. In Fig. 5, the evaluation results for the autonomous mobile robot are plotted with a solid line, and the evaluation results for the stationary robot are plotted with a dotted line. As can be seen from Fig. 5, the ^R2 score is larger when the bounding box information of the autonomous mobile robot is input to the estimation model at any time, which shows that the autonomous mobile robot is the target wireless communication terminal.

Next, the bounding box information recognized as the target wireless communication terminal, i.e., the bounding box information (terminal information) extracted by the extraction unit 15, was input to the prediction unit 18, and the RSSI for two seconds ahead was predicted. In Figure 6, the predicted RSSI value is shown by a dotted line, and the actual measured RSSI value is shown by a solid line. The root mean square error (RMSE) at this time was 1.97 dB. It can be seen from the figure that it is possible to predict future communication conditions (RSSI).

Furthermore, we investigated the prediction error when the position of the base station 120 was moved. When the bounding box information of the autonomous mobile robot obtained when the position of the base station 120 was moved and the environment changed was input into the estimation model corresponding to the environment in Figure 3 to predict the RSSI, the RMSE was 6.36 dB, and the prediction error was large. Anomalies such as environmental changes can be detected based on the change in prediction error.

As described above, the terminal information output system 1 of this embodiment includes a collection unit 11 that collects environmental information obtained by sensing the space in which the wireless communication terminal exists, a wireless communication unit 12 that acquires actual values obtained by measuring the wireless communication quality of the wireless communication terminal, a detection unit 13 that detects candidate objects of the wireless communication terminal from the environmental information, an estimation unit 14 that uses a prediction model that has learned the relationship between the environmental information and the wireless communication quality by machine learning, inputs the environmental information into the prediction model, and outputs an estimated value of the wireless communication quality for each candidate object, and an extraction unit 15 that compares the estimated value and the actual measurement value for each candidate object, and extracts terminal information related to the candidate object with the highest estimation accuracy from the environmental information. This makes it possible to extract terminal information of the target wireless communication terminal present in the environmental information and use it to predict communication quality.

Each of the devices included in the terminal information output system 1 described above can be, for example, a general-purpose computer system including a central processing unit (CPU) 901, memory 902, storage 903, communication device 904, input device 905, and output device 906, as shown in Figure 7. In this computer system, each of the devices included in the terminal information output system 1 is realized by the CPU 901 executing a predetermined program loaded onto the memory 902. This program can be recorded on a computer-readable recording medium such as a magnetic disk, optical disk, or semiconductor memory, or can be distributed via a network.

Reference Signs List 1: Terminal information output system 11: Collection unit 12: Wireless communication unit 13: Detection unit 14: Estimation unit 15: Extraction unit 16: Model generation unit 17: Model selection unit 18: Prediction unit 19: Anomaly detection unit 20: Terminal information output device 110: Camera 120: Base station 130: Stationary robot 140: Autonomous mobile robot

Claims (7)

An information output system for outputting terminal information of a wireless communication terminal that performs wireless communication,
A collection device that collects environmental information by sensing a space in which the wireless communication terminal exists;
a communication device for acquiring an actual measurement value of wireless communication quality of the wireless communication terminal;
a detection unit that detects a candidate object of the wireless communication terminal from the environmental information;
an estimation unit that uses a prediction model that has been machine-learned to understand a relationship between the environmental information and wireless communication quality, inputs the environmental information into the prediction model, and outputs an estimated value of wireless communication quality for each of the candidate objects;
an extraction unit that compares the estimated value and the actual measured value for each candidate object, and extracts terminal information related to the candidate object with the highest estimation accuracy from the environmental information. The information output system according to claim 1,
an information output system comprising: a model generation device that inputs the environmental information and the wireless communication quality as training data, and performs machine learning on a relationship between the environmental information and the wireless communication quality to generate the prediction model. 3. The information output system according to claim 1,
An information output system comprising a prediction device that inputs the terminal information and predicts future wireless communication quality of the wireless communication terminal. 4. The information output system according to claim 1,
An information output system comprising: an anomaly detection device that detects an anomaly based on the estimation accuracy obtained by comparing the estimated value with the actual measured value. An information output method for outputting terminal information of a wireless communication terminal that performs wireless communication, comprising:
A collection device collects environmental information obtained by sensing a space in which the wireless communication terminal exists;
A communication device acquires an actual measurement value obtained by measuring a wireless communication quality of the wireless communication terminal;
The computer
Detecting a candidate object for the wireless communication terminal from the environmental information;
using a prediction model that has been machine-learned to understand the relationship between the environmental information and wireless communication quality, inputting the environmental information into the prediction model and outputting an estimated value of wireless communication quality for each of the candidate objects;
the estimated value and the actual measured value for each of the candidate objects are compared, and terminal information relating to the candidate object with the highest estimation accuracy is extracted from the environmental information. An information output device that outputs terminal information of a wireless communication terminal that performs wireless communication,
a detection unit that detects a candidate object of the wireless communication terminal from environmental information obtained by sensing a space in which the wireless communication terminal exists;
an estimation unit that uses a prediction model that has been machine-learned to understand a relationship between the environmental information and wireless communication quality, inputs the environmental information into the prediction model, and outputs an estimated value of wireless communication quality for each of the candidate objects;
an extraction unit that compares the estimated value for each candidate object with an actual measurement value of wireless communication quality of the wireless communication terminal, and extracts terminal information related to the candidate object with the highest estimation accuracy from the environmental information. A program for causing a computer to operate as each part of the information output device described in claim 6.

Images