JP7502682B2 - Information output system, information output method, estimation device, and program - Google Patents

Description

The present invention relates to an information output system, an information output method, an estimation device, and a program.

The Internet of Things (IoT), in which various devices are connected to the Internet, is becoming a reality, and various devices such as automobiles, drones, and construction machinery vehicles are being connected wirelessly. Wireless communication standards include wireless LAN defined by the standard IEEE 802.11, Bluetooth (registered trademark), cellular communication using LTE and 5G, Low Power Wide Area (LPWA) communication for IoT, Electronic Toll Collection System (ETC), Vehicle Information and Communication System (VICS), and ARIB-STD-T109, and are expected to become more widespread in the future.

While wireless communication is used for various purposes, it is problematic that, depending on the service, the requirements for wireless communication quality cannot always be met. The radio wave propagation environment between the communication partner changes depending on the terminal state such as the position, attitude, and movement of the wireless communication terminal, and the static or dynamic objects around it, affecting the communication quality and having a significant impact on the service or system realized by the wireless communication by the wireless communication terminal. In particular, the higher the frequency used, the stronger the directivity of the radio wave, and the more susceptible it is to the influence of the communication quality. In other words, the wireless communication of the wireless communication terminal is affected by both the state of itself and the surrounding environment. For example, the movement of the wireless communication terminal or surrounding objects may change the direction of the antenna directivity and the propagation environment, which may affect the communication quality. Non-Patent Document 1 shows that it is possible to predict that objects around the wireless communication terminal will block the communication path using a depth camera, and the usefulness of image information obtained from cameras, sensors, etc. is also attracting attention in wireless communication.

Nishio, Takayuki, et al. "Proactive received power prediction using machine learning and depth images for mmWave networks." IEEE Journal on Selected Areas in Communications 37.11 (2019): 2413-2427. J. Redmon and A. Farhadi, "Yolov3: An incremental improvement," CoRR, vol. abs/1804.02767, Apr. 2018.

However, in order to take into account the impact on communication conditions caused by changes in the antenna directivity and propagation environment due to the movement of the wireless communication terminal or surrounding objects, it is necessary to know the position and status of the wireless communication terminal.

The present invention has been made in consideration of the above, and aims to extract information in the real space of a wireless communication terminal performing wireless communication.

An information output system of one embodiment of the present invention is an information output system that outputs status information of a wireless communication terminal performing wireless communication, and includes a collection device that collects real world information sensed in a space in which the wireless communication terminal is located, a communication device that acquires wireless communication information indicating the wireless communication quality of the wireless communication terminal, and an estimation device that inputs the real world information and the wireless communication information and extracts status information of the wireless communication terminal from the real world information based on the relationship between the real world information and the wireless communication information, wherein the estimation device detects object information related to wireless communication from the real world information, and extracts status information of the wireless communication terminal from the real world information by inputting the detected object information and the wireless communication information of the wireless communication terminal into a real world/communication model that has been machine-learned .

According to the present invention, it is possible to extract information in the real space of a wireless communication terminal performing wireless communication.

FIG. 1 is a diagram showing an example of the configuration of a state information output system according to the present embodiment. FIG. 2 is a diagram illustrating an example of a configuration of an estimation device. FIG. 3A is a diagram showing an example of an image to be input to the estimation device. FIG. 3B is a diagram showing an example of a detection result obtained by detecting an object from FIG. 3A. FIG. 3C is a diagram showing an example in which a target object is extracted from the detection result of FIG. 3B. FIG. 4 is a diagram illustrating an example of another configuration of the estimation device. FIG. 5 is a flowchart showing an example of a process flow of the estimation device. FIG. 6 is a plan view of the indoor experimental area. FIG. 7 shows an image of the experimental area obtained from a camera. FIG. 8 is a diagram showing the evaluation results of the prediction performance. FIG. 9 is a diagram showing an example of a hardware configuration of a device included in the status information output system.

An embodiment of the present invention will be described with reference to the drawings. The embodiment described below is an example of the present invention, and the present invention is not limited to the following embodiment. Note that components with the same reference numerals in this specification and drawings are considered to be identical to each other.

First, referring to FIG. 1, an example of the configuration of the state information output system 1 of this embodiment will be described. The state information output system 1 shown in FIG. 1 includes an estimation device 10, a communication information collection unit 20, a real-world information collection unit 30, and a model generation device 40. The state information output system 1 is a system that extracts and outputs state information of a wireless communication terminal that performs wireless communication from real-world data acquired by a camera or sensor. The state information is information that indicates the state of the wireless communication terminal, such as the position, attitude, speed, orientation, velocity, acceleration, angular velocity, usage form, and holding state of the wireless communication terminal, for example.

The communication information collection unit 20 collects wireless communication information of the wireless communication terminal. The wireless communication information is information related to wireless communication, such as the identification number of the communication terminal that is the communication partner, the communication method, radio wave propagation information of the wireless communication signal, channel relationship information related to the radio wave propagation information, feedback information via the wireless communication signal, received signal power, signal-to-noise power ratio, signal-to-interference-plus-noise power ratio, Received Signal Strength Indication (RSSI), Received Signal Reference Quality (RSRQ), Quality Of Service (QoS), Quality Of Experience (QoE), packet error rate, number of arriving bits, bit error rate, number of arriving bits per unit time, modular code It represents the index (MCS), the number of retransmissions, packet arrival delay time, error correction technology settings, communication terminal user contract information, differential information of these values, and indices calculated from these values using a formula, frequency conditions such as the frequency of the wireless communication system and the bandwidth of the resources used, information on the communication network to which the wireless communication system is connected, and communication system settings that affect these indices.

A plurality of communication information collection units 20 may be provided for each wireless system. For example, the status information output system 1 may include a communication information collection unit 20 for each wireless system such as wireless LAN, LTE, and 5G, and each communication information collection unit 20 collects the communication quality of each wireless system. Wireless communication information is obtained as information on the wireless system or the wired network to which the wireless system is connected.

The real-world information collection unit 30 collects real-world information such as the physical positions and movements of objects (including wireless communication terminals) that exist in the real world from sensors. Real-world information is sensing data of the space in which the target wireless communication terminal exists, and is, for example, information on the physical positions and movements of two-dimensional or three-dimensional ground, walls, structures, objects, etc. obtained from sensors such as RGB cameras, depth cameras, frame difference/background difference cameras, and LiDAR. The real-world information may further include static map information and spatial semantic information such as stores, products, intersections, and charging locations. The real-world information collection unit 30 may collect real-world information from multiple sensors, or may collect multiple types of real-world information. Real-world information is also called environmental information or sensor information.

The model generation device 40 inputs the wireless communication information collected by the communication information collection unit 20 and the real world information collected by the real world information collection unit 30, and generates a real world/communication model that represents the relationship between the wireless communication information and the real world information. In other words, as machine learning, the wireless communication information and real world information are input as training data, and training is performed to update the coefficients of the real world/communication model that outputs parameters related to the real world or wireless communication information, and output the data. For example, when wireless communication information of a target wireless communication terminal and real world information obtained by sensing the environment in which the target wireless communication terminal exists are input to the real world/communication model, real world information related to the target wireless communication terminal is obtained. The wireless communication information and real world information may be generated in a simulation space, or may be measured data.

Machine learning can use any machine learning algorithm, such as neural networks, decision trees, random forests, etc. Machine learning can also use unsupervised learning, semi-supervised learning, and supervised learning.

When unsupervised learning is used, the estimation device 10 performs clustering, grouping, and latent structure analysis on data that combines real-world information and wireless communication information, classifies the relationship between the real-world information and the wireless communication information, selects information from the real-world information that has the same cluster, group, and feature as the target wireless communication terminal, and extracts state information such as the position, posture, and movement of events that are closely related to the wireless communication terminal in the real world, that is, the position, posture, and movement of the wireless communication terminal, its owner, or its inclusion. As a specific example, an RGB image is acquired as real-world information, the relationship between the RGB image and the wireless communication information obtained from the wireless communication terminal is learned, and the part that is closely related in the image is extracted to determine the wireless communication terminal, the owner of the wireless communication terminal, or the inclusion of the wireless communication terminal. Hereinafter, the person (owner) who holds the wireless communication terminal and the inclusion such as a vehicle or robot equipped with the wireless communication terminal may be simply referred to as a wireless communication terminal. This takes advantage of the fact that the movement of a wireless communication terminal affects wireless communication information, and that when the wireless communication terminal moves, the conditions for radio wave propagation change, causing changes in the wireless communication information to be visible.By looking at relationships over time, for example, it is possible to extract which real-world information has a strong influence on the wireless communication information of the wireless communication terminal to be obtained, and thereby identify the wireless communication terminal.

When semi-supervised learning is used, the model generation device 40 generates an object detection model using an object detection algorithm (see Non-Patent Document 2) trained to detect objects related to wireless communication. An object related to wireless communication is an object that may be equipped with a wireless communication terminal, such as a person, a car, a motorcycle, a bicycle, a robot, or a drone, or may affect radio wave propagation, and also includes objects that do not ultimately affect wireless communication quality. The estimation device 10 classifies the relationship between the object information (for example, the time series data of the detected object class and bounding box information) obtained by inputting real-world information into the object detection model and the wireless communication information, selects information from the object information that is the same cluster, group, and feature as the target wireless communication terminal, and can know the class of the wireless communication terminal and its position on the video. In addition, as a different form of semi-supervised learning, the above-mentioned unsupervised learning or semi-supervised learning may be used, and the wireless communication terminal real-world information on the extracted wireless communication terminal real-world information may be labeled as related to the wireless communication terminal, and the communication/real-world model may be trained by supervised learning so that the real-world information of the newly generated label can be extracted.

When using supervised learning, the model generation device 40 generates camera images and sensor information after knowing in advance what the wireless communication terminal is detected as by the camera or sensor, models the relationship between the wireless communication information for the wireless communication terminal and the obtained real-world information by machine learning, and trains to extract status information of the wireless communication terminal in the real-world information from any wireless communication information. For example, an RGB image is acquired as real-world information, and an object detection algorithm trained with a separately prepared RGB image with class information of the wireless communication terminal is used to model the time series data of the bounding box information of the obtained object with the wireless communication information of the wireless communication terminal by supervised learning. In other words, in semi-supervised learning, object information corresponding to the wireless communication terminal is extracted using an object related to wireless communication, whereas in supervised learning, the relationship between the wireless communication information is learned using object information corresponding to the wireless communication terminal obtained after identifying the wireless communication terminal by the camera or sensor. By training the real-world/communication model to output the bounding box of the wireless communication terminal using wireless communication information and real-world information, a real-world/communication model is formed so that status information of the target wireless communication terminal existing in the real-world information can be extracted using the real-world information and wireless communication information.

The estimation device 10 uses the real world/communication model generated by the model generation device 40 to extract the state of the target wireless communication terminal from the real world information based on the relationship between the real world information and the wireless communication information. For example, when wireless communication information of the target wireless communication terminal is being observed, when the wireless communication information of the observed target wireless communication terminal and real world information sensed from the real world are input to the estimation device 10, the estimation device 10 extracts state information such as the position, attitude, and movement of the target wireless communication terminal from the real world information.

Figure 2 shows an example of the configuration of the estimation device 10. The estimation device 10 shown in the figure includes a detection unit 11 and an extraction unit 12.

The detection unit 11 inputs video information as real-world information and detects the state of an object related to wireless communication, such as the position and speed. The detection unit 11 may use a machine-learned model to detect an object related to wireless communication from real-world information. For example, when a surveillance camera image shown in FIG. 3A is input to the detection unit 11, the detection unit 11 detects the position of a car in the image as shown in FIG. 3B. In FIG. 3B, the object detected by the detection unit 11 is shown by a bounding box. Many cars exist in the image, and many cars are detected by the detection unit 11. The detection unit 11 may input continuous images in a time series and detect various states, such as the traveling direction and speed of the car. In addition to the image, the real-world information extraction unit may further refer to, for example, location information reported from a wireless communication terminal in the vicinity of the image. It is also possible to obtain low-precision location information, such as GPS, from a wireless communication terminal from which the state of the wireless communication terminal is to be extracted, and input it to the extraction unit 12, and to increase the accuracy of the location information as the state of the wireless communication terminal.

The extraction unit 12 uses the real world/communication model to extract, from the objects detected by the detection unit 11, objects that are highly correlated with the wireless communication information of the target wireless communication terminal being observed, and outputs status information such as the position and speed of the extracted objects. Figure 3C shows the result of extracting objects from the detection results in Figure 3B. The car shown by the bounding box in Figure 3C is the car equipped with the target wireless communication terminal. The wireless communication information of the target wireless communication terminal is obtained, for example, from a base station that receives a signal from the target wireless communication terminal.

The extraction unit 12 may classify the relationship between the objects detected by the detection unit 11 and the wireless communication information, extract from the objects those objects that have the same cluster, group, or features as the target wireless communication terminal, and output status information such as the position and speed of the extracted objects.

The state of the wireless communication terminal obtained by the estimation device 10 can be used for the communication quality prediction described in Non-Patent Document 1, or can be used to predict the next behavior of the owner/inclusion of the wireless communication terminal, such as starting communication, based on the state of the target wireless communication terminal and the behavioral history and preferences of the owner/inclusion of the wireless communication terminal. Alternatively, the predicted behavior can be used to control base station switching, bandwidth allocation, and base station antenna and beam pattern allocation. For example, if it is known that a wireless communication terminal will soon be in a condition with poor communication quality, it is possible to switch base stations in advance, increase the allocated bandwidth, or assign better antennas and beam patterns to the wireless communication terminal.

Figure 4 shows an example of another configuration of the estimation device 10. The estimation device 10 shown in the figure is the estimation device 10 of Figure 2 equipped with a prediction unit 13.

The prediction unit 13 inputs the wireless communication information of the target wireless communication terminal and the state information of the target wireless communication terminal extracted by the extraction unit 12 from the real world information, and predicts the future wireless communication information of the target wireless communication terminal using a real world/communication model. The real world/communication model used by the prediction unit 13 can be a real world/communication model that represents the relationship between the wireless communication information generated by the model generation device 40 and the real world information. The prediction unit 13 may predict the future wireless communication information using only the real world information corresponding to the wireless communication terminal input from the extraction unit 12, or may predict the future wireless communication information using the real world information corresponding to the wireless communication terminal extracted by the extraction unit 12 and real world information input from other detection units 11 that are not the wireless communication terminal, or may predict the future wireless communication information including the current or past wireless communication information acquired by the communication information collection unit 20. The output of the prediction unit 13 may not be a prediction result of future wireless communication quality, but may be wireless communication control information for improving the prediction result. The control information includes, for example, base station switching, band allocation, selection of data rate, modulation method, error correction code, spatial multiplexing number, communication mode, and allocation of base station antennas and beam patterns.

Next, referring to the flowchart in Figure 5, the processing flow of the estimation device 10 will be explained.

In step S11, the detection unit 11 inputs the image of Figure 3A, detects objects from the image, and outputs the detection result shown in Figure 3B.

In step S12, the extraction unit 12 inputs the detection result in FIG. 3B and the wireless communication information of the target wireless communication terminal, and extracts the state of the target wireless communication terminal. The relationships between all objects obtained in step S11 can be classified by unsupervised learning, or identified by supervised learning that learns from training data of object detection results and wireless communication information measured in advance. In addition, in step S12, only objects that are likely to be related to the communication terminal may be evaluated. If it is known in advance that the wireless communication terminal is a car, the target wireless communication terminal can be extracted using only the detection results related to the car. In step S11, if the object is extracted using a model that can also identify the wireless communication terminal, step S12 can be omitted.

In step S13, the prediction unit 13 inputs the wireless communication information of the target wireless communication terminal and the state extracted by the extraction unit 12, and predicts future wireless communication information of the target wireless communication terminal. Step S13 may output control information for the wireless communication information instead of predicting the wireless communication information. Also, if the purpose of the system is to estimate the state of the wireless communication terminal in the real world, the output of step S12 may be used and step S13 may be omitted.

Next, with reference to Figures 6 to 8, we will explain the experiments conducted to demonstrate the effects of this embodiment and their results.

The wireless communication terminal used in the experiment was an autonomous mobile robot equipped with LiDAR, which moved randomly within the experimental area shown in Figure 6. The autonomous mobile robot can collect its own position information as terminal information. The self-position is calculated by Adaptive Monte Carlo Localization (AMCL) of Robot OS from the measurement results of odometry and LiDAR.

Two cameras 110, 120 and a base station 130 were installed within the experimental area. The experimental area was photographed by the cameras 110, 120. Figure 7 shows an image obtained from the camera 110. The image shows an autonomous mobile robot 140 and a stationary robot 150. The stationary robot 150 is another robot placed in the center of the experimental area. Both the autonomous mobile robot 140 and the stationary robot 150 are detected in the image, and bounding boxes are obtained for each detected position.

The base station 130 performs wireless communication with the autonomous mobile robot. Wireless communication uses wireless LAN (IEEE802.11ac), and the median RSSI value was measured for 0.2 seconds.

By grouping the relationship between the four pieces of information, the position (X, Z) in the image of the bounding box, its width W, and its height H, and the target's RSSI using unsupervised learning, it can be concluded that the bounding box information of an autonomous mobile robot has a high relationship with the target's RSSI. In this way, if wireless communication information such as the RSSI for a certain wireless communication terminal is obtained, it is possible to derive a bounding box corresponding to the target wireless communication terminal or the owner/inclusion of the wireless communication terminal by calculating the relationship between the wireless communication information and real-world information such as the bounding box of an object detected in the image using unsupervised learning.

Next, the prediction accuracy of wireless communication information in which future RSSI is predicted using the derived bounding box will be described. Here, prediction accuracy was evaluated for the following cases: when the position of an autonomous mobile robot is extracted using images obtained by two cameras according to this embodiment and future RSSI is predicted; when future RSSI is predicted using the results of highly accurate self-location estimation by LiDAR mounted on an autonomous mobile robot; and when future RSSI is predicted from past RSSI information. Note that the prediction model for predicting RSSI from the position of an autonomous mobile robot was constructed using random forest.

Figure 8 shows the evaluation of prediction accuracy for each method at future times. The horizontal axis of the graph is the predicted future time, and the vertical axis is the evaluation of prediction accuracy. The prediction accuracy on the vertical axis was evaluated using the R2 score, which is expressed as the following formula.

where Ri is the RSSI value for the i-th sample, Ri with a hat is the RSSI value predicted by the random forest, and Rave is the average RSSI value for the data used. A higher R2 indicates better prediction performance.

In Figure 8, the graph plotted with a solid line is an evaluation of the prediction accuracy when predicting future RSSI using the bounding box of the autonomous mobile robot obtained by this embodiment. The graph plotted with a dotted line is an evaluation of the prediction accuracy when predicting future RSSI using the self-position of the autonomous mobile robot. The graph plotted with a dashed dotted line is an evaluation of the prediction accuracy when predicting future RSSI from past RSSI information.

From Figure 8, it can be seen that the communication quality predicted by this embodiment has the highest prediction performance. The same prediction performance can be obtained by using highly accurate location information, but this requires frequently obtaining feedback of location information from the wireless communication terminal, which can strain the wireless communication line. Although RSSI can be measured on the base station side, prediction performance is significantly reduced when predicting from past RSSI.

As described above, the status information output system 1 of this embodiment includes a real world information collection unit 30 that collects real world information obtained by sensing the space in which the wireless communication terminal exists, a communication information collection unit 20 that acquires the wireless communication quality of the wireless communication terminal, and an estimation device 10 that inputs the real world information and the wireless communication quality and extracts status information of the wireless communication terminal in the real world information based on the relationship between the real world information and the wireless communication quality. This makes it possible to extract status information of the wireless communication terminal performing wireless communication from the real world information.

The status information output system 1 of this embodiment includes a prediction unit that predicts future wireless communication information from status information of the wireless communication terminal extracted from the real world information, based on the relationship between the real world information and the wireless communication information. This makes it possible to predict future wireless communication information from the real world information and the wireless communication information.

The device of the above-described status information output system 1 may be, for example, a general-purpose computer system including a central processing unit (CPU) 901, a memory 902, a storage 903, a communication device 904, an input device 905, and an output device 906, as shown in FIG. 9. In this computer system, the device of the status information output system 1 is realized by the CPU 901 executing a predetermined program loaded on the memory 902. This program can be recorded on a computer-readable recording medium such as a magnetic disk, an optical disk, or a semiconductor memory, or can be distributed via a network. In addition, the device of the status information output system 1 may be realized using hardware such as a programmable logic device (PLD) and a field programmable gate array (FPGA).

Reference Signs List 1: State information output system 10: Estimation device 11: Detection unit 12: Extraction unit 13: Prediction unit 20: Communication information collection unit 30: Real world information collection unit 40: Model generation device

Claims (9)

An information output system for outputting status information of a wireless communication terminal that performs wireless communication,
A collection device that collects real world information by sensing a space in which the wireless communication terminal exists;
a communication device that acquires wireless communication information indicating a wireless communication quality of the wireless communication terminal;
an estimation device that receives the real world information and the wireless communication information, and extracts state information of the wireless communication terminal from the real world information based on a relationship between the real world information and the wireless communication information ;
The estimation device detects object information related to wireless communication from the real world information, and inputs the detected object information and wireless communication information of the wireless communication terminal into a machine-learned real world/communication model, thereby extracting state information of the wireless communication terminal from the real world information.
Information output system. The information output system according to claim 1,
The estimation device classifies the relationship between real world information and wireless communication information, and extracts real world information corresponding to the wireless communication information of the wireless communication terminal as status information of the wireless communication terminal. The information output system according to claim 1,
The estimation device detects object information related to wireless communication from the real world information, classifies the relationship between the detected object information and the wireless communication information, and extracts object information corresponding to the wireless communication information of the wireless communication terminal as status information of the wireless communication terminal. The information output system according to claim 1,
and a model generation device that inputs object information corresponding to the wireless communication terminal detected from real world information and the wireless communication information as training data, and performs machine learning on the relationship between the object information and the wireless communication information to generate the real world/communication model.
Information output system. 5. An information output system according to claim 1,
The estimation device predicts future wireless communication information of the wireless communication terminal from status information of the wireless communication terminal based on the relationship between real world information and wireless communication quality, and outputs the future wireless communication quality or a wireless communication control method. 1. An information output method for outputting status information of a wireless communication terminal that performs wireless communication, comprising:
a collection device that collects real world information obtained by sensing a space in which the wireless communication terminal exists;
A communication device acquires wireless communication information indicating a wireless communication quality of the wireless communication terminal;
an estimation device inputting the real world information and the wireless communication information, and extracting state information of the wireless communication terminal from the real world information based on a relationship between the real world information and the wireless communication information;
The estimation device detects object information related to wireless communication from the real world information, and inputs the detected object information and wireless communication information of the wireless communication terminal into a machine-learned real world/communication model, thereby extracting state information of the wireless communication terminal from the real world information.
Information output method. An estimation device that outputs state information of a wireless communication terminal that performs wireless communication,
a detection unit that detects object information related to wireless communication from real world information obtained by sensing a space in which the wireless communication terminal exists;
an extraction unit that receives wireless communication information indicating a wireless communication quality of the wireless communication terminal, and extracts, from the object information, object information that is highly correlated with the wireless communication information of the wireless communication terminal based on a relationship between real world information and the wireless communication information, as status information of the wireless communication terminal ;
The extraction unit inputs the object information and the wireless communication information of the wireless communication terminal into a machine-learned real world/communication model, and extracts state information of the wireless communication terminal from the real world information.
Estimation device. The estimation device according to claim 7,
An estimation device comprising: a prediction unit that predicts future wireless communication information of the wireless communication terminal from state information of the wireless communication terminal based on a relationship between real world information and wireless communication information, and outputs future wireless communication quality or a wireless communication control method. A program for causing a computer to operate as each part of the estimation device according to claim 7 or 8.

Images