JP7642943B2 - Wireless communication method, wireless terminal, and program for wireless terminal - Google Patents

Description

This disclosure relates to a wireless communication method, a wireless terminal, and a program for the wireless terminal, and in particular to a wireless communication method, a wireless terminal, and a program for the wireless terminal that are suitable for achieving high communication efficiency in a network that includes multiple wireless terminals and multiple wireless base stations, etc.

As wireless communication systems develop, heterogeneous networks using multi-band, multi-access systems are reaching the stage of practical application. In cellular communications, fifth-generation mobile communications (5G) has been put to practical use. 5G communications systems use a wide range of frequencies, from sub-1 GHz bands to millimeter wave bands. In addition, an environment is being developed in which cells of various sizes, from small cells to macro cells, are provided in an overlapping manner.

Wireless LANs, another typical wireless access system, use radio frequencies in the 2.4/5/60 GHz bands, and the use of the 6 GHz band is also being considered. Wireless terminals such as smartphones are generally provided with interfaces (IFs) for accessing both cellular and wireless LANs. Furthermore, each IF is configured to support multiple frequency bands.

Wireless terminals generally select a wireless base station or access point (AP) to connect to from multiple frequency bands and access methods to establish communications. With functions such as dual connectivity, a single wireless terminal can also use multiple wireless base stations in an integrated manner. In such a heterogeneous environment, it is important to control and optimize across the entire system which IF a wireless terminal uses and which base station it selects in order to make effective use of system resources.

Regarding technology for effectively utilizing system resources in a heterogeneous environment, for example, Non-Patent Documents 1 and 2 below disclose a method for optimizing connections between wireless terminals and APs through reinforcement learning. More specifically, Non-Patent Document 2 discloses a technology for optimizing connections between wireless terminals using multiple IFs and multiple APs using Deep Q-Learning or Double Deep Q-Learning.

In addition, the following non-patent document 3 discloses Deep Q-Network (DQN), a method for handling high-dimensional inputs using reinforcement learning, and the following non-patent document 4 discloses Double Deep Q-Network (DDQN).

“Reinforcement Learning-aided Distributed User-to-Access Points Association in Interfering Networks”, Thi Ha Ly Dinh, Megumi Kaneko, Keisuke Wakao, Hirantha Abeysekera, Yasushi Takatori, IEEE Globecom, Waikoloa, Hawaii, USA, Dec. 2019 “Deep Reinforcement Learning-based User Association in Sub6GHz/mmWave Integrated Networks”, Thi Ha Ly Dinh, Megumi Kaneko, Keisuke Wakao, Kenichi Kawamura, Takatsune Moriyama, Hirantha Abeysekera and Yasushi Takatori, in IEEE CCNC, Jan. 2021 “Human-level control through deep reinforcement learning,” Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg & Demis Hassabis, Nature. 2015 Feb 26; Vol.518(7540): P529-533. doi: 10.1038/nature14236 “Deep Reinforcement Learning with Double Q-Learning,” H. Van Hasselt, A. Guez, and D. Silver, in Thirtieth AAAI conference on artificial intelligence, 2016

In a multi-band, multi-access heterogeneous environment, mobile terminals such as smartphones and tablets may be used as wireless terminals. The communication environment surrounding the mobile terminal changes as the mobile terminal moves. In addition, when many wireless terminals exist in the same area, changes in the requests issued by each wireless terminal affect the load on the wireless terminals and wireless base stations belonging to that area. As a result, the communication environment surrounding each wireless terminal changes significantly depending on the state of other wireless terminals. For these reasons, the optimal solution regarding which wireless base station and which IF a wireless terminal placed in a heterogeneous environment should connect to may not be a stable one, but rather a dynamic one that changes drastically.

When trying to optimize connections between wireless terminals and wireless base stations using reinforcement learning such as DQN or DDQN, if the communication environment changes, it is necessary to update the reinforcement learning parameters to adapt to the changed environment. Furthermore, in an environment where the communication environment changes frequently, such updates need to be performed frequently.

In an environment where a wireless terminal is connected to many wireless base stations using multiple IFs, the greater the number of wireless base stations and the number of IFs, the greater the load caused by the above updates. Because the processing capacity of a wireless terminal is finite, as the number of wireless base stations and the number of available IFs increase, it becomes difficult for the wireless terminal to keep up with the frequent parameter updates.

The present disclosure has been made in light of the above-mentioned problems, and aims to provide a wireless communication method, a wireless terminal, and a program for a wireless terminal that can reduce the computational load of updating parameters required for optimization in a network including multiple wireless terminals and multiple base stations, etc.

In order to achieve the above object, a first aspect of the present disclosure is a wireless communication method for determining a combination of connections between a plurality of wireless base stations and a wireless terminal, the method comprising:
preparing a neural network that includes an input layer and an output layer corresponding to a plurality of wireless base stations, and a hidden layer including two or more layers located therebetween, and functions as a value function that outputs, in response to an input of a state, a value of each action that the wireless terminal can take under the state;
detecting a state of connection between the plurality of wireless base stations and the wireless terminal;
providing the sensed state to the input layer and determining, according to a result appearing at the output layer, a behavior to be adopted by the wireless terminal under the sensed state;
transmitting a request to said plurality of radio base stations corresponding to an action to be taken;
obtaining responses to the request from a plurality of wireless base stations;
calculating a reward for the behavior based on the response;
updating a value for adopting the behavior under the state based on the reward;
learning parameters of the value function based on the updated values;
limiting the update range of the parameters to parameters associated with one or more layers in the hidden layer, excluding at least the frontmost layer;
It is preferable that the present invention includes the following:

A second aspect of the present disclosure is a wireless terminal having a function of connecting to a plurality of wireless base stations,
The neural network includes an input layer and an output layer corresponding to a plurality of wireless base stations, and a hidden layer including two or more layers located between them, and includes a memory unit storing a neural network that functions as a value function that outputs, in response to an input of a state, a value of each of the actions that the wireless terminal can take under the state, and a CPU;
The CPU,
A process of detecting a state of connection between the plurality of wireless base stations and the wireless terminal;
providing the sensed state to the input layer and determining the action to be taken by the wireless terminal under the sensed state according to the result appearing in the output layer;
transmitting a request corresponding to an action to be taken to said plurality of wireless base stations;
obtaining responses to the request from a plurality of wireless base stations;
calculating a reward for the behavior based on the response; and
updating a value for taking the action under the state based on the reward;
A process of learning parameters of the value function based on the updated value;
A process of limiting the update range of the parameters to parameters associated with one or more layers in the hidden layer, excluding at least the frontmost layer;
It is advisable to carry out the following:

A third aspect of the present disclosure is a program for a wireless terminal for realizing a function of connecting to a plurality of wireless base stations, comprising:
A CPU provided in the wireless terminal,
a process of reading from a memory unit a neural network that includes an input layer and an output layer corresponding to a plurality of wireless base stations, and a hidden layer including two or more layers located between them, and that functions as a value function that outputs, in response to an input of a state, a value of each action that the wireless terminal can take under the state;
A process of detecting a state of connection between the plurality of wireless base stations and the wireless terminal;
providing the sensed state to the input layer and determining the action to be taken by the wireless terminal under the sensed state according to the result appearing in the output layer;
transmitting a request corresponding to an action to be taken to said plurality of wireless base stations;
obtaining responses to the request from a plurality of wireless base stations;
calculating a reward for the behavior based on the response; and
updating a value for taking the action under the state based on the reward;
A process of learning parameters of the value function based on the updated value;
A process of limiting the update range of the parameters to parameters associated with one or more layers in the hidden layer, excluding at least the frontmost layer;
It is preferable that the program include a program for executing the above.

According to the present disclosure, in a network including multiple wireless terminals and multiple base stations, etc., the parameter update range of a neural network that determines the behavior of wireless terminals can be limited to parameters linked to layers subsequent to the hidden layer. Therefore, according to the present disclosure, it is possible to reduce the calculation load associated with parameter updates compared to when all parameters are updated. As a result, even if the number of wireless base stations or the number of wireless communication interfaces used there increases, it becomes possible to update parameters frequently, making it possible to always effectively utilize system resources.

1 is a diagram for explaining an overall configuration of a wireless communication system according to a first embodiment of the present disclosure. FIG. 2 is a diagram for explaining an example of a connection state between a wireless terminal and a wireless base station in the first embodiment of the present disclosure. FIG. 2 is a diagram illustrating a wireless terminal moving in a heterogeneous environment according to the first embodiment of the present disclosure. FIG. 2 is a block diagram for explaining a functional configuration of a radio base station used in the first embodiment of the present disclosure. FIG. 2 is a block diagram for explaining a functional configuration of a wireless terminal used in the first embodiment of the present disclosure. 2 is a diagram for explaining the hardware configuration of a wireless terminal used in the first embodiment of the present disclosure. FIG. FIG. 1 is a diagram for explaining an overview of DQN used in the first embodiment of the present disclosure. FIG. 13 is a diagram for explaining the contents of a process related to updating a DQN used in the first embodiment of the present disclosure. FIG. 13 is a diagram for explaining the initialization of DQN parameters used in the first embodiment of the present disclosure. FIG. 13 is a diagram for explaining partial update of parameters of DQN used in the first embodiment of the present disclosure. 11 is a flowchart for explaining the processing flow when a wireless terminal in the first embodiment of the present disclosure initializes DQN parameters. 11 is a flowchart for explaining a processing flow when a wireless terminal according to the first embodiment of the present disclosure partially updates a DQN parameter.

Embodiment 1.
[Overall Configuration of First Embodiment]
Fig. 1 is a diagram for explaining an overall configuration of a wireless communication system 10 according to a first embodiment of the present disclosure. As shown in Fig. 1, the wireless communication system 10 includes a wired communication network 12. A plurality of wireless base stations 16 are connected to the network 12 via a switch 14. Each wireless base station 16 can communicate with another wireless base station 16 via the network 12 and via a wireless path.

The wireless communication system 10 shown in FIG. 1 includes multiple wireless terminals 18. In this embodiment, the wireless terminals 18 include mobile terminals such as smartphones or tablet terminals.

Each wireless terminal 18 has multiple sets of applications and IFs for wireless communication. For example, wireless terminal 18 has applications and IFs used for communication at 1 GHz or less, applications and IFs used for communication in the millimeter wave band, and applications and IFs used for communication in the 2.4/5/60 GHz bands used in wireless LANs.

The above-mentioned wireless base stations 16 are also equipped with corresponding IFs, etc. Then, each wireless terminal 18 can wirelessly communicate with one or more wireless base stations 16 via the above-mentioned applications and IFs. In this way, the wireless communication system 10 of this embodiment realizes the function of a heterogeneous network with multi-band multi-access.

The wireless communication system 10 of this embodiment is compatible with wireless LAN access points (APs) in addition to wireless base stations for cellular communication installed by telecommunications carriers. For the sake of convenience, the following description will be given assuming that "wireless base station 16" encompasses both.

[Features of the First Embodiment]
Next, the features of the wireless terminal 18 in this embodiment will be described with reference to Figures 2 to 12. In the following description, it is assumed that the wireless base station 16 and the wireless terminal 18 are provided with an interface IF1 corresponding to communication in a first frequency band and an interface IF2 corresponding to communication in a second frequency band. IF1 is an interface corresponding to a frequency band of, for example, 1 GHz or less, and IF2 is an interface corresponding to frequencies in the millimeter wave band, for example.

In the following description, when it is necessary to indicate a specific wireless terminal 18 among a plurality of terminals, the reference symbol k will be used to indicate the wireless terminal 18 _k . Furthermore, when it is necessary to distinguish between the wireless base stations 16, reference symbols with suffixes, such as 16-1 and 16-2, will be used.

Fig. 2 shows a state where a wireless terminal 18 _k belongs to two wireless base stations 16-1 and 16-2. In the communication environment surrounding the wireless terminal 18 _k , the wireless terminal 18 k can handle an upper limit number B _BU of wireless base stations 16 as communication targets, which is determined by hardware constraints. Fig. 2 shows a state where the wireless terminal 18 _k is connected to the wireless base stations 16-1 and 16-2 out of the B _BU wireless base stations 16.

As described above, the wireless terminal 18 _k and the wireless base station 16 have two interfaces, IF1 and IF2, corresponding to different bands. In the example shown in Fig. 2, the wireless terminal 18 _k is associated with the first wireless base station 16-1 at IF1 and is associated with the second wireless base station 16-2 at IF2.

3 shows a state where a wireless terminal 18 _k is moving in a heterogeneous environment with multi-band multi-access. First, assume an environment in which the wireless terminal 18 _k is located at the leftmost position in the figure. In this environment, if it is optimal for the wireless terminal 18 _k to belong to the wireless base stations 16-1 and 16-2, it is desirable for the connection request from the wireless terminal 18 _k to be issued so as to realize that state.

If the wireless terminal _18k moves from the leftmost position in the figure to the center of the figure, or further to the right side of the figure, the optimal connection combination will change from the original combination. Similar changes also occur when other wireless terminals 18 (not shown) belonging to the same environment as the wireless terminal _18k move, or when the requested data rates from them change. For this reason, when the communication environment changes, it is desirable for the wireless terminal _18k to issue a connection request with a combination that is optimal for the changed environment.

In this embodiment, the wireless terminal 18 determines the wireless base station 16 and IF to which it wishes to connect using Deep Q-Network (DQN) or Double Deep Q-Network (DDQN). For this reason, it is desirable that the DQN or DDQN used here quickly converges to an optimal state for a stable communication environment, and quickly follows environmental changes in a dynamic communication environment.

Figure 4 is a block diagram for functionally explaining an example of a wireless base station 16. The wireless base station 16 is equipped with an antenna unit 20. The antenna unit 20 is used by the wireless base station 16 to transmit and receive wireless signals to and from other wireless base stations 16 and surrounding wireless terminals 18.

The antenna unit 20 is connected to the wireless communication unit 22. The wireless communication unit 22 includes an RF (Radio Frequency) unit 24 that processes high-frequency signals, and a MAC (Media Access Control) function unit 26 that has a function of media access control in wireless communication. The wireless communication unit 22 is configured to be able to realize functions corresponding to multiple frequency bands, specifically, a function as IF1 corresponding to 1 GHz or less, and a function as IF2 corresponding to millimeter wave band frequencies.

The wireless base station 16 also includes a quality measurement unit 28. The quality measurement unit 28 includes a wireless environment measurement unit 30 and a traffic measurement unit 32. The wireless environment measurement unit 30 acquires information related to the wireless communication environment surrounding the wireless base station 16. The traffic measurement unit 32 acquires information related to the traffic between the wireless base station 16 and the wireless terminal 18. The quality measurement unit 28 calculates information on the communication quality between the wireless base station 16 and the wireless terminal 18 based on the information. The communication quality information includes, for example, the received power from the wireless terminal 18, the requested data rate of the wireless terminal 18, the actual transmission rate, and the bandwidth usage rate in the wireless base station 16.

The wireless base station 16 includes a communication IF unit 34. The communication IF unit 34 is used for wired communication via the network 12. The wireless base station 16 can acquire communication quality information acquired by other wireless base stations 16 by wired communication via the communication IF unit 34, or by wireless communication via the wireless communication unit 22 and antenna unit 20. The communication quality information acquired from other wireless base stations 16 is managed centrally in the quality measurement unit 28 together with the communication quality information acquired by the wireless base station 16 itself.

The wireless base station 16 includes an information notification unit 36. The information notification unit 36 can read out the communication quality information aggregated in the quality measurement unit 28. The information notification unit 36 can provide the read out communication quality information to the communication IF unit 34 and the wireless communication unit 22. This allows the wireless base station 16 to provide its own communication quality information to other wireless base stations 16 via a wired or wireless route.

The wireless base station 16 receives an attribution request from a wireless terminal 18 located in the vicinity via the wireless communication unit 22 and the antenna unit 20. The attribution request is processed by the request information evaluation unit 38. The request information evaluation unit 38 determines whether or not to allow attribution based on the bandwidth and data rate requested by the wireless terminal 18, as well as its own load state, etc.

The result of the determination by the request information evaluation unit 38 is sent to the request response notification unit 40 and the belonging information recording unit 42. The request response notification unit 40 provides the result of the determination as to whether or not belonging is possible to the wireless terminal 18 that issued the belonging request via the wireless communication unit 22 and the antenna unit 20. The belonging information recording unit 42 updates and records information regarding which wireless terminal 18 belongs to the wireless base station 16 based on the result of the determination.

Figure 5 is a block diagram for explaining the functions of an example of a wireless terminal 18. The wireless terminal 18 includes an antenna unit 44 and a wireless communication unit 46. The wireless communication unit 46 includes an RF unit 24 and a MAC function unit 50. These functions are substantially the same as the functions of the corresponding blocks included in the wireless base station 16, so their explanation will be omitted here.

The wireless terminal 18 includes a control information calculation unit 52. The control information calculation unit 52 acquires communication quality information from the wireless base station 16 to which the wireless terminal 18 belongs via the wireless communication unit 46 and the antenna unit 44. Then, the control information calculation unit 52 performs a process of determining, based on the communication quality information, an optimal action a _k for connecting each of a plurality of wireless communication IFs to which wireless base station 16 by DQN or DDQN, and performs learning for this purpose.

The control information calculation unit 52 includes a state calculation unit 54. The state calculation unit 54 calculates a state S _k (t) to be provided to the input layer of the DQN or DDQN based on the communication quality information.

The control information calculation unit 52 also includes a reward calculation unit 56. The reward calculation unit 56 calculates a reward Γ _k obtained by the adopted action a _k , based on the communication quality information.

The control information calculation unit 52 further includes a DQN/DDQN update unit 58. The DQN/DDQN update unit 58 updates the parameter θ of the DQN or DDQN used in reinforcement learning. More specifically, the weighting coefficient and bias value (hereinafter referred to as "weighting coefficient, etc.") that determine the characteristics of the DQN or DDQN as a value function are updated.

The DQN or DDQN updated by the DQN/DDQN update unit 58 is used in the belonging calculation unit 60. That is, the belonging calculation unit 60 calculates the action a _k to be adopted this time, that is, which IF should be attributed to which wireless base station 16, according to the updated DQN or DDQN.

In DQN and DDQN, when a state S _k is given to the input layer, the adoption value of each of the actions a _k that the wireless terminal 18 can take is displayed in the output layer. Therefore, by selecting the action with the largest value, it is possible to select an action that is considered optimal at the current learning stage. However, if the next action a _k is always selected based on the above value, it may become trapped in a local solution selected by chance, and the truly optimal action may never be selected.

In order to avoid such a situation, the assignment calculation unit 60 calculates the action a _k according to the ε-greedy method. Therefore, the assignment calculation unit 60 randomly selects the next action a _k with a probability ε, regardless of DQN or DDQN. Also, the assignment calculation unit 60 selects the action a _k that DQN or DDQN judges to be optimal with a probability of 1-ε.

Furthermore, the destination calculation unit 60 performs a decay process on ε each time learning is repeated. The possibility of falling into a local solution selected by chance is large in the early stages of learning and decreases as learning progresses. Therefore, the necessity of randomly selecting an action a _k decreases as learning progresses. By performing a decay process on ε, the frequency with which the action a _k is randomly selected can be reduced in line with the decrease in the necessity. Therefore, according to the destination calculation unit 60 of this embodiment, learning can be efficiently advanced and the parameter θ can be converged to an optimal value in a short period of time.

By the way, transfer learning is known as a method for learning parameters for DQN or DDQN. Transfer learning is a method in which the final output layer is replaced and trained based on a trained model. In transfer learning, parameters close to the input layer are left unchanged from their trained values, and only the parameters for the replaced output layer and the parameters for several layers before that are retrained with original data. With transfer learning, only some parameters are subject to update, so the computational load associated with updating can be reduced compared to updating all parameters.

As described above, in the wireless communication system 10 of this embodiment, it is desirable to have the DQN or DDQN quickly follow environmental changes in a dynamic communication environment. To achieve this, it is necessary to repeatedly update the DQN or DDQN at a high frequency. And, in order to repeatedly update at a high frequency, it is necessary to sufficiently reduce the computational load associated with the updates.

In this embodiment, based on the above viewpoint, the idea of transfer learning is applied to the update of the DQN or DDQN to perform partial parameter updates. In other words, after parameter initialization is completed, among the hidden layers included in the DQN or DDQN, the parameters corresponding to the previous stage are fixed, and only the parameters corresponding to the subsequent stages are updated. With this method, the computational load associated with updating the DQN or DDQN can be significantly reduced compared to when all parameters are subject to update.

As described above, the wireless terminal 18 in this embodiment handles the B _UB wireless base stations 16 permitted by its hardware as communication targets. When an attempt is made to update the DQN or DDQN with the B _UB wireless base stations 16 as communication targets, the number of possible combinations that can be established between the two IFs included in the wireless terminal 18 and the two IFs included in the B _UB wireless base stations 16 becomes the size of the state space.

However, the B _UB wireless base stations 16 include some that are not actually targets for belonging because the signal strength between them and the wireless terminal 18 is weak. If such wireless base stations 16 are excluded from the targets for belonging, the size of the state space to be considered in updating the DQN or DDQN becomes smaller, and the calculation load associated with the update can be reduced.

To achieve the above functions, the wireless terminal 18 includes an environmental change detection unit 62 and an update range control unit 64. The environmental change detection unit 62 performs the following three functions.
1. Based on the signal received from the surrounding wireless base stations 16, a received signal strength indicator (RSSI) indicating the channel quality is measured.
2. Select Bmax wireless base stations 16 in descending order of RSSI strength, where Bmax is a number smaller than _BUB .
3. When the combination of wireless terminals 18 included in Bmax or the combination of IFs available thereto changes, a change in the environment is recognized.

The update range control unit 64 performs the following three functions.
1. In the stage of initializing the parameter θ, the DQN/DDQN update unit 58 is instructed to update all of the hidden layers. At the same time, the reward calculation unit 56 and the DQN/DDQN update unit 58 are instructed to treat all of the B _UB wireless base stations 6 as active targets.
2. After the initialization is completed, instruct the DQN/DDQN update unit 58 to update only the latter stages of the hidden layer. At the same time, instruct the reward calculation unit 56 and the DQN/DDQN update unit 58 to treat only the Bmax wireless base stations 16 with large RSSI as active targets.
3. When the environmental change detection unit 62 recognizes an environmental change, it instructs the reward calculation unit 56 and the DQN/DDQN update unit 58 to save the learned parameters and change the active targets to the new Bmax wireless base stations 16.

The reward calculation unit 56 and the DQN/DDQN update unit 58 switch the update method for the parameter θ according to instructions from the update range control unit 64. This sufficiently reduces the load associated with updating the parameter θ after the initialization is completed.

In this embodiment, the wireless terminal 18 includes a memory unit 66. The memory unit 66 stores the following information as experience e _k (t):
1. The state S _k (t) provided to the input layer of DQN or DDQN in this learning,
2. Action a _k (t) determined according to the state S _k (t),
3. The reward Γ _k (t) obtained by the action a _k (t),
4. State S _k (t+1) to which the transition is made by the action a _k (t).

The wireless terminal 18 also includes an attachment request unit 68. The attachment request unit 68 receives the action a _k from the attachment calculation unit 60 and generates a request corresponding to the action a _k . For example, the request generated is to connect the wireless terminal 18 to the wireless base station 16-1 via IF1, or to connect the wireless terminal 18 to the wireless base station 16-2 via IF2. The generated request is transmitted to one or more wireless base stations 16 via the wireless communication unit 46 and the antenna unit 20.

The wireless terminal 18 is equipped with an attachment destination recording unit 70. The information received from the wireless base station 16 includes a response to the request as to whether it is OK or not. For example, the information includes a response to allow a new connection, or a response to allow an update of a connection. Based on these responses, the attachment destination recording unit 70 updates and records information about the wireless base station 16 to which it is attached.

Figure 6 is a diagram for explaining the hardware configuration of the wireless terminal 18. Here, a case where the wireless terminal 18 is a smartphone will be explained. However, the wireless terminal 18 is not limited to a smartphone, and may be a tablet terminal or a general-purpose computer system.

The wireless terminal 18 is equipped with a CPU (Central Processing Unit) 72. The CPU 72 is connected to various hardware elements equipped in the wireless terminal 18 via a bus wiring 74. Various memory devices, such as a ROM (Read Only Memory) 76, a RAM (Random Access Memory) 78, and a storage 80, are connected to the bus wiring 74.

The storage 80 includes a non-volatile memory such as a flash ROM. Programs and data necessary for executing various applications are stored in the storage 80. The memory unit 66 and the destination recording unit 70 described above are realized by the storage 80. The CPU 72 also realizes the functions of the control information calculation unit 52, the environmental change detection unit 62, the update range control unit 64, and the like shown in FIG. 5 by carrying out processing in accordance with the programs stored in the storage 80.

The wireless terminal 18 includes a communication interface 82. The communication interface 82 corresponds to the antenna unit 44 and wireless communication unit 46 described above. The wireless terminal 18 further includes an operation unit 84 and a display unit 86. The operation unit 84 includes a touch panel integrated with the display unit 86, in addition to various mechanical switch mechanisms and electronic sensor mechanisms.

[Overview of DQN]
Fig. 7 is a diagram for explaining an overview of DQN. Hereinafter, an overview of DQN, which is an example of deep reinforcement learning used by the wireless terminal 18 _k , will be explained with reference to Fig. 7. The DQN used in this embodiment has an input layer, a hidden layer, and an output layer. In addition, a parameter θ _i ^t corresponding to the learning state at the current time t is set in the DQN. The characteristics of the DQN as a value function are determined by the parameter θ _i ^t .

7, the input layer of the DQN has M nodes S ₁ to S _M. Elements constituting the state S _k of the wireless terminal 18 _k are input to these nodes. The elements of the state S _k include, for example, the following elements:
1. Bandwidth utilization rate φb of wireless base station 16-b,
2. The required data rate R _kf of the interface f of the wireless terminal 18 _k ,
3. The relationship x _bkf between the wireless terminal 18 _k and the wireless base station 16-b (1 if the wireless terminal 18 k belongs to the wireless base station 16-b, 0 if not),
4. A request a _bkf for association of the interface f of the wireless terminal 18 _k to the wireless base station 16-b (1 if there is a request, 0 if not),
5. The transmission rate r _bkf established between the interface f of the wireless terminal 18 _k and the wireless base station 16-b.

The number of hidden layers and the number of nodes are arbitrary. However, in this embodiment, the number of hidden layers is two or more. The output layer has a number of nodes equal to the total number N of actions _ai that the wireless terminal _18k can take in the state _Sk . Some of the actions _ai are shown below.
-Only IF1 is associated with the wireless base station 16-1.
Both IF1 and IF2 are associated with the wireless base station 16-1.
-Only IF2 is associated with the wireless base station 16-1.
- IF1 is associated with the wireless base station 16-1, and IF2 is associated with the wireless base station 16-2.
・
・
・

Each node in the output layer displays a Q value = Q (S _k , a _i ; θ _i ^t ) for each action a _i (i = 1 to N) that the wireless terminal 18 _k can take. The Q value is the adoption value for the action a _i calculated by the DQN under the parameter θ _i ^t .

In a method using DQN, the largest value is selected from the Q values appearing in the output layer, and the action a _i (t) that produces that Q value is determined as the optimal action a _k (t) at the current time t. In this way, DQN makes it possible to determine an action a _k (t) that can be determined as optimal at the time t in each learning stage.

Fig. 8 is a diagram for explaining the flow in which the control information calculation unit 52 of the wireless terminal 18 _k updates the parameter θ _i of the DQN. Note that in Fig. 8, the action a _k (t) is shown after the DQN/DDQN for convenience, but the action a _k (t) is determined by the DQN or DDQN only when the probability is 1-ε, as described above. Under the probability ε, the action a _k (t) is determined randomly. The parameter θ _i reflects the experience e _k (t) obtained in both cases.

After the action a _k (t) is determined in the wireless terminal 18 _k , the reward Γ _k (t) for the action a _k (t) is calculated. The reward Γ _k (t) is calculated, for example, by the following formula (1).

In the above formula (1), the first term on the right side is a point-adding item, and the second term is a point-deducting item. ω _1k and ω _2k attached to both are coefficients that determine the allocation ratio for them. Also, ν is an identifier to distinguish between the 1 GHz or lower (sub) term that IF1 should support and the millimeter wave (mW) term that IF2 should support.

Furthermore, in the above formula (1), c _1k ^v (t) in the first term on the right side means the added point when a transmission rate r _bkf (t) that satisfies the required data rate R _kf is obtained. The value is calculated by the following formula (2).

The above formula (2) means that an additional point r _bkf (t)/W _v is given according to the transmission rate r _bkf (t) in the following cases: where W _v in the last term is a factor for offsetting and normalizing the difference between the (sub) and (mW) frequency bands.
1. The interface f of the wireless terminal 18 _k is requesting to belong to the wireless base station 16-b (a _bkf (t)=1).
2. The wireless base station 16-b recognizes the association of the interface f of the wireless terminal 18 _k (x _bkf (t) = 1), and a transmission rate r _bkf (t) exceeding the requested data rate R _kf is obtained (r _bkf (t) > Rkf).

In the above formula (1), c _2k ^v (t) in the second term on the right side means the points deducted in the following two cases.
1. A request for association is made (a _bkf (t)=1) and association is granted (x _bkf (t)=1), but a transmission rate r _bkf (t) that satisfies the requested data rate R _kf is not available (r _bkf (t)<R _kf ).
2. An association request is made (a _bkf (t)=1), but the association is dropped by the radio base station 16 (x _bkf ^v (t)=0).

The deduction point c _2k ^sub (t) in the case of (sub) can be expressed, for example, as in the following formula (3). Also, the deduction point c _2k ^mW (t) in the case of (mW) can be expressed, for example, as in the following formulas (4) and (5).

In this way, if the action a _k (t) results in a transmission rate r _bkf (t) that satisfies the required data rate R _kf , the reward Γ _k (t) will be a large value. On the other hand, if a sufficient transmission rate r _bkf (t) is not obtained, the reward Γ _k (t) will be a small value.

Once the reward _Γk (t) is calculated, the current experience e( _t ) is stored in memory (step 88). Specifically, the current state S(t), the selected action _a (t), the obtained reward _{Γk (} t), and the state S( _t +1) after the transition are stored in memory as the experience _e (t).

If there is a time correlation in the sample sequence, learning may not proceed properly. To solve this problem, when updating the parameter θ _i , memory samples are randomly taken from the experience e _k stored in the memory and processed in batches (step 90).

The parameter _θi of the DQN/DDQN is updated using the randomly acquired memory sample, i.e., the experience e _k including (S _k , a _k , Γ _k , S _k ′) (step 92). Here, S _k ′ refers to the state to which the transition is made by the action a _k . Specifically, the process of this step 92 proceeds as follows.

When the reward Γ _k for the action a _k is determined, a "new Q value" = Q(S _k , a _k ) for selecting the action a _k in the state S _k can be calculated based on the reward Γ _k . The new Q value is calculated, for example, as shown in the following formula (6).

Q(S _k , a _k )←Γk+γ*maxQ(S _k ′, a _k ′) ・・・(6)
where γ is the time discount rate.

The right-hand side of the above equation (6), _Γk +γ*maxQ( _Sk ', _ak '), is the value that you want the "trained DQN" to calculate as the Q value for the combination of ( _Sk , _ak ). Therefore, the parameter _θi of the DQN can be updated so that the value appears in the output layer for the input of ( _Sk , _ak ).

The following equation (7) shows a loss function L _k used when updating the DQN parameter θ _i in this embodiment.

In the above formula (7), Q( _S , _a ; _θ ) in the first term on the right side is the Q value calculated by the main network having the parameter _θ for the combination of ( _S , _a ), and Q(S _' , _a '; _θ- ⁾ in the second term on the right side is the Q value calculated by the target network having ^the parameter _θ- for the combination of (S _' , _a ').

When updating a DQN, if the values of the network to be updated are calculated using the network being updated, learning may not be stable. For this reason, when updating a DQN, it is common to use a target network for calculating the update values in addition to the main network to be updated. In this embodiment, the second term of equation (7), which corresponds to the new Q value, is calculated using the target network, following the above-mentioned normal method.

In this step 92, the parameters θ _i of the DQN, i.e., the parameters θ _i of the main network, are updated so as to minimize the loss function L _k shown in the above equation (7). As a result, the DQN learns to express the value on the right side of the above equation (6) as the Q value of the action a _k for the input of the state S _k .

The target network is the same as the previous main network except immediately after the update. The target network is then updated at a predetermined cycle (step 94). That is, when the time for updating arrives at the predetermined cycle, ^the parameter θ _i of the target network is overwritten by the parameter θ _i of the main network. The above process completes the update of the DQN.

In addition, the following formula (8) shows the loss function L _k used in the above step 92 when the DDQN method is used in this embodiment. The principle of the update is the same as that of the DQN, so a detailed explanation of this will be omitted.

[Initialization of parameter _θi ]
9 is a diagram showing a schematic diagram of the characteristics of DQN when the parameter _θi is initialized. Here, the number of hidden layers is set to "two". The parameter θi includes a weighting coefficient Wh ₁ a ₁ for the h ₁ node of the hidden layer to select the a ₁ node of the output _layer , etc. When the parameter _θi is initialized, as described above, the update range control unit 64 instructs that all of the hidden layers are to be updated and that all of the B _UB wireless base stations 16 are to be treated as active.

The input layer and output layer of DQN and DDQN are provided with the number of nodes necessary to handle B _UB wireless base stations 16. For example, the output layer is provided with nodes a 1 to a|A _k | equal in number to the total number of combinations between the two IFs of the wireless terminal ₁₈ and the IFs of each of the B _UB wireless base stations 16. At the stage of initializing the parameter θ _i , all of these nodes are set to an active state. Then, all of the weighting coefficients W and the like set between the nodes of DQN or DDQN are updated by learning.

At this time, the reward Γ _k (t) is also calculated for B _UB wireless base stations 16. That is, as shown in the above formulas (1) to (4), the reward Γ _k (t) is calculated for set B of wireless base stations 16. Here, the calculations of the above formulas (1) to (4) are performed assuming that set B includes B _UB wireless base stations 16.

[Partial update of parameter _θi ]
FIG. 10 is a diagram showing the characteristics of DQN when the parameter θ _i is partially updated. After the initialization is completed, as described above, the update range control unit 64 instructs that only the rear hidden layer is to be updated. In response to this instruction, after the initialization is completed, only the rear layer 96 of the two hidden layers is to be updated, and the weighting coefficients W and the like related to the layer before the hidden layer are fixed. If the weighting coefficients W and the like related to the layer before the hidden layer are fixed, the number of parameters θ _i to be updated is reduced. Therefore, the calculation load associated with the update of the parameter θ _i is reduced compared to the calculation load during initialization.

The update range control unit 64 also instructs that only Bmax wireless base stations 16 with large RSSIs are treated as active after the initialization is completed. Nodes for handling wireless base stations 16 not included in Bmax are prepared in the input layer and output layer of DQN and DDQN. At this stage, those nodes are made inactive. For example, if the wireless base station 16-1 is not included in Bmax, the node for accommodating the action a _k such as "connecting IF1 of the wireless terminal 18 _k to the wireless base station 16-1" or "connecting both IF1 and IF2 of the wireless terminal 18 _k to the wireless base station 16-1" is made inactive.

In this case, the weighting coefficients W etc. related to the inactive nodes are fixed, and only the parameters _θi related to the active nodes are updated. If the weighting coefficients W etc. related to the inactive nodes are fixed, the number of parameters _θi to be updated is reduced. Therefore, the calculation load involved in updating the parameters _θi is further reduced compared to the calculation load at the time of initialization.

Regarding the calculation of reward Γ _k (t), the number of variables to be handled in the above formulas (1) to (4) is reduced by reducing the number of elements in set B from B _UB to Bmax, and the calculation load of reward Γ _k (t) is also significantly reduced compared to the initialization case.

In this manner, in this embodiment, when the initialization of the parameter θ _i is completed, the full update of the parameter θ _i is switched to a partial update thereafter. According to the partial update, the computation load associated with the learning can be significantly reduced without compromising the characteristics of the DQN or DDQN for deriving the optimal action a _k .

[Processing by the wireless terminal CPU]
Fig. 11 is a flowchart for explaining the flow of processing executed by the CPU 72 of the wireless terminal 18 to realize the above function. The routine shown in Fig. 11 is executed when the wireless terminal _18k is started up in a communication environment for the first time. Note that, here, a case where DQN is used as the reinforcement learning method will be explained.

In the routine shown in FIG. 11, first, a state S _k (t) to be provided to the input layer of DQN is set based on communication quality information and the like acquired from surrounding wireless base stations 16 (step 100).

Next, ε to be used in this learning is set (step 102). For example, a value of about 0.1 is given to ε as an initial value, and then a decay process is performed each time this routine is repeated.

Next, a random number between 0 and 1 is generated. It is then determined whether the random number is less than or equal to ε (step 104).

If it is determined that the random number≦ε holds, the next action a _k (t) for the current state S _k (t) is determined randomly (step 106).

On the other hand, if it is not determined in step 104 that the random number ≦ ε holds, the state S _k (t) is input to the input layer of the DQN (step 108). Then, the Q values appearing in the output layer of the DQN are compared, and the action a _i (t) showing the largest Q value is selected as the optimal action a _k (t) at the current time (step 110).

Next, based on the action a _k (t) selected by the above process, an association request is issued to the surrounding wireless base stations 16 (step 112).

When a response to the request is received, a reward _{Γ k} (t) for the selected action a k (t) is calculated based on the information contained in the response (step 114).

Next, the experience e _k (t) acquired in the current routine, that is, the set of (S _k (t), a _k (t), Γ _k (t), S _k (t+1)), is stored in the RAM 78 constituting the memory unit 66 (step 116).

Thereafter, the DQN is updated by batch processing (step 118), and the current routine is terminated. Here, all the parameters _θi characterizing the DQN are updated. The above process is repeated until the initialization of the parameters _θi is completed.

12 is a flow chart for explaining the flow of processing executed by the CPU 72 for partially updating the parameter _θi after the initialization is completed. In FIG. 12, steps that are substantially the same as those shown in FIG. 11 are denoted by the same reference numerals and their explanations are omitted or simplified.

In the routine shown in FIG. 12, first, Bmax wireless base stations 16 with superior RSSI are selected from the surrounding wireless base stations 16 (step 120).

Next, it is determined whether or not a change has occurred in the selection results of the Bmax wireless base stations 16 (step 122). More specifically, it is determined whether or not a change has occurred in the combination of communication interfaces available in the selected Bmax wireless base stations 16.

After the initialization is completed, if Bmax wireless base stations 16 are selected for the first time, it is determined that the above-mentioned "change" has occurred. Also, if the combination of Bmax wireless base stations 16 and available IFs selected in the current cycle is different from the combination selected in the previous cycle, it is determined that a "change" has occurred. In these cases, next, all of the parameters θ _i that have been learned at that time are saved (step 124).

Next, among the nodes in the input/output layer, only those nodes related to possible actions are activated (step 126). More specifically, among the nodes in the input and output layers of the DQN, only those that correspond to possible actions for the selected Bmax wireless base stations 16 are activated, and the other input/output nodes are deactivated.

If it is determined in step 120 that no changes have occurred to the Bmax combinations, steps 124 and 126 are skipped. Following these steps, a process is executed in which only the latter hidden layers of the DQN are updated, and the earlier hidden layers are not updated (step 128).

Thereafter, the processes from step 100 onwards are executed in the same manner as in the case shown in Fig. 11. In the routine shown in Fig. 12, each time a change is detected for the Bmax combinations, the attenuated ε may be returned to its initial value in step 102. When ε is reset to its initial value, the frequency of random search execution is restored, making it easier for the parameter _θi to follow changes in the communication environment.

In the routine shown in FIG. 12, in step 118, the parameters related to the layer before the hidden layer are held at fixed values. Then, the parameters related to the layer after the hidden layer are updated as the learning progresses.

12, in step 118, the parameter _θi is partially updated by regarding only the input/output nodes corresponding to the Bmax wireless base stations 16 as active. In other words, only the weighting coefficients W etc. located between the active input/output nodes are updated, and the other parameters are maintained at fixed values.

12, when the communication environment surrounding the wireless terminal _18k changes and the combination of the Bmax wireless base stations 16 or the combination of IFs available therein changes, the learned parameter _θi at that point is stored. Then, thereafter, a partial update of the parameter _θi is performed for the new Bmax wireless base stations 16 to derive the optimal action a _k .

According to the above process, even if the number of wireless base stations 16 belonging to the environment surrounding the wireless terminal 18 _k increases, or even if the number of IFs available therein increases, the wireless terminal 18 k can frequently learn the DQN parameter θ _i without imposing an excessive calculation load. Therefore, according to the present embodiment, in a heterogeneous environment with multi-band multi-access, system resources can be constantly and effectively utilized.

In the above-described first embodiment, the wireless terminal 18 and the wireless base station 16 use two IFs for wireless communication, but the present disclosure is not limited to this. For example, there may be three or more IFs for wireless communication. Furthermore, the IFs for wireless communication may include a mixture of IFs for licensed bands and IFs for unlicensed bands used for wireless LANs and the like. Alternatively, there may be only one IF for wireless communication, and deep reinforcement learning such as DQN may be used to determine which wireless base station 16 the wireless terminal 18 should optimally request a connection to.

In addition, in the above-mentioned first embodiment, DQN or DDQN is used as the value function that determines the action. However, these are merely examples, and the present disclosure is not limited to these. Any function that can determine the value of each action that can be taken under a specific state can be used as the value function of the present disclosure.

In the above-mentioned first embodiment, the technique of updating only the rear of the hidden layer and the technique of treating only Bmax wireless base stations 16 with strong signal strength as active are combined, but it is not essential to use them in combination. For example, it is also possible to reduce the calculation load by treating all B _UB wireless base stations 16 as active and updating only the rear of the hidden layer.

In addition, in the above-mentioned embodiment 1, the case where there are two hidden layers is described, but the number of hidden layers is not limited to two, and as mentioned above, there may be two or more layers. Furthermore, when there are three or more hidden layers, the layer to be updated is not limited to the final layer. In this case, as long as the layer immediately preceding it is excluded, some of the later layers may be updated.

10 Wireless communication system 16, 16-1, 16-2, 16-3 Wireless base station 18, 18 _k Wireless terminal 52 Control information calculation unit 60 Destination calculation unit 62 Environmental change detection unit 64 Update range control unit Γ _k (t) Reward a _k (t) Action e _k (t) Experience S _k (t) State S _k (t+1) State after transition R _kf Requested data rate r _bkf Transmission rate θ _i Parameter

Claims (8)

A wireless communication method for determining a combination of connections between a plurality of wireless base stations and a wireless terminal, comprising:
preparing a neural network that includes an input layer and an output layer corresponding to a plurality of wireless base stations, and a hidden layer including two or more layers located therebetween, and functions as a value function that outputs, in response to an input of a state, a value of each action that the wireless terminal can take under the state;
detecting a state of connection between the plurality of wireless base stations and the wireless terminal;
providing the sensed state to the input layer and determining, according to a result appearing at the output layer, a behavior to be adopted by the wireless terminal under the sensed state;
transmitting a request to said plurality of radio base stations corresponding to an action to be taken;
obtaining responses to the request from a plurality of wireless base stations;
calculating a reward for the behavior based on the response;
updating a value for adopting the behavior under the state based on the reward;
learning parameters of the value function based on the updated values;
limiting the update range of the parameters to parameters associated with one or more layers in the hidden layer, excluding at least the frontmost layer;
The wireless terminal executes the wireless communication method. In the step of initializing the parameters, an update range of the parameters is set to all parameters of the neural network;
and storing all of the parameters after the initialization is completed ,
The wireless communication method according to claim 1 , wherein the learning in a state in which the update range of the parameter is limited is executed after the initialization is completed and after the storage is completed. the neural network includes an input layer and an output layer corresponding to B _UB wireless base stations;
selecting, from among the B _UB radio base stations, only Bmax radio base stations having excellent communication quality, the number of which is smaller than the B _UB radio base stations;
further limiting the update range of the parameters to parameters associated with actions related to the Bmax radio base stations;
The wireless communication method according to claim 1 or 2, further comprising the step of : The wireless communication method according to any one of claims 1 to 3, wherein the neural network is a Deep Q-Network or a Double Deep Q-Network. A wireless terminal having a function of connecting to a plurality of wireless base stations,
The neural network includes an input layer and an output layer corresponding to a plurality of wireless base stations, and a hidden layer including two or more layers located between them, and includes a memory unit storing a neural network that functions as a value function that outputs, in response to an input of a state, a value of each of the actions that the wireless terminal can take under the state, and a CPU;
The CPU,
A process of detecting a state of connection between the plurality of wireless base stations and the wireless terminal;
providing the sensed state to the input layer and determining the action to be taken by the wireless terminal under the sensed state according to the result appearing in the output layer;
transmitting a request corresponding to an action to be taken to said plurality of wireless base stations;
obtaining responses to the request from a plurality of wireless base stations;
calculating a reward for the behavior based on the response; and
updating a value for taking the action under the state based on the reward;
A process of learning parameters of the value function based on the updated value;
A process of limiting the update range of the parameters to parameters associated with one or more layers in the hidden layer, excluding at least the frontmost layer;
A wireless terminal that runs The CPU includes:
A process of setting the update range of the parameters to all parameters of the neural network in the step of initializing the parameters;
and storing all the parameters after the initialization is completed.
The wireless terminal according to claim 5 , wherein the learning in a state in which the update range of the parameter is limited is executed after the initialization is completed and after the storage is completed. the neural network includes an input layer and an output layer corresponding to B _UB wireless base stations;
The CPU includes:
a process of selecting, from among the _B radio base stations, only Bmax radio base stations having excellent communication quality, the Bmax being smaller than the B _radio base stations;
further limiting the update range of the parameters to parameters associated with actions related to the Bmax wireless base stations;
7. The wireless terminal according to claim 5 or 6, further comprising: A program for a wireless terminal for realizing a function of connecting to a plurality of wireless base stations,
A CPU provided in the wireless terminal,
a process of reading from a memory unit a neural network that includes an input layer and an output layer corresponding to a plurality of wireless base stations, and a hidden layer including two or more layers located between them, and that functions as a value function that outputs, in response to an input of a state, a value of each action that the wireless terminal can take under the state;
A process of detecting a state of connection between the plurality of wireless base stations and the wireless terminal;
providing the sensed state to the input layer and determining the action to be taken by the wireless terminal under the sensed state according to the result appearing in the output layer;
transmitting a request corresponding to an action to be taken to said plurality of wireless base stations;
obtaining responses to the request from a plurality of wireless base stations;
calculating a reward for the behavior based on the response; and
updating a value for taking the action under the state based on the reward;
A process of learning parameters of the value function based on the updated value;
A process of limiting the update range of the parameters to parameters associated with one or more layers in the hidden layer, excluding at least the frontmost layer;
A program for a wireless terminal including a program for executing the above.

Images