JP7618133B2 - Wireless communication method, wireless terminal, and program for wireless terminal - Google Patents

Description

This disclosure relates to a wireless communication method, a wireless terminal, and a program for the wireless terminal, and in particular to a wireless communication method, a wireless terminal, and a program for the wireless terminal that are suitable for achieving high communication efficiency in a network that includes multiple wireless terminals and multiple wireless base stations, etc.

As wireless communication systems develop, heterogeneous networks using multi-band, multi-access systems are reaching the stage of practical application. In cellular communications, fifth-generation mobile communications (5G) has been put to practical use. 5G communications systems use a wide range of frequencies, from sub-1 GHz bands to millimeter wave bands. In addition, an environment is being developed in which cells of various sizes, from small cells to macro cells, are provided in an overlapping manner.

Wireless LANs, another typical wireless access system, use radio frequencies in the 2.4/5/60 GHz bands, and the use of the 6 GHz band is also being considered. Wireless terminals such as smartphones are generally provided with interfaces (IFs) for accessing both cellular and wireless LANs. Furthermore, each IF is configured to support multiple frequency bands.

Wireless terminals generally select a wireless base station or access point (AP) to connect to from multiple frequency bands and access methods to establish communications. With functions such as dual connectivity, a single wireless terminal can also use multiple wireless base stations in an integrated manner. In such a heterogeneous environment, it is important to control and optimize across the entire system which IF a wireless terminal uses and which base station it selects in order to make effective use of system resources.

Regarding technology for effectively utilizing system resources in a heterogeneous environment, for example, Non-Patent Documents 1 and 2 below disclose a method for optimizing connections between wireless terminals and APs through reinforcement learning. More specifically, Non-Patent Document 2 discloses a technology for optimizing connections between wireless terminals using multiple IFs and multiple APs using Deep Q-Learning or Double Deep Q-Learning.

In addition, the following non-patent document 3 discloses Deep Q-Network (DQN), a method for handling high-dimensional inputs using reinforcement learning, and the following non-patent document 4 discloses Double Deep Q-Network (DDQN).

“Reinforcement Learning-aided Distributed User-to-Access Points Association in Interfering Networks”, Thi Ha Ly Dinh, Megumi Kaneko, Keisuke Wakao, Hirantha Abeysekera, Yasushi Takatori, IEEE Globecom, Waikoloa, Hawaii, USA, Dec. 2019 “Deep Reinforcement Learning-based User Association in Sub6GHz/mmWave Integrated Networks”, Thi Ha Ly Dinh, Megumi Kaneko, Keisuke Wakao, Kenichi Kawamura, Takatsune Moriyama, Hirantha Abeysekera and Yasushi Takatori, in IEEE CCNC, Jan. 2021 “Human-level control through deep reinforcement learning,” Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg & Demis Hassabis, Nature. 2015 Feb 26; Vol.518(7540): P529-533. doi: 10.1038/nature14236 “Deep Reinforcement Learning with Double Q-Learning,” H. Van Hasselt, A. Guez, and D. Silver, in Thirtieth AAAI conference on artificial intelligence, 2016

In deep reinforcement learning of DQN and DDQN as disclosed in the non-patent literature mentioned above, learning is performed using the ε-greedy method. According to the ε-greedy method, the next action is selected randomly with a probability of ε at each learning iteration. In addition, the action that is judged to be optimal based on the learning results at that time is selected as the next action with a probability of 1-ε.

According to the ε-greedy method described above, the current optimal action is selected with a probability of 1-ε, and a new "optimum" is searched for by random actions with a probability of ε. Therefore, this method makes it possible to proceed with learning toward the true optimal solution without getting stuck on a locally optimal solution found by chance.

The ε-greedy method used in DQN and DDQN also applies a decay process to the probability ε each time learning is repeated. With this process, as learning progresses, the probability of selecting a random action decreases and the probability of selecting an optimal action increases. As a result, unnecessary searches at an advanced learning stage are avoided, and the time required to converge to an optimal solution is shortened.

In a multi-band, multi-access heterogeneous environment, mobile terminals such as smartphones and tablets may be used as wireless terminals. The communication environment surrounding the mobile terminal changes as the mobile terminal moves. In addition, when many wireless terminals exist in the same area, changes in the requests from each wireless terminal affect the load on the wireless terminals and wireless base stations belonging to that area. As a result, the communication environment surrounding each wireless terminal changes significantly depending on the state of other wireless terminals. For these reasons, the optimal solution regarding which wireless base station and which IF a wireless terminal placed in a heterogeneous environment should connect to may not be a stable one, but rather a dynamic one that changes drastically.

Conventional learning methods that attenuate ε as learning progresses are effective in quickly obtaining an optimal solution in a stable communication environment. However, in a rapidly changing, dynamic communication environment, if ε decays, it becomes impossible to efficiently search for an optimal solution by following environmental changes. In this respect, conventional learning methods that use DQN, DDQN, etc. have the problem that they cannot necessarily use system resources effectively in a dynamic communication environment.

The present disclosure has been made in light of the above-mentioned problems, and aims to provide a wireless communication method, a wireless terminal, and a program for a wireless terminal that can always effectively utilize system resources in a network including multiple wireless terminals and multiple base stations, regardless of changes in the communication environment.

In order to achieve the above object, a first aspect of the present disclosure is a wireless communication method for determining a combination of connections between a plurality of wireless base stations and a wireless terminal, the method comprising:
detecting a state of connection between the plurality of wireless base stations and the wireless terminal;
randomly determining with probability ε a behavior to be adopted by said wireless terminal under detected conditions;
determining, with probability (1-ε), an action to be taken by said wireless terminal under the sensed conditions in accordance with a result of a value function;
transmitting a request to said plurality of radio base stations corresponding to an action to be taken;
obtaining responses from the plurality of radio base stations to the request;
calculating a reward for the behavior based on the response;
updating a value for adopting the behavior under the state based on the reward;
learning parameters of the value function based on the updated values;
detecting a change in a communication environment surrounding the wireless terminal;
performing a time-dependent attenuation process on the ε under a condition where a change in the communication environment is not detected;
initializing the ε when a change in the communication environment is detected;
It is preferable that the present invention includes the following:

A second aspect of the present disclosure is a wireless terminal having a function of connecting to a plurality of wireless base stations,
A CPU is provided,
The CPU,
A process of detecting a state of connection between the plurality of wireless base stations and the wireless terminal;
a process of randomly determining with probability ε an action to be taken under the detected condition;
determining, with probability (1-ε), the action to be taken under the detected condition according to the result of the value function;
transmitting a request corresponding to an action to be taken to said plurality of wireless base stations;
obtaining responses from the plurality of wireless base stations to the request;
calculating a reward for the behavior based on the response;
updating a value for taking the action under the state based on the reward;
A process of learning parameters of the value function based on the updated value;
A process of detecting a change in a communication environment surrounding the wireless terminal;
a process of performing a time-dependent attenuation process on the ε under a situation where a change in the communication environment is not detected;
A process of initializing the ε when a change in the communication environment is detected;
It is advisable to carry out the following:

A third aspect of the present disclosure is a program for a wireless terminal for realizing a function of connecting to a plurality of wireless base stations, comprising:
A CPU provided in the wireless terminal,
A process of detecting a state of connection between the plurality of wireless base stations and the wireless terminal;
a process of randomly determining with probability ε an action to be taken under the detected condition;
determining, with probability (1-ε), the action to be taken under the detected condition according to the result of the value function;
transmitting a request corresponding to an action to be taken to said plurality of wireless base stations;
obtaining responses from the plurality of wireless base stations to the request;
calculating a reward for the behavior based on the response;
updating a value for taking the action under the state based on the reward;
A process of learning parameters of the value function based on the updated value;
A process of detecting a change in a communication environment surrounding the wireless terminal;
a process of performing a time-dependent attenuation process on the ε under a situation where a change in the communication environment is not detected;
A process of initializing the ε when a change in the communication environment is detected;
It is preferable that the program include a program for executing the above.

According to the present disclosure, in a network including multiple wireless terminals and multiple base stations, etc., it is possible to continually update the connection combination to an optimal state regardless of changes in the communication environment. Therefore, according to the present disclosure, it is possible to always effectively utilize system resources in a heterogeneous environment.

1 is a diagram for explaining an overall configuration of a wireless communication system according to a first embodiment of the present disclosure. FIG. 2 is a diagram for explaining an example of a connection state between a wireless terminal and a wireless base station in the first embodiment of the present disclosure. FIG. 2 is a diagram illustrating a wireless terminal moving in a heterogeneous environment according to the first embodiment of the present disclosure. FIG. 2 is a block diagram for explaining a functional configuration of a radio base station used in the first embodiment of the present disclosure. FIG. 2 is a block diagram for explaining a functional configuration of a wireless terminal used in the first embodiment of the present disclosure. 2 is a diagram for explaining the hardware configuration of a wireless terminal used in the first embodiment of the present disclosure. FIG. FIG. 1 is a diagram for explaining an overview of DQN used in the first embodiment of the present disclosure. FIG. 11 is a diagram for explaining the contents of a process related to updating a DQN used in the first embodiment of the present disclosure. 1 is a flowchart for explaining features of a wireless terminal according to a first embodiment of the present disclosure.

Embodiment 1.
[Overall Configuration of First Embodiment]
Fig. 1 is a diagram for explaining an overall configuration of a wireless communication system 10 according to a first embodiment of the present disclosure. As shown in Fig. 1, the wireless communication system 10 includes a wired communication network 12. A plurality of wireless base stations 16 are connected to the network 12 via a switch 14. Each wireless base station 16 can communicate with another wireless base station 16 via the network 12 and via a wireless path.

The wireless communication system 10 shown in FIG. 1 includes multiple wireless terminals 18. In this embodiment, the wireless terminals 18 include mobile terminals such as smartphones or tablet terminals.

Each wireless terminal 18 has multiple sets of applications and IFs for wireless communication. For example, wireless terminal 18 has applications and IFs for communication below 1 GHz, applications and IFs for communication in the millimeter wave band, and applications and IFs for communication in the 2.4/5/60 GHz bands used in wireless LANs.

The above-mentioned wireless base stations 16 are also equipped with corresponding IFs, etc. Then, each wireless terminal 18 can wirelessly communicate with one or more wireless base stations 16 via the above-mentioned applications and IFs. In this way, the wireless communication system 10 of this embodiment realizes the function of a heterogeneous network with multi-band multi-access.

The wireless communication system 10 of this embodiment is compatible with wireless LAN access points (APs) in addition to wireless base stations for cellular communication installed by telecommunications carriers. For the sake of convenience, the following description will be given assuming that "wireless base station 16" encompasses both.

[Features of the First Embodiment]
Next, features of the wireless terminal 18 in this embodiment will be described with reference to Figures 2 to 9. In the following description, it is assumed that the wireless base station 16 and the wireless terminal 18 are provided with an interface IF1 corresponding to communication in a first frequency band and an interface IF2 corresponding to communication in a second frequency band. IF1 is an interface corresponding to a frequency band of, for example, 1 GHz or less, and IF2 is an interface corresponding to frequencies in the millimeter wave band, for example.

In the following description, when it is necessary to indicate a specific wireless terminal 18 among a plurality of terminals, the reference symbol k will be used to indicate the wireless terminal 18 _k . Furthermore, when it is necessary to distinguish between the wireless base stations 16, reference symbols with suffixes, such as 16-1 and 16-2, will be used.

2 shows a state where a wireless terminal 18 _k belongs to two wireless base stations 16-1 and 16-2. The wireless terminal 18 k measures a received signal strength indicator (RSSI) that indicates the state of channel quality in the communication environment surrounding the wireless terminal 18 k. Then, the wireless terminal 18 _k selects Bmax wireless base stations 16 as communication targets in descending order of the strength indicated by the RSSI. FIG. 2 shows a state where the wireless terminal 18 _k is connected to the wireless base stations 16-1 and 16-2 out of the Bmax wireless base stations 16.

As described above, the wireless terminal 18 _k and the wireless base station 16 are provided with two interfaces, IF1 and IF2, corresponding to different bands. If Bmax wireless base stations 16 are the targets of communication, the number of possible combinations between the two IFs provided by the wireless terminal 18 _k and the two IFs provided by the Bmax wireless base stations 16 is the size of the state space. In the example shown in Fig. 2, a state is formed in which the wireless terminal 18 _k belongs to the first wireless base station 16-1 at IF1 and belongs to the second wireless base station 16-2 at IF2.

3 shows a state where a wireless terminal 18 _k moves in a heterogeneous environment with multi-band multi-access. First, assume an environment in which the wireless terminal 18 _k is located at the leftmost position in the figure. In this environment, if it is optimal for the wireless terminal 18 _k to belong to the wireless base stations 16-1 and 16-2, it is desirable for the connection request from the wireless terminal 18 _k to be issued so as to realize that state.

However, if the wireless terminal _18k moves from the leftmost position in the figure to the center of the figure, or even to the right side of the figure, the optimal connection combination will change from the original combination. Similar changes also occur due to the movement of other wireless terminals 18 (not shown) that belong to the same environment as the wireless terminal _18k , or changes in the requested data rates from them, etc. For this reason, when the communication environment changes, it is desirable for the wireless terminal _18k to issue a connection request with a combination that is optimal for the changed environment.

In this embodiment, the wireless terminal 18 determines the wireless base station 16 and IF to which it wishes to connect using Deep Q-Network (DQN) or Double Deep Q-Network (DDQN). For this reason, it is desirable that the DQN or DDQN used here quickly converges to an optimal state for a stable communication environment, and quickly follows environmental changes in a dynamic communication environment.

Figure 4 is a block diagram for functionally explaining an example of a wireless base station 16. The wireless base station 16 is equipped with an antenna unit 20. The antenna unit 20 is used by the wireless base station 16 to transmit and receive wireless signals to and from other wireless base stations 16 and surrounding wireless terminals 18.

The antenna unit 20 is connected to the wireless communication unit 22. The wireless communication unit 22 includes an RF (Radio Frequency) unit 24 that processes high-frequency signals, and a MAC (Media Access Control) function unit 26 that has a function of media access control in wireless communication. The wireless communication unit 22 is configured to be able to realize functions corresponding to multiple frequency bands, specifically, a function as IF1 corresponding to 1 GHz or less, and a function as IF2 corresponding to millimeter wave band frequencies.

The wireless base station 16 also includes a quality measurement unit 28. The quality measurement unit 28 includes a wireless environment measurement unit 30 and a traffic measurement unit 32. The wireless environment measurement unit 30 acquires information related to the wireless communication environment surrounding the wireless base station 16. The traffic measurement unit 32 acquires information related to the traffic between the wireless base station 16 and the wireless terminal 18. The quality measurement unit 28 calculates information on the communication quality between the wireless base station 16 and the wireless terminal 18 based on the information. The communication quality information includes, for example, the received power from the wireless terminal 18, the required data rate of the wireless terminal 18, the actual transmission rate, and the bandwidth usage rate in the wireless base station 16.

The wireless base station 16 includes a communication IF unit 34. The communication IF unit 34 is used for wired communication via the network 12. The wireless base station 16 can acquire communication quality information acquired by other wireless base stations 16 by wired communication via the communication IF unit 34, or by wireless communication via the wireless communication unit 22 and antenna unit 20. The communication quality information acquired from other wireless base stations 16 is managed in a unified manner in the quality measurement unit 28, together with the communication quality information acquired by the wireless base station 16 itself.

The wireless base station 16 includes an information notification unit 36. The information notification unit 36 can read out the communication quality information aggregated in the quality measurement unit 28. The information notification unit 36 can provide the read out communication quality information to the communication IF unit 34 and the wireless communication unit 22. This allows the wireless base station 16 to provide its own communication quality information to other wireless base stations 16 via a wired or wireless route.

The wireless base station 16 receives an attribution request from a wireless terminal 18 located in the vicinity via the wireless communication unit 22 and the antenna unit 20. The attribution request is processed by the request information evaluation unit 38. The request information evaluation unit 38 determines whether or not to allow attribution based on the bandwidth and data rate requested by the wireless terminal 18, as well as its own load state, etc.

The result of the determination by the request information evaluation unit 38 is sent to the request response notification unit 40 and the belonging information recording unit 42. The request response notification unit 40 provides the result of the determination as to whether or not belonging is possible to the wireless terminal 18 that issued the belonging request via the wireless communication unit 22 and the antenna unit 20. The belonging information recording unit 42 updates and records information regarding which wireless terminal 18 belongs to the wireless base station 16 based on the result of the determination.

Figure 5 is a block diagram for explaining the functions of an example of a wireless terminal 18. The wireless terminal 18 includes an antenna unit 44 and a wireless communication unit 46. The wireless communication unit 46 includes an RF unit 24 and a MAC function unit 50. These functions are substantially the same as the functions of the corresponding blocks included in the wireless base station 16, so their explanation will be omitted here.

The wireless terminal 18 includes a control information calculation unit 52. The control information calculation unit 52 acquires communication quality information from the wireless base station 16 to which the wireless terminal 18 belongs via the wireless communication unit 46 and the antenna unit 44. Then, the control information calculation unit 52 performs a process of determining, based on the communication quality information, an optimal action a _k to which wireless base station 16 each of a plurality of wireless communication IFs should be connected by a DQN or DDQN method, and performs learning for this purpose.

The control information calculation unit 52 includes a state calculation unit 54. The state calculation unit 54 calculates a state S _k (t) to be provided to the input layer of the DQN or DDQN based on the communication quality information.

The control information calculation unit 52 also includes a reward calculation unit 56. The reward calculation unit 56 calculates a reward Γ _k (t) obtained by the adopted action a _k (t) based on the communication quality information.

The control information calculation unit 52 further includes a DQN/DDQN update unit 58. The DQN/DDQN update unit 58 updates the parameter θ of the DQN or DDQN used in reinforcement learning. More specifically, the weighting coefficient w that determines the characteristics of the DQN or DDQN as a value function, etc., are updated.

The DQN or DDQN updated by the DQN/DDQN update unit 58 is used in the attribute calculation unit 60. That is, the attribute calculation unit 60 calculates the action a _k (t) to be adopted this time, that is, which IF should be attributed to which wireless base station 16, according to the updated DQN or DDQN.

In DQN and DDQN, when a state S _k (t) is given to the input layer, the value of adoption for each of the actions a _k (t) that the wireless terminal 18 can take is displayed in the output layer. Therefore, by selecting the action with the largest value, it is possible to select an action that is considered optimal at the current learning stage. However, if the next action a _k (t) is always selected based on the above value, it may become trapped in a local solution selected by chance, and the truly optimal action may never be selected.

In order to avoid such a situation, the assignment calculation unit 60 calculates the action a _k (t) according to the ε-greedy method. Therefore, the assignment calculation unit 60 randomly selects the next action a _k (t) with a probability ε, regardless of whether it is DQN or DDQN. Also, the assignment calculation unit 60 selects the action a _k (t) that DQN or DDQN judges to be optimal with a probability of 1-ε.

Furthermore, the destination calculation unit 60 performs a decay process on ε each time learning is repeated. The possibility of falling into a local solution selected by chance is large in the early stages of learning and decreases as learning progresses. Therefore, the necessity of randomly selecting an action a _k (t) decreases as learning progresses. By performing a decay process on ε, the frequency with which the action a _k (t) is randomly selected can be reduced in line with the decrease in the necessity. Therefore, according to the destination calculation unit 60 of this embodiment, learning can be efficiently advanced and the parameter θ can be converged to an optimal value in a short period of time.

However, in the wireless communication system 10 of this embodiment, the communication environment surrounding the wireless terminal 18 may change significantly as a result of the wireless terminal 18 moving, or as a result of changes in the bandwidth or data rate required by other wireless terminals 18 located in the vicinity. If the communication environment changes, the parameter θ, which was the optimal value before the change, may no longer be the optimal value.

In such a situation, that is, in a situation where a new optimum needs to be searched for, it is desirable that the next action a _k (t) is adopted randomly with high frequency. However, in a situation where ε is decaying as the learning progresses, such a requirement cannot be met.

To meet the above requirements, the wireless terminal 18 of this embodiment is equipped with an environmental change detection unit 62 and an ε initialization determination unit 64. The environmental change detection unit 62 detects changes in the communication environment surrounding the wireless terminal 18. For example, when the combination of wireless base stations 16 that the wireless terminal 18 can access changes, or when the combination of IFs available on the wireless base station 16 side changes, the environmental change detection unit 62 determines that the communication environment surrounding the wireless terminal 18 has changed.

The ε initialization determination unit 64 determines that ε needs to be initialized when a change in the communication environment surrounding the wireless terminal 18 is detected. The determination of the ε initialization determination unit 64 is provided to the belonging calculation unit 60. Then, when the belonging calculation unit 60 receives the determination of ε initialization, it initializes the attenuated ε to its original value. This increases the probability ε that the next action a _k (t) is randomly determined, and increases the frequency of searching for a new optimum. Therefore, according to the belonging calculation unit 60 of this embodiment, in an environment in which the communication environment surrounding the wireless terminal 18 changes, the parameters of DQN and DDQN can be appropriately tracked to the change.

The wireless terminal 18 includes a memory unit 66. The memory unit 66 stores the following information as experience e _k (t):
1. The state S _k (t) provided to the input layer of DQN or DDQN in this learning,
2. Action a _k (t) determined according to the state S _k (t),
3. The reward Γ _k (t) obtained by the action a _k (t),
4. State S _k (t+1) to which the transition is made by the action a _k (t).

The wireless terminal 18 also includes an attachment request unit 68. The attachment request unit 68 receives the action a _k (t) from the attachment calculation unit 60, and generates a request corresponding to the action a _k (t). For example, the request generated is to connect the wireless terminal 18 to the wireless base station 16-1 via IF1, or to connect the wireless terminal 18 to the wireless base station 16-2 via IF2. The generated request is transmitted to one or more wireless base stations 16 via the wireless communication unit 46 and the antenna unit 20.

The wireless terminal 18 is equipped with an attachment destination recording unit 70. The information received from the wireless base station 16 includes a response to the request as to whether or not the request is accepted. For example, the information includes a response to permit a new connection, or a response to permit an update of a connection. Based on these responses, the attachment destination recording unit 70 updates and records information about the wireless base station 16 to which the wireless terminal 18 is attached.

Figure 6 is a diagram for explaining the hardware configuration of the wireless terminal 18. Here, a case where the wireless terminal 18 is a smartphone will be explained. However, the wireless terminal 18 is not limited to a smartphone, and may be a tablet terminal or a general-purpose computer system.

The wireless terminal 18 is equipped with a CPU (Central Processing Unit) 72. The CPU 72 is connected to various hardware elements equipped in the wireless terminal 18 via a bus wiring 74. Various memory devices, such as a ROM (Read Only Memory) 76, a RAM (Random Access Memory) 78, and a storage 80, are connected to the bus wiring 74.

The storage 80 includes a non-volatile memory such as a flash ROM. Programs and data necessary for executing various applications are stored in the storage 80. The memory unit 66 and the destination recording unit 70 described above are realized by the storage 80. The CPU 72 also realizes the functions of the control information calculation unit 52, the environmental change detection unit 62, and the epsilon initialization determination unit 64 shown in FIG. 5 by carrying out processing in accordance with the programs stored in the storage 80.

The wireless terminal 18 includes a communication interface 82. The communication interface 82 corresponds to the antenna unit 44 and wireless communication unit 46 described above. The wireless terminal 18 further includes an operation unit 84 and a display unit 86. The operation unit 84 includes a touch panel integrated with the display unit 86, in addition to various mechanical switch mechanisms and electronic sensor mechanisms.

[Overview of DQN]
Fig. 7 is a diagram for explaining an overview of DQN. Hereinafter, an overview of DQN, which is an example of deep reinforcement learning used by the wireless terminal 18 _k , will be explained with reference to Fig. 7. The DQN used in this embodiment has an input layer, a hidden layer, and an output layer. In addition, a parameter θ _i ^t corresponding to the learning state at the current time t is set in the DQN. The characteristic of the DQN as a value function is determined by the parameter θ _i ^t .

7, the input layer of the DQN has M nodes S ₁ to S _M. Elements constituting the state S _k of the wireless terminal 18 _k are input to these nodes. The elements of the state S _k include, for example, the following elements:
1. Bandwidth utilization rate φb of wireless base station 16-b,
2. The required data rate R _kf of the interface f of the wireless terminal 18 _k ,
3. The relationship x _bkf between the wireless terminal 18 _k and the wireless base station 16-b (1 if it belongs, 0 if not)
4. A request a _bkf for association of the interface f of the wireless terminal 18 _k to the wireless base station 16-b (1 if there is a request, 0 if not),
5. The transmission rate r _bkf established between the interface f of the wireless terminal 18 _k and the wireless base station 16-b.

The number of hidden layers and the number of nodes are arbitrary. The output layer has a number of nodes equal to the total number N of actions _ai that the wireless terminal _18k can take in the state _Sk . Some of the actions _ai are shown below.
-Only IF1 is associated with the wireless base station 16-1.
Both IF1 and IF2 are associated with the wireless base station 16-1.
- Only IF2 is associated with the wireless base station 16-1.
- IF1 is associated with the wireless base station 16-1, and IF2 is associated with the wireless base station 16-2.
・
・
・

Each node in the output layer displays a Q value = Q (S _k , a _i ; θ _i ^t ) for each action a _i (i = 1 to N) that the wireless terminal 18 _k can take. The Q value is the adoption value for the action a _i calculated by the DQN under the parameter θ _i ^t .

In the method using DQN, the largest value is selected from the Q values appearing in the output layer, and the action a _i that produces that Q value is determined as the optimal action a _k (t) at the current time t. In this way, according to DQN, it is possible to determine the action a _k (t) that can be determined as optimal at the time t in each learning stage.

Fig. 8 is a diagram for explaining the flow in which the control information calculation unit 52 of the wireless terminal 18 _k updates the parameter θ _i of the DQN. Note that in Fig. 8, the action a _k (t) is shown after the DQN/DDQN for convenience, but the action a _k (t) is determined by the DQN or DDQN only when the probability is 1-ε, as described above. Under the probability ε, the action a _k (t) is determined randomly. The parameter θ _i reflects the experience e _k (t) obtained in both cases.

After the action a _k (t) is determined in the wireless terminal 18 _k , the reward Γ _k (t) for the action a _k (t) is calculated. The reward Γ _k (t) is calculated, for example, by the following formula (1).

In the above formula (1), the first term on the right side is a point-adding item, and the second term is a point-deducting item. ω _1k and ω _2k attached to both are coefficients that determine the allocation ratio for them. Also, ν is an identifier for distinguishing between the 1 GHz or lower (sub) term that IF1 should support and the millimeter wave (mW) term that IF2 should support.

Furthermore, in the above formula (1), c _1k ^ν in the first term on the right side means the added point when a transmission rate r _bkf that satisfies the required data rate R _kf is obtained. The value is calculated by the following formula (2).

The above formula (2) means that an additional point r _bkf /W _v according to the transmission rate r _bkf is given in the following cases: where W _v in the last term is a factor for offsetting and normalizing the difference between the (sub) and (mW) frequency bands.
1. The interface f of the wireless terminal 18 _k is requesting to belong to the wireless base station 16-b (a _bkf (t)=1).
2. The wireless base station 16-b recognizes the association of the interface f of the wireless terminal _18k ( _xbkf = 1), and a transmission rate _rbkf exceeding the requested data rate _Rkf is obtained ( _rbkf (t)> _Rkf ).

In the above formula (1), c _2k ^ν in the second term on the right side means the point deduction in the following two cases.
1. A request for association is made (a _bkf (t)=1) and association is granted (x _bkf (t)=1), but a transmission rate r _bkf that satisfies the requested data rate R _kf is not available (r _bkf (t)<R _kf ).
2. An association request is made (a _bkf (t)=1), but the association is dropped by the radio base station 16 (x _bkf ^v (t)=0).

The deduction point c _2k ^sub in the case of (sub) can be expressed, for example, as in the following formula (3). Also, the deduction point c _2k ^mW in the case of (mW) can be expressed, for example, as in the following formulas (4) and (5).

In this way, if the action a _k (t) results in a transmission rate r _bkf (t) that satisfies the required data rate R _kf , the reward Γ _k (t) will be a large value. On the other hand, if a sufficient transmission rate r _bkf (t) is not obtained, the reward Γ _k (t) will be a small value.

Once the reward _Γk (t) is calculated, the current experience _e (t) is stored in memory (step 88). Specifically, the current state _S (t), the selected action _a (t), the obtained reward _Γk (t), and the state _S (t+1) after the transition are stored in memory as the experience _e (t).

If there is a time correlation in the sample sequence, learning may not proceed properly. To solve this problem, when updating the parameter θ _i , memory samples are randomly taken from the experience e _k stored in the memory and processed in batches (step 90).

The parameter _θi of the DQN/DDQN is updated using the randomly acquired memory sample, i.e., the experience e _k including (S _k , a _k , Γ _k , S _k ′) (step 92). Here, S _k ′ refers to the state to which the transition is made by the action a _k . Specifically, the process of this step 92 proceeds as follows.

When the reward Γ _k for the action a _k is determined, a "new Q value" = Q(S _k , a _k ) for selecting the action a _k in the state S _k can be calculated based on the reward Γ _k . The new Q value is calculated, for example, as shown in the following formula (6).

Q(S _k , a _k )←Γ _k +γ*maxQ(S _k ′, a _k ′) ・・・(6)
where γ is the time discount rate.

The right-hand side of the above equation (6), _Γk +γ*maxQ(Sk _' , _ak '), is the value that you want the "trained DQN" to calculate as the Q value for the combination of ( _Sk , _ak ). Therefore, the parameter _θi of the DQN can be updated so that the value appears in the output layer for the input of ( _Sk , _ak ).

The following equation (7) shows a loss function L _k used when updating the DQN parameter θ _i in this embodiment.

In the above formula (7), Q( _S , _a ; _θ ) in the first term on the right side is the Q value calculated by the main network having the parameter _θi for the combination of (S, _a ), and Q( _{S '} , a _' ; _θ- ⁾ in the second term on the right side is the Q value calculated by the target network having ^the parameter _θ- for the combination of (S _' , a _' ).

When updating a DQN, if the network values to be updated are calculated using the network being updated, learning may not be stable. For this reason, when updating a DQN, it is common to use a target network for calculating the update values in addition to the main network to be updated. In this embodiment, the second term of equation (7), which corresponds to the new Q value, is calculated using the target network, following the above-mentioned normal method.

In this step 92, the parameters θ _i of the DQN, i.e., the parameters θ _i of the main network, are updated so as to minimize the loss function L _k shown in the above equation (7). As a result, the DQN learns to express the value on the right side of the above equation (6) as the Q value of the action a _k for the input of the state S _k .

The target network is the same as the previous main network except immediately after the update. The target network is then updated at a predetermined cycle (step 94). That is, when the time for updating arrives at the predetermined cycle, ^the parameter θ _i of the target network is overwritten by the parameter θ _i of the main network. The above process completes the update of the DQN.

In addition, the following formula (8) shows the loss function L _k used in the above step 92 when the DDQN method is used in this embodiment. The principle of the update is the same as that of the DQN, so a detailed explanation of this will be omitted.

[Processing by the wireless terminal CPU]
FIG. 9 is a flowchart for explaining the flow of the process executed by the CPU 72 of the wireless terminal 18 to realize the above functions.

In the routine shown in FIG. 9, first, the communication environment surrounding the wireless terminal 18 is confirmed (step 100). Specifically, accessible wireless base stations 16 are confirmed for each interface.

Next, it is determined whether a change has been detected in the communication environment (step 102).

If it is determined that no change has been observed, the ε coefficient is attenuated (step 104).

On the other hand, if a change in the communication environment is detected, the ε coefficient is reset to its initial value (step 106). This causes the attenuated ε to return to its original value.

Next, a random number between 0 and 1 is generated. It is then determined whether the random number is less than or equal to ε (step 108).

If it is determined that the random number≦ε holds, the next action a _k (t) for the current state S _k (t) is randomly determined (step 110).

On the other hand, if it is not determined in step 108 that the random number ≦ ε holds, the state S _k (t) is input to the input layer of the DQN (step 112). Then, the Q values appearing in the output layer of the DQN are compared, and the action a _i (t) showing the largest Q value is selected as the optimal action a _k (t) at the current time (step 114).

Next, based on the action a _k (t) selected by the above process, an association request is issued to the surrounding wireless base stations 16 (step 116).

When a response to the request is received, a reward Γ _k (t) for the selected action a _k (t) is calculated based on the information contained in the response (step 118).

Next, the experience e _k (t) acquired in the current routine, that is, the set of (S _k (t), a _k (t), Γ _k (t), S _k (t+1)), is stored in the RAM 78 constituting the memory unit 66 (step 120).

Then, the DQN is updated through batch processing (step 122), and this routine ends.

As described above, according to the wireless communication system 10 of this embodiment, in a heterogeneous environment including multiple wireless terminals 18 and multiple wireless base stations 16, the connection state can be managed using a neural network such as DQN or DDQN. Furthermore, by using the ε-greedy method, the learning of the network can be converged early. Furthermore, by initializing ε when the communication environment changes, the characteristics of the network as a value function can be appropriately adapted to the situation after the change. Therefore, according to the configuration of this embodiment, the system resources related to wireless communication can be constantly and effectively utilized.

In the above-described first embodiment, the wireless terminal 18 and the wireless base station 16 use two IFs for wireless communication, but the present disclosure is not limited to this. For example, there may be three or more IFs for wireless communication. Furthermore, the IFs for wireless communication may include a mixture of IFs for licensed bands and IFs for unlicensed bands used for wireless LANs and the like. Alternatively, there may be only one IF for wireless communication, and deep reinforcement learning such as DQN may be used to determine which wireless base station 16 the wireless terminal 18 should optimally request a connection to.

In addition, in the above-mentioned first embodiment, DQN or DDQN is used as the value function that determines the action. However, these are merely examples, and the present disclosure is not limited to these. Any function that can determine the value of each action that can be taken under a specific state can be used as the value function of the present disclosure.

10 Wireless communication system 16, 16-1, 16-2, 16-3 Wireless base station 18, 18 _k Wireless terminal 52 Control information calculation unit 60 Destination calculation unit 62 Environmental change detection unit 64 ε determination unit Γ _k (t) Reward a _k (t) Action e _k (t) Experience S _k (t) State S _k (t+1) State after transition R _kf Requested data rate r _bkf Transmission rate

Claims (8)

A wireless communication method for determining a combination of connections between a plurality of wireless base stations and a wireless terminal, comprising:
detecting a state of connection between the plurality of wireless base stations and the wireless terminal;
randomly determining with probability ε a behavior to be adopted by said wireless terminal under detected conditions;
determining, with probability (1-ε), an action to be taken by said wireless terminal under the sensed conditions in accordance with a result of a value function;
transmitting a request to said plurality of radio base stations corresponding to an action to be taken;
obtaining responses from the plurality of radio base stations to the request;
calculating a reward for the behavior based on the response;
updating a value for adopting the behavior under the state based on the reward;
learning parameters of the value function based on the updated values;
detecting a change in a communication environment surrounding the wireless terminal;
performing a time-dependent attenuation process on the ε under a condition where a change in the communication environment is not detected;
initializing the ε when a change in the communication environment is detected;
Including,
A wireless communication method in which the value function, when a state is given to an input layer, expresses in an output layer the value of adoption for each action that the wireless terminal can take . At least some of the plurality of wireless base stations have a plurality of wireless communication interfaces having different specifications,
the wireless terminal has the plurality of interfaces for wireless communication,
the state includes information on which wireless base station the wireless terminal is connected to for each interface;
The wireless communication method according to claim 1 , wherein the action includes information regarding which interface of the wireless terminal is to be connected to which wireless base station. The wireless communication method according to claim 1 or 2, wherein the value function is a Deep Q-Network. The wireless communication method according to claim 1 or 2, wherein the value function is a Double Deep Q-Network. A wireless terminal having a function of connecting to a plurality of wireless base stations,
A CPU is provided,
The CPU,
A process of detecting a state of connection between the plurality of wireless base stations and the wireless terminal;
a process of randomly determining with probability ε an action to be taken under the detected condition;
determining, with probability (1-ε), the action to be taken under the detected condition according to the result of the value function;
transmitting a request corresponding to an action to be taken to said plurality of wireless base stations;
obtaining responses from the plurality of wireless base stations to the request;
calculating a reward for the behavior based on the response;
updating a value for taking the action under the state based on the reward;
A process of learning parameters of the value function based on the updated value;
A process of detecting a change in a communication environment surrounding the wireless terminal;
a process of performing a time-dependent attenuation process on the ε under a situation where a change in the communication environment is not detected;
A process of initializing the ε when a change in the communication environment is detected;
Run
The value function is a wireless terminal that, when a state is given to an input layer, displays in an output layer the value of adoption for each action that the wireless terminal can take . A wireless communication device having a plurality of wireless communication interfaces each having different specifications;
The state includes information on which wireless base station the wireless terminal is connected to for each interface,
The wireless terminal according to claim 5 , wherein the action includes information regarding which interface of the wireless terminal is to be connected to which wireless base station. A program for a wireless terminal for realizing a function of connecting to a plurality of wireless base stations,
A CPU provided in the wireless terminal,
A process of detecting a state of connection between the plurality of wireless base stations and the wireless terminal;
a process of randomly determining with probability ε an action to be taken under the detected condition;
determining, with probability (1-ε), the action to be taken under the detected condition according to the result of the value function;
transmitting a request corresponding to an action to be taken to said plurality of wireless base stations;
obtaining responses from the plurality of wireless base stations to the request;
calculating a reward for the behavior based on the response;
updating a value for taking the action under the state based on the reward;
A process of learning parameters of the value function based on the updated value;
A process of detecting a change in a communication environment surrounding the wireless terminal;
a process of performing a time-dependent attenuation process on the ε under a situation where a change in the communication environment is not detected;
A process of initializing the ε when a change in the communication environment is detected;
A program for executing
A program for a wireless terminal , wherein the value function, when a state is given to an input layer, displays in an output layer the value of adoption for each action that the wireless terminal can take . the wireless terminal has a plurality of wireless communication interfaces having different specifications,
The state includes information on which wireless base station the wireless terminal is connected to for each interface,
8. The program for a wireless terminal according to claim 7, wherein the action includes information as to which interface of the wireless terminal is to be connected to which wireless base station.

Images