JP7618132B2 - Wireless communication method, wireless terminal, and program for wireless terminal - Google Patents

Description

This disclosure relates to a wireless communication method, a wireless terminal, and a program for the wireless terminal, and in particular to a wireless communication method, a wireless terminal, and a program for the wireless terminal that are suitable for achieving high communication efficiency in a network including multiple wireless terminals and multiple wireless base stations, etc.

As wireless communication systems develop, heterogeneous networks using multi-band, multi-access systems are reaching the stage of practical application. In cellular communications, fifth-generation mobile communications (5G) has been put to practical use. 5G communications systems use a wide range of frequencies, from sub-1 GHz bands to millimeter wave bands. In addition, an environment is being developed in which cells of various sizes, from small cells to macro cells, are provided in an overlapping manner.

Wireless LANs, another typical wireless access system, use radio frequencies in the 2.4/5/60 GHz bands, and the use of the 6 GHz band is also being considered. Wireless terminals such as smartphones are generally provided with interfaces (IFs) for accessing both cellular and wireless LANs. Furthermore, each IF is configured to support multiple frequency bands.

Wireless terminals generally select a wireless base station or access point (AP) to connect to from multiple frequency bands and access methods to establish communications. With functions such as dual connectivity, a single wireless terminal can also use multiple wireless base stations in an integrated manner. In such a heterogeneous environment, it is important to control and optimize across the entire system which IF a wireless terminal uses and which base station it selects in order to make effective use of system resources.

Regarding technology for effectively utilizing system resources in a heterogeneous environment, for example, Non-Patent Documents 1 and 2 below disclose a method for optimizing connections between wireless terminals and APs through reinforcement learning. More specifically, Non-Patent Document 2 discloses a technology for optimizing connections between wireless terminals using multiple IFs and multiple APs using Deep Q-Learning or Double Deep Q-Learning.

In addition, the following non-patent document 3 discloses Deep Q-Network (DQN), a method for handling high-dimensional inputs using reinforcement learning, and the following non-patent document 4 discloses Double Deep Q-Network (DDQN).

“Reinforcement Learning-aided Distributed User-to-Access Points Association in Interfering Networks”, Thi Ha Ly Dinh, Megumi Kaneko, Keisuke Wakao, Hirantha Abeysekera, Yasushi Takatori, IEEE Globecom, Waikoloa, Hawaii, USA, Dec. 2019 “Deep Reinforcement Learning-based User Association in Sub6GHz/mmWave Integrated Networks”, Thi Ha Ly Dinh, Megumi Kaneko, Keisuke Wakao, Kenichi Kawamura, Takatsune Moriyama, Hirantha Abeysekera and Yasushi Takatori, in IEEE CCNC, Jan. 2021 “Human-level control through deep reinforcement learning,” Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg & Demis Hassabis, Nature. 2015 Feb 26; Vol.518(7540): P529-533. doi: 10.1038/nature14236 “Deep Reinforcement Learning with Double Q-Learning,” H. Van Hasselt, A. Guez, and D. Silver, in Thirtieth AAAI conference on artificial intelligence, 2016

The above-mentioned non-patent document 1 discloses a method for updating DQN in the following procedure.
1. The "action a _k (t)" is which wireless base station the wireless terminal requests connection to.
2. When a wireless terminal selects a certain action a _k (t), a reward Γ _k (t) is calculated based on how much transmission data rate can be obtained from the wireless base station as a result.
3. DQN is used to calculate the value of adoption for each action a _k (t) that can be taken under a state S _k (t).
4. When an action a _k (t) is selected in a certain state S _k (t) and a reward Γ _k (t) is obtained as a result, the "value" of selecting that action a _k (t) in that state S _k (t) is updated based on that Γ _k (t).
5. The parameter θ characterizing the DQN is updated so that the “value” calculated by the DQN for the same combination of (S _k (t), a _k (t)) becomes the updated value.

This learning method allows multiple wireless base stations to determine connections between them so that the transmission rates they provide to multiple wireless terminals are optimized. Therefore, the above learning method is useful for making effective use of the resources of a wireless communication system.

However, between a wireless base station and a wireless terminal, in addition to downlink (DL) communication from the former to the latter, uplink (UL) communication from the latter to the former also takes place. In conventional learning methods, including the techniques described in Non-Patent Documents 1 and 2 above, connections are generally optimized by taking into account only DL communication. However, techniques that attempt optimization by taking into account only DL communication, such as the above learning method, may not be able to optimize both DL and UL.

The present disclosure has been made in light of the above-mentioned problems, and aims to provide a wireless communication method, a wireless terminal, and a program for a wireless terminal that can optimize both DL and UL in a network including multiple wireless terminals and multiple base stations, etc.

In order to achieve the above object, a first aspect of the present disclosure is a wireless communication method for determining a combination of connections between a plurality of wireless base stations and a wireless terminal, the method comprising:
detecting a state of connection between the plurality of radio base stations and the radio terminal by distinguishing between DL communication and UL communication;
inputting said state into a value function;
a step in which the value function outputs a value of each of actions that the wireless terminal can take under the state, and each of the actions is determined by distinguishing between DL communication and UL communication and determining to which wireless base station a connection request should be made;
transmitting a request to a radio base station corresponding to an action to be taken;
obtaining a response to the request from the wireless base station;
a reward calculation step of calculating a reward reflecting both the DL evaluation and the UL evaluation based on the response;
updating a value for adopting the behavior under the state based on the reward;
learning parameters of the value function based on the updated values;
It is preferable that the present invention includes the following:

A second aspect of the present disclosure is a wireless terminal having a function of connecting to a plurality of wireless base stations,
A CPU is provided,
The CPU,
detecting a state of connection between the plurality of wireless base stations and the wireless terminal by distinguishing between DL communication and UL communication;
inputting said state into a value function;
a process of calculating a value of each of actions that the wireless terminal can take under the state by using the value function, and determining to which wireless base station a connection request is to be made for each of the actions by distinguishing between DL communication and UL communication;
transmitting a request to a wireless base station corresponding to the action to be taken;
obtaining a response to the request from a wireless base station;
A reward calculation process for calculating a reward reflecting both the DL evaluation and the UL evaluation based on the response;
updating a value for taking the action under the state based on the reward;
A process of learning parameters of the value function based on the updated value;
It is advisable to carry out the following:

A third aspect of the present disclosure is a program for a wireless terminal for realizing a function of connecting to a plurality of wireless base stations, comprising:
A CPU provided in the wireless terminal,
detecting a state of connection between the plurality of wireless base stations and the wireless terminal by distinguishing between DL communication and UL communication;
inputting said state into a value function;
a process of calculating a value of each of actions that the wireless terminal can take under the state by using the value function, and determining to which wireless base station a connection request is to be made for each of the actions by distinguishing between DL communication and UL communication;
transmitting a request to a wireless base station corresponding to the action to be taken;
obtaining a response to the request from a wireless base station;
A reward calculation process for calculating a reward reflecting both the DL evaluation and the UL evaluation based on the response;
updating a value for taking the action under the state based on the reward;
A process of learning parameters of the value function based on the updated value;
It is preferable that the program include a program for executing the above.

According to the present disclosure, both the DL evaluation and the UL evaluation are reflected in the reward for an action selected by a wireless terminal under a certain state. Then, parameters of an evaluation function indicating the optimal action under a certain state are learned based on the reward. Through such learning, it is possible to converge the value function so that both DL and UL are comprehensively optimized. Therefore, according to the present disclosure, it is possible to comprehensively optimize two-way communication in a heterogeneous environment where DL and UL coexist.

1 is a diagram for explaining an overall configuration of a wireless communication system according to a first embodiment of the present disclosure. FIG. 2 is a diagram for explaining an example of a connection state between a wireless terminal and a wireless base station in the first embodiment of the present disclosure. 1 is a diagram showing a state in which the sub-6 GHz band and the millimeter wave band are all used for downlink communication in a heterogeneous environment according to a first embodiment of the present disclosure. FIG. FIG. 4 is a diagram showing a state in which a part of the millimeter wave band is used for uplink communication in the environment shown in FIG. 3. FIG. 2 is a block diagram for explaining a functional configuration of a radio base station used in the first embodiment of the present disclosure. FIG. 2 is a block diagram for explaining a functional configuration of a wireless terminal used in the first embodiment of the present disclosure. 2 is a diagram for explaining the hardware configuration of a wireless terminal used in the first embodiment of the present disclosure. FIG. FIG. 1 is a diagram for explaining an overview of DQN used in the first embodiment of the present disclosure. FIG. 11 is a diagram for explaining the contents of a process related to updating a DQN used in the first embodiment of the present disclosure. 1 is a flowchart for explaining a flow of a process executed by a wireless terminal in the first embodiment of the present disclosure to achieve an optimal connection state.

Embodiment 1.
[Overall Configuration of First Embodiment]
Fig. 1 is a diagram for explaining an overall configuration of a wireless communication system 10 according to a first embodiment of the present disclosure. As shown in Fig. 1, the wireless communication system 10 includes a wired communication network 12. A plurality of wireless base stations 16 are connected to the network 12 via a switch 14. Each wireless base station 16 can communicate with another wireless base station 16 via the network 12 and via a wireless path.

The wireless communication system 10 shown in FIG. 1 includes multiple wireless terminals 18. In this embodiment, the wireless terminals 18 include mobile terminals such as smartphones or tablet terminals.

Each wireless terminal 18 has multiple sets of applications and IFs for wireless communication. For example, wireless terminal 18 has applications and IFs for communication in the band below 6 GHz (hereinafter referred to as "sub6GHz"), and applications and IFs for communication in the millimeter wave band (hereinafter referred to as "mmWave"). Furthermore, wireless terminal 18 may have applications and IFs for communication in the 2.4/5/60 GHz bands used in wireless LANs.

The above-mentioned wireless base stations 16 are also equipped with corresponding IFs, etc. Then, each wireless terminal 18 can wirelessly communicate with one or more wireless base stations 16 via the above-mentioned applications and IFs. In this way, the wireless communication system 10 of this embodiment realizes the function of a heterogeneous network with multi-band multi-access.

In addition, in this embodiment, the wireless terminal 18 manages the connections used for communication for each downlink (DL) and uplink (UL). More specifically, the wireless terminal 18 manages the correspondence between each of the multiple applications and the IF to be used and the wireless base station 16 to which the applications belong, distinguishing between DL and UL.

Meanwhile, the wireless base station 16 judges the DL connection request and the UL connection request from the wireless terminal 18 for each application, and feeds back a response as to whether or not the request is accepted. Then, based on the received response, the wireless terminal 18 calculates the reward Γ for the currently requested combination of connections, taking into account both the results obtained in DL and UL.

The wireless communication system 10 of this embodiment may also be compatible with wireless LAN access points (APs) in addition to wireless base stations for cellular communication installed by telecommunications carriers. For the sake of convenience, the following description will be given assuming that "wireless base station 16" encompasses both.

[Problems with the First Embodiment]
Next, the features of the wireless terminal 18 in this embodiment will be described with reference to Figures 2 to 12. In the following description, it is assumed that the wireless base station 16 and the wireless terminal 18 are provided with an interface IF1 corresponding to communication in a first frequency band and an interface IF2 corresponding to communication in a second frequency band. IF1 is, for example, an interface corresponding to sub6GHz, and IF2 is, for example, an interface corresponding to mmWave.

In the following description, when it is necessary to specify a specific wireless terminal 18 among multiple wireless terminals, a reference symbol including a letter such as k will be used, such as "wireless terminal 18 _k ." Furthermore, when it is necessary to distinguish between the multiple wireless base stations 16, reference symbols with suffixes, such as 16-1 and 16-2, will be used.

2 shows a state in which a wireless terminal 18 _k is associated with two wireless base stations 16-1 and 16-2. As described above, the wireless terminal 18 _k and the wireless base station 16 are provided with two interfaces, IF1 and IF2, corresponding to different bands. In the example shown in FIG. 2, a state is formed in which the wireless terminal 18 _k is associated with a first wireless base station 16-1 at IF1 and is associated with a second wireless base station 16-2 at IF2.

Fig. 3 shows a situation in which four wireless base stations 16-1 to 16-4 and three wireless terminals 18 _i , 18 _j , and 18 _k exist in a heterogeneous environment with multi-band multi-access. As in the case shown in Fig. 2, each of them has IF1 corresponding to sub6 GHz and IF2 corresponding to mmWave.

Sub-6GHz communication waves have the advantage of being able to get around obstacles, making them suitable for securing a wide communication area. In FIG. 3, the elliptical areas drawn around each of the wireless base stations 16-1 to 16-4 indicate the communication areas that each of the wireless base stations 16-1 to 16-4 achieves by using IF1, which supports sub-6GHz.

Although mmWave communication waves have a tendency to travel in a straight line and are not suitable for covering a wide area like sub6GHz, they are suitable for realizing high-speed, large-capacity communication. In Fig. 3, ellipses containing arrows are drawn in three places, including between the wireless base station 16-2 and the wireless terminal _18k . These ellipses represent the direction and width of the beamformed waveform.

3, it is assumed that all of the wireless terminals _18i , _18j , and _18k are connected to the wireless base stations 16-1 to 16-4 by DL communication. According to this assumption, the system capacity can be maximized by determining the connection destination such that the transmission rate of data received by each of the wireless terminals _18i , _18j , and _18k from the multiple wireless base stations 16 is maximized. However, in the wireless communication system 10 of the present embodiment, UL communication may be used.

4 shows a state where IF2 of the wireless terminal 18 _k is used for UL communication with the wireless base station 16-2. In an environment where DL communication and UL communication coexist as shown in FIG. 4, a situation occurs where the system capacity cannot be maximized by optimization that evaluates only DL communication. Therefore, in this embodiment, DL communication and UL communication are distinguished, and both are evaluated to optimize the connection.

[Configuration of wireless base station]
5 is a block diagram for functionally explaining an example of the wireless base station 16. The wireless base station 16 includes an antenna unit 20. The antenna unit 20 is used by the wireless base station 16 to transmit and receive wireless signals to and from other wireless base stations 16 and surrounding wireless terminals 18.

The antenna unit 20 is connected to the wireless communication unit 22. The wireless communication unit 22 includes an RF (Radio Frequency) unit 24 that processes high-frequency signals, and a MAC (Media Access Control) function unit 26 that has a function of media access control in wireless communication. The wireless communication unit 22 can realize functions corresponding to multiple frequency bands, specifically, a function as IF1 corresponding to sub6GHz, and a function as IF2 corresponding to mmWave.

In this embodiment, each of the multiple IFs (here, IF1 and IF2) realized by the wireless communication unit 22 may be used for both DL and UL communications. However, the optimization process described below is performed assuming that each IF is assigned to only one of DL and UL. In addition, this optimization process is performed assuming that IFs of multiple wireless base stations 16 are not assigned simultaneously to one wireless terminal 18 for DL or UL.

The wireless base station 16 includes a quality measurement unit 28. The quality measurement unit 28 includes a wireless environment measurement unit 30 and a traffic measurement unit 32. The wireless environment measurement unit 30 acquires information related to the wireless communication environment surrounding the wireless base station 16. The traffic measurement unit 32 acquires information related to the traffic between the wireless base station 16 and the wireless terminal 18. The quality measurement unit 28 calculates information on the communication quality between the wireless base station 16 and the wireless terminal 18 based on the information. The communication quality information includes, for example, the received power from the wireless terminal 18, the required data rate of the wireless terminal 18, the actual transmission rate, and the bandwidth usage rate in the wireless base station 16.

The wireless base station 16 includes a communication IF unit 34. The communication IF unit 34 is used for wired communication via the network 12. The wireless base station 16 can acquire communication quality information acquired by other wireless base stations 16 by wired communication via the communication IF unit 34, or by wireless communication via the wireless communication unit 22 and antenna unit 20. The communication quality information acquired from other wireless base stations 16 is managed in a unified manner in the quality measurement unit 28, together with the communication quality information acquired by the wireless base station 16 itself.

The wireless base station 16 includes an information notification unit 36. The information notification unit 36 can read out the communication quality information aggregated in the quality measurement unit 28. The information notification unit 36 can provide the read out communication quality information to the communication IF unit 34 and the wireless communication unit 22. This allows the wireless base station 16 to provide its own communication quality information to other wireless base stations 16 via a wired or wireless route.

The wireless base station 16 receives DL and UL attribution requests from nearby wireless terminals 18 via the wireless communication unit 22 and antenna unit 20. These attribution requests are processed by the request information evaluation unit 38. The request information evaluation unit 38 determines whether or not to allow attribution based on the bandwidth and data rate requested by the wireless terminal 18, as well as its own load state, etc.

The result of the determination by the request information evaluation unit 38 is sent to the request response notification unit 40 and the belonging information recording unit 42. The request response notification unit 40 provides the determination result regarding the possibility of belonging for each of the DL and UL to the wireless terminal 18 that issued the belonging request via the wireless communication unit 22 and the antenna unit 20. The belonging information recording unit 42 updates and records the information of the wireless terminal 18 belonging to the wireless base station 16 for each of the DL and UL based on the result of the determination.

[Wireless Terminal Configuration]
6 is a block diagram for functionally explaining an example of the wireless terminal 18. The wireless terminal 18 includes an antenna unit 44 and a wireless communication unit 46. The wireless communication unit 46 includes an RF unit 48 and a MAC function unit 50. These functions are substantially the same as the functions of the corresponding blocks included in the wireless base station 16, and therefore description thereof will be omitted here.

In this embodiment, the wireless terminal 18 includes a DL/UL management unit 52. The DL/UL management unit 52 executes processes necessary for DL communication and UL communication in the wireless terminal 18.

The wireless terminal 18 includes a control information calculation unit 54. The control information calculation unit 54 acquires communication quality information from the wireless base station 16 to which the wireless terminal 18 belongs via the wireless communication unit 46 and the antenna unit 44. The control information calculation unit 54 further performs an optimization process using the DQN or DDQN based on the acquired communication quality information. Specifically, the control information calculation unit 54 executes a process of determining, using the DQN or DDQN, the optimal action a _k (t) for connecting each of a plurality of IFs to which IF of which wireless base station 16 and for which purpose of DL and UL, and a learning process of the DQN or DDQN.

The control information calculation unit 54 includes a state calculation unit 56. The state calculation unit 56 calculates a state S _k (t) to be provided to an input layer of the DQN or DDQN based on the communication quality information. The state S _k (t) includes information on which IF of which wireless base station 16 each of the IFs owned by the mobile station belongs to for each of the DL and UL.

The control information calculation unit 54 also includes a reward calculation unit 58. The reward calculation unit 58 calculates a reward Γ _k (t) obtained by the adopted action a _k (t) based on the communication quality information. The reward Γ _k (t) is calculated so as to reflect both the evaluation regarding the DL communication and the evaluation regarding the UL communication. The method of this calculation will be described in detail later.

The control information calculation unit 54 further includes a DQN/DDQN update unit 60. The DQN/DDQN update unit 60 updates the parameter θ of the DQN or DDQN used in reinforcement learning. More specifically, the weighting coefficient and bias value (hereinafter referred to as "weighting coefficient, etc.") that determine the characteristics of the DQN or DDQN as a value function are updated.

The DQN or DDQN updated by the DQN/DDQN update unit 60 is used in the attribute calculation unit 62. That is, the attribute calculation unit 62 calculates the action a _k (t) to be adopted this time, that is, which IF should be attributed to which IF of which wireless base station 16 and for which purpose of DL and UL, according to the updated DQN or DDQN.

In DQN and DDQN, when a state S _k (t) is given to the input layer, the value of adoption for each of the actions a _k (t) that the wireless terminal 18 can take is displayed in the output layer. Therefore, by selecting the action with the largest value, it is possible to select an action that is considered optimal at the current learning stage. However, if the next action a _k (t) is always selected based on the above value, it may become trapped in a local solution selected by chance, and the truly optimal action may never be selected.

In order to avoid such a situation, the assignment calculation unit 62 calculates the action a _k (t) according to the ε-greedy method. Therefore, the assignment calculation unit 62 randomly selects the next action a _k (t) with a probability ε, regardless of whether it is DQN or DDQN. Also, the assignment calculation unit 62 selects the action a _k (t) that DQN or DDQN judges to be optimal with a probability of 1-ε.

Furthermore, the destination calculation unit 62 performs a decay process on ε each time learning is repeated. The possibility of falling into a local solution selected by chance is large in the early stages of learning and decreases as learning progresses. Therefore, the necessity of randomly selecting an action a _k (t) decreases as learning progresses. By performing a decay process on ε, the frequency with which the action a _k (t) is randomly selected can be reduced in line with the decrease in the necessity. Therefore, according to the destination calculation unit 62 of this embodiment, learning can be efficiently advanced and the parameter θ can be converged to an optimal value in a short period of time.

The wireless terminal 18 includes a memory unit 66. The memory unit 66 stores the following information as experience e _k (t):
1. The state S _k (t) provided to the input layer of DQN or DDQN in this learning,
2. Action a _k (t) determined according to the state S _k (t),
3. The reward Γ _k (t) obtained by the action a _k (t),
4. State S _k (t+1) to which the transition is made by the action a _k (t).

The wireless terminal 18 also includes an attachment request unit 68. The attachment request unit 68 receives the action a _k (t) from the attachment calculation unit 62, and generates a request corresponding to the action a _k (t). For example, the request is generated to connect to the wireless base station 16-1 using IF1 for DL use, or to connect to the wireless base station 16-2 using IF2 for UL use. The generated request is transmitted to one or more wireless base stations 16 via the wireless communication unit 46 and the antenna unit 20.

The wireless terminal 18 is equipped with an attachment destination recording unit 70. The information received from the wireless base station 16 includes a response to the request as to whether or not the request is accepted. For example, the information includes a response to permit a new connection, or a response to permit an update of a connection. Based on these responses, the attachment destination recording unit 70 updates and records information about the wireless base station 16 to which the wireless terminal 18 is attached.

Figure 7 is a diagram for explaining the hardware configuration of the wireless terminal 18. Here, a case where the wireless terminal 18 is a smartphone will be explained. However, the wireless terminal 18 is not limited to a smartphone, and may be a tablet terminal or a general-purpose computer system.

The wireless terminal 18 is equipped with a CPU (Central Processing Unit) 72. The CPU 72 is connected to various hardware elements equipped in the wireless terminal 18 via a bus wiring 74. Various memory devices, such as a ROM (Read Only Memory) 76, a RAM (Random Access Memory) 78, and a storage 80, are connected to the bus wiring 74.

The storage 80 includes a non-volatile memory such as a flash ROM. Programs and data necessary for executing various applications are stored in the storage 80. The memory unit 66 and the destination recording unit 70 described above are realized by the storage 80. The CPU 72 also realizes the functions of the DL/UL management unit 52 and the control information calculation unit 54 shown in FIG. 6 by carrying out processing according to the programs stored in the storage 80.

The wireless terminal 18 includes a communication interface 82. The communication interface 82 corresponds to the antenna unit 44 and wireless communication unit 46 described above. The wireless terminal 18 further includes an operation unit 84 and a display unit 86. The operation unit 84 includes a touch panel integrated with the display unit 86, in addition to various mechanical switch mechanisms and electronic sensor mechanisms.

[Overview of DQN]
Fig. 8 is a diagram for explaining an overview of DQN. Hereinafter, an overview of DQN, which is an example of deep reinforcement learning used by the wireless terminal 18 _k , will be explained with reference to Fig. 8. The DQN used in this embodiment has an input layer, a hidden layer, and an output layer. In addition, a parameter θ _i ^t corresponding to the learning state at the current time t is set in the DQN. The characteristics of the DQN as a value function are determined by the parameter θ _i ^t .

8, the input layer of DQN has M nodes S ₁ to S _M. Elements constituting the state S _k of the wireless terminal 18 _k are input to these nodes. The elements of the state S _k include, for example, the following elements. These are handled separately as those related to DL and those related to UL.
1. Bandwidth utilization rate φb of wireless base station 16-b,
2. The required data rate R _kf of the interface f of the wireless terminal 18 _k ,
3. The relationship x _bkf between the wireless terminal 18 _k and the wireless base station 16-b (1 if the wireless terminal 18 k belongs to the wireless base station 16-b, 0 if not),
4. A request a _bkf for association of the interface f of the wireless terminal 18 _k to the wireless base station 16-b (1 if there is a request, 0 if not),
5. The transmission rate r _bkf established between the interface f of the wireless terminal 18 _k and the wireless base station 16-b.

The number of hidden layers and the number of nodes are arbitrary. The output layer has a number of nodes equal to the total number N of actions _ai that the wireless terminal _18k can take in the state _Sk . Some of the actions _ai are shown below.
- IF1 is assigned to the wireless base station 16-1 for DL use.
- IF1 is attributed to the wireless base station 16-1 for UL use.
- IF2 is assigned to the wireless base station 16-1 for DL use.
- IF2 is attributed to the wireless base station 16-1 for UL use.
IF1 is attributed to the wireless base station 16-1 for DL use, and IF2 is attributed to the wireless base station 16-2 for UL use.
IF1 is attributed to the wireless base station 16-1 for UL use, and IF2 is attributed to the wireless base station 16-2 for DL use.
・
・
・

Each node in the output layer displays a Q value = Q (S _k , a _i ; θ _i ^t ) for each action a _i (i = 1 to N) that the wireless terminal 18 _k can take. The Q value is the adoption value for the action a _i calculated by the DQN under the parameter θ _i ^t .

In the method using DQN, the largest value is selected from the Q values appearing in the output layer, and the action a _i that produces that Q value is determined as the optimal action a _k (t) at the current time t. In this way, according to DQN, it is possible to determine the action a _k (t) that can be determined as optimal at the time t in each learning stage.

Fig. 9 is a diagram for explaining the flow in which the control information calculation unit 54 of the wireless terminal 18 _k updates the parameter θ _i of the DQN. Note that in Fig. 8, the action a _k (t) is shown after the DQN/DDQN for convenience, but the action a _k (t) is determined by the DQN or DDQN only when the probability is 1-ε, as described above. Under the probability ε, the action a _k (t) is determined randomly. The parameter θ _i reflects the experience e _k (t) obtained in both cases.

After the action a _k (t) is determined in the wireless terminal 18 _k , the reward Γ _k (t) for the action a _k (t) is calculated. The reward Γ _k (t) is calculated, for example, by the following formula (1).

In the above formula (1), the first term on the right side is a point-adding item, and the second term is a point-deducting item. ω _1k and ω _2k attached to both are coefficients that determine the allocation ratio for them. λ is an identifier that indicates each element of the set Λ = {DL, UL}. ν is an identifier that indicates each element of the set L = {sub6GHz, mmWave}.

Furthermore, in the above formula (1), c _1k ^λ,ν in the first term on the right side means the added point when the transmission rate r _bkf (t) that satisfies the required data rate R _kf ^λ obtained at λ (DL or UL) is obtained at ν (sub6GHz or mmWave). The value is calculated by the following formula (2).

The above formula (2) means that an additional point r _bkf (t)/W _λ,ν according to the transmission rate r _bkf (t) is given in the following cases, where W _λ,ν in the last term is a factor for offsetting and normalizing the difference in frequency bands caused by the difference in λ (DL or UL) and ν (sub6GHz or mmWave).
1. The wireless base station 16-b recognizes the association of the interface f of the wireless terminal 18 _-k with respect to v (sub6GHz or mmWave) for the purpose of λ (DL or UL) (x _bkf ^{λ, v} =1), and a transmission rate r _bkf (t) equal to or greater than the requested data rate R _kf ^λ is obtained for λ (DL or UL) (r _bkf (t) ≧ R _kf ^λ ).
2. The interface f of the wireless terminal 18 _k has requested association at v (sub6 GHz or mmWave) for the purpose of λ (DL or UL) to the wireless base station 16 - b (a _bkf ^{λ, v} (t) = 1).

In the above formula (1), c _2k ^λ,ν in the second term on the right side means the point deduction when the transmission rate r _bkf (t) satisfying the required data rate R _kf ^λ at λ (DL or UL) cannot be obtained at ν (sub6GHz or mmWave). The value is calculated, for example, by the following formula (3).

The above formula (3) means that in the following two cases, the reward Γ _k (t) is deducted according to the ratio between the requested data rate R _kf ^λ and the transmission rate r _bkf (t).
1. For an application of λ (DL or UL), a request for association at ν (sub6GHz or mmWave) is made (a _bkf ^λ,ν (t) = 1), and association is granted (x _bkf ^λ,ν (t) = 1), but a transmission rate r _bkf (t) that satisfies the requested data rate R _kf ^λ is not obtained (r _bkf (t) < R _kf ^λ ).
2. A request for association at v (sub6GHz or mmWave) is made for the purpose of λ (DL or UL) (a _bkf ^λ,v (t) = 1), but the radio base station 16 does not permit association for the requested λ (DL or UL) (x _bkf ^λ,v (t) = 0).

In this way, the reward Γ _k (t) becomes large if the action a _k (t) results in a transmission rate r _bkf (t) that satisfies the required data rate R _kf ^λ . On the other hand, if a sufficient transmission rate r _bkf (t) is not obtained, the reward Γ _k (t) becomes small.

The transmission rate r _bkf (t) included in the above formula (3) can be calculated by the following formula (4) for each of the following four situations.
1. A wireless terminal 18 _k belongs to the wireless base station 16-b in sub6 GHz for DL use.
2. A wireless terminal 18 _k belongs to the wireless base station 16-b at sub6 GHz for UL use.
3. The wireless terminal 18 _k is associated with the wireless base station 16-b by mmWave for DL use.
4. The wireless terminal 18 _k is associated with the wireless base station 16-b by mmWave for UL purposes.

Moreover, γ _bkf ^λ,ν (t) included in the four right-hand sides of the above formula (4) is the SINR (Signal to Interference plus Noise power Ratio) under each condition, which can be calculated by the following formulas (5) to (8).

In the above formula (5), p _bkf ^DL,sub (t) is the transmission power in DL, sub6GHz. Also, h _bk ^sub (t) is the correlation coefficient in sub6GHz between the wireless base station 16-b and the wireless terminal 18k. The value becomes large when the distance between the two is short, and is affected by propagation loss and fading. _{h b'k} ^sub (t) and h _k'k ^sub (t) are also correlation coefficients under the environment indicated by the respective subscripts. Furthermore, W _DL,sub at the end of the denominator represents the bandwidth in DL, sub6GHz, and is used to calculate the power of the noise part. The same applies to p _bkf ^λ,ν (t), h _bk ^ν (t), etc., and W _λ,ν in the above formulas (6) to (8).

As described above, in this embodiment, the communication quality between the wireless terminal 18 _k and the wireless base station 16 is evaluated for each of the DL and UL. Then, both of these are reflected in the reward Γ _k (t) for the selected action a _k (t).

Once the reward _Γk (t) has been calculated in this manner, the current experience _e (t) is stored in memory (step 88). Specifically, the current state _S (t), the selected action _a (t), the obtained reward _Γk (t), and the state S( _t +1) after the transition are stored in memory as the experience _e (t).

If there is a time correlation in the sample sequence, learning may not proceed properly. To solve this problem, when updating the parameter θ _i , memory samples are randomly taken from the experience e _k stored in the memory and processed in batches (step 90).

The parameter _θi of the DQN/DDQN is updated using the randomly acquired memory sample, i.e., the experience e _k including (S _k , a _k , Γ _k , S _k ′) (step 92). Here, S _k ′ refers to the state to which the transition is made by the action a _k . Specifically, the process of this step 92 proceeds as follows.

When the reward Γ _k for the action a _k is determined, a "new Q value" = Q(S _k , a _k ) for selecting the action a _k in the state S _k can be calculated based on the reward Γ _k . The new Q value is calculated, for example, as shown in the following formula (9).

Q(S _k , a _k )←Γ _k +γ*maxQ(S _k ′, a _k ′) ・・・(9)
where γ is the time discount rate.

The right-hand side of the above equation (9), _Γk +γ*maxQ(Sk _' , _ak '), is the value that you want the "trained DQN" to calculate as the Q value for the combination of ( _Sk , _ak ). Therefore, the parameter _θi of the DQN can be updated so that the value appears in the output layer for the input of ( _Sk , _ak ).

The following equation (10) shows a loss function L _k used when updating the DQN parameter θ _i in this embodiment.

In the above formula (10), Q( _S , _a ; _θ ) in the first term on the right side is the Q value calculated by the main network having the parameter _θ for the combination of ( ^S , _a ), and Q( _{S '} , a _' ; _θ- ⁾ in the second term on the right side is the Q value calculated by the target network having the parameter _θ- for the combination of (S _' , a _' ).

When updating a DQN, if the network values to be updated are calculated using the network being updated, learning may not be stable. For this reason, when updating a DQN, it is common to use a target network for calculating the update values in addition to the main network to be updated. In this embodiment, the second term of equation (10), which corresponds to the new Q value, is calculated using the target network, following the above-mentioned normal method.

In this step 92, the parameters θ _i of the DQN, i.e., the parameters θ _i of the main network, are updated so as to minimize the loss function L _k shown in the above equation (10). As a result, the DQN learns to express the value on the right side of the above equation (9) as the Q value of the action a _k for the input of the state S _k .

The target network is the same as the previous main network except immediately after the update. The target network is then updated at a predetermined cycle (step 94). That is, when the time for updating arrives at the predetermined cycle, ^the parameter θ _i of the target network is overwritten by the parameter θ _i of the main network. The above process completes the update of the DQN.

In addition, the following formula (11) shows the loss function L _k used in the above step 92 when the DDQN method is used in this embodiment. The principle of the update is the same as that of the DQN, so a detailed explanation of this will be omitted.

[Processing by the wireless terminal CPU]
10 is a flowchart for explaining the flow of processing executed by the CPU 72 of the wireless terminal 18 to realize the above-mentioned functions. Note that, here, a case where DQN is used as the reinforcement learning method will be explained.

10, first, a state S _k (t) to be provided to the input layer of DQN is set based on communication quality information and the like acquired from the surrounding wireless base stations 16 (step 100). As described above, the state S _k (t) includes connection information related to DL and connection information related to UL.

Next, ε to be used in this learning is set (step 102). For example, a value of about 0.1 is given to ε as an initial value, and then ε is decayed each time this routine is repeated.

Next, a random number between 0 and 1 is generated. It is then determined whether the random number is less than or equal to ε (step 104).

If it is determined that the random number ≦ε, the next action a _k (t) for the current state S _k (t) is randomly determined (step 106). This action a _k (t) includes information specifying the wireless base station 16 to which the mobile station is to belong for each of the DL and UL.

On the other hand, if it is not determined in step 104 that the random number ≦ ε holds, the state S _k (t) is input to the input layer of the DQN (step 108). Then, the Q values appearing in the output layer of the DQN are compared, and the action a _i (t) showing the largest Q value is selected as the optimal action a _k (t) at the current time (step 110).

Next, based on the action a _k (t) selected by the above process, an attribution request is issued to the surrounding wireless base stations 16 (step 112). Specifically, one or more connection requests are issued that specify whether the wireless base station is DL or UL, whether it is sub6 GHz or mmWave, and the wireless base station 16-b to which the wireless base station is to be connected.

In the wireless base station 16 that has received the connection request, it is determined whether or not a connection can be established for each of the DL and UL. The determination of whether or not a connection can be established is performed for each of the DL and UL by, for example, a branch and bound method, in the following procedure.
1. All connection requests from the wireless terminals 18 are sorted in descending order of transmission efficiency (r _bkf /φ _bkf ) (where φ _bkf is the load when the wireless base station 16-b obtains the transmission rate r _bkf for the application f of the wireless terminal 18 _k ).
2. Select the candidates for belonging in the order of the above sorting and judge whether they can be accommodated or not. If they can be accommodated with the remaining capacity, the connection is allowed, otherwise it is rejected.
3. After completing the judgment for all the candidates, a response including the result is transmitted to the nearby wireless terminal 18.

When the wireless terminal _18k receives the response to the request, it acquires the communication quality information contained in the response (step 114).

Next, γ _bkf ^λ,ν (t) is calculated using the above formulas (5) to (8) based on the communication quality information (step 116). This allows the SINR to be calculated for each condition determined by either DL or UL, and either sub6GHz or mmWave.

Next, the reward Γ _k (t) for the current action a _k (t) is calculated by applying the calculated value of γ _bkf ^λ,ν (t) to the above formulas (1) to (4) (step 118). This allows the reward Γ _k (t) that reflects both the evaluation regarding DL and the evaluation regarding UL to be calculated.

When the above processing is completed, the experience e _k (t) acquired in the current routine, i.e., the set of (S _k (t), a _k (t), Γ _k (t), S _k (t+1)), is stored in the RAM 78 constituting the memory unit 66 (step 120).

Then, the DQN is updated through batch processing (step 122), and this routine ends.

As described above, the wireless terminal 18 in this embodiment calculates the reward Γ _k (t) reflecting both the evaluation related to DL and the evaluation related to UL. Then, the parameter θ _i of the DQN or DDQN is updated based on the Q value updated based on the reward Γ _k (t). Therefore, according to this embodiment, in a heterogeneous environment in which DL and UL coexist, it is possible to learn the DQN or DDQN so as to comprehensively optimize the quality of both of them. Then, the connections between the multiple wireless terminals 18 and the multiple wireless base stations 16 are determined according to the DQN or DDQN, so that bidirectional communication can be comprehensively optimized in a heterogeneous environment in which DL and UL coexist.

In the above-described first embodiment, the wireless terminal 18 and the wireless base station 16 use two IFs for wireless communication, but the present disclosure is not limited to this. For example, the number of IFs for wireless communication may be three or more. Furthermore, the IFs for wireless communication may include a mixture of IFs for licensed bands and IFs for unlicensed bands used for wireless LANs, etc.

In addition, in the above-mentioned first embodiment, DQN or DDQN is used as the value function that determines the action. However, these are merely examples, and the present disclosure is not limited to these. Any function that can determine the value of each action that can be taken under a specific state can be used as the value function of the present disclosure.

10 Wireless communication system 16, 16-1, 16-2, 16-3 Wireless base station 18, 18 _i , 18 _j , 18 _k Wireless terminal 52 DL/UL management unit 54 Control information calculation unit 60 DQN/DDQN update unit Γ _k (t) Reward a _k (t) Action e _k (t) Experience S _k (t) State S _k (t+1) State after transition R _kf Requested data rate r _bkf Transmission rate θ _i Parameter

Claims (8)

A wireless communication method for determining a combination of connections between a plurality of wireless base stations and a wireless terminal, comprising:
detecting a state of connection between the plurality of radio base stations and the radio terminal by distinguishing between DL communication and UL communication;
inputting said state into a value function;
a step in which the value function outputs a value of each of actions that the wireless terminal can take under the state, and each of the actions is determined by distinguishing between DL communication and UL communication and determining to which wireless base station a connection request should be made;
transmitting a request to a radio base station corresponding to an action to be taken;
obtaining a response to the request from the wireless base station;
a reward calculation step of calculating a reward reflecting both the DL evaluation and the UL evaluation based on the response;
updating a value for adopting the behavior under the state based on the reward;
and learning parameters of the value function based on the updated values ;
A wireless communication method, wherein the state input to the value function includes the presence or absence of a DL-related association request and the presence or absence of a UL-related association request of the wireless terminal to each of the plurality of wireless base stations . At least some of the plurality of wireless base stations have a plurality of wireless communication interfaces having different specifications,
the wireless terminal has the plurality of interfaces for wireless communication,
The state includes information regarding which radio base station the radio terminal is connected to, through which interface, and whether DL or UL, the radio terminal is connected to;
The wireless communication method according to claim 1 , wherein the action includes information on which interface of the wireless terminal is to be connected to which wireless base station in either DL or UL. The remuneration calculation step includes:
calculating a SINR for DL communication requested by the wireless terminal;
calculating a transmission rate obtained in DL communication based on the SINR for DL;
calculating a SINR in an UL communication requested by the wireless terminal;
calculating a transmission rate obtained in an UL communication based on the SINR for the UL;
calculating the reward based on the transmission rate for DL and the transmission rate for UL;
3. The wireless communication method according to claim 1, further comprising: The wireless communication method according to any one of claims 1 to 3, wherein the value function is Deep Q-Network or Double Deep Q-Network. A wireless terminal having a function of connecting to a plurality of wireless base stations,
Equipped with a CPU,
The CPU,
detecting a state of connection between the plurality of wireless base stations and the wireless terminal by distinguishing between DL communication and UL communication;
inputting said state into a value function;
a process of calculating a value of each of actions that the wireless terminal can take under the state by using the value function, and determining to which wireless base station a connection request is to be made for each of the actions by distinguishing between DL communication and UL communication;
transmitting a request to a wireless base station corresponding to the action to be taken;
obtaining a response to the request from a wireless base station;
A reward calculation process for calculating a reward reflecting both the DL evaluation and the UL evaluation based on the response;
updating a value for taking the action under the state based on the reward;
and learning parameters of the value function based on the updated value .
The state input to the value function includes the presence or absence of a DL-related association request and the presence or absence of a UL-related association request of the wireless terminal with respect to each of the plurality of wireless base stations . At least some of the plurality of wireless base stations have a plurality of wireless communication interfaces having different specifications,
the wireless terminal has the plurality of interfaces for wireless communication,
The state includes information regarding which radio base station the radio terminal is connected to, through which interface, and whether DL or UL, the radio terminal is connected to;
The wireless terminal according to claim 5 , wherein the action includes information regarding which interface of the wireless terminal is to be connected to which wireless base station in either DL or UL. The reward calculation process includes:
A process of calculating an SINR related to DL communication requested by the wireless terminal;
Calculating a transmission rate obtained in DL communication based on the SINR for DL;
A process of calculating a SINR related to UL communication requested by the wireless terminal;
calculating a transmission rate obtained in UL communication based on the SINR for UL;
calculating the reward based on the transmission rate for DL and the transmission rate for UL;
7. The wireless terminal according to claim 5 or 6, comprising: A program for a wireless terminal for realizing a function of connecting to a plurality of wireless base stations,
A CPU provided in the wireless terminal,
detecting a state of connection between the plurality of wireless base stations and the wireless terminal by distinguishing between DL communication and UL communication;
inputting said state into a value function;
a process of calculating a value of each of actions that the wireless terminal can take under the state by using the value function, and determining to which wireless base station a connection request is to be made for each of the actions by distinguishing between DL communication and UL communication;
transmitting a request to a wireless base station corresponding to the action to be taken;
obtaining a response to the request from a wireless base station;
A reward calculation process for calculating a reward reflecting both the DL evaluation and the UL evaluation based on the response;
updating a value for taking the action under the state based on the reward;
A process of learning parameters of the value function based on the updated value;
A program for executing
A program for a wireless terminal, wherein the state input to the value function includes the presence or absence of a DL-related association request and the presence or absence of a UL-related association request of the wireless terminal with respect to each of the plurality of wireless base stations .

Images