JP6351671B2 - Program, system, and method for adjusting neural network structure and parameters using neuro evolution - Google Patents

Description

  The present invention relates to a program, a system, and a method for adjusting the structure and parameters of a neural network using neuro evolution, and more particularly, to output a plurality of actions of one or more game characters in a game using neuro evolution. The present invention relates to a program, system, and method for adjusting the structure and parameters of a neural network (NN).

  A neural network is a model conceived to simulate brain neurons and synapses. Neuro Evolution is a combination of evolutionary methods and neural networks, and provides a method for searching for optimal neural networks using genetic methods.

  In such a neuro evolution, a method of evolving the structure and weighting of a neural network by performing machine learning using a genetic algorithm is known (for example, Non-Patent Document 1). Non-Patent Document 1 proposes a method called NeuroEvolution of Augmenting Topics (NEAT) that uses a genetic algorithm to learn an optimal solution related to the structure and weighting of a neural network.

“Evolving Neural Networks Through Augmentation Topics”, Kenneth O. Stanley, Risto Mikukulainen

  However, if the NEAT technology is used for learning the game play AI, a game-specific improvement is required. More specifically, when the game environment and user character parameters fluctuate due to the progress of the game, the evaluation may fluctuate greatly each time the game is attempted. However, it has been difficult to search for an optimal solution related to the structure and weighting of the neural network.

  An object of the embodiment of the present invention is to adjust the structure and parameters of a neural network using neuro evolution in order to determine the action of a game character. Other objects of the embodiments of the present invention will become apparent by referring to the entire specification.

  A method according to an embodiment of the present invention includes: extracting a game parameter as an input value to the one or more computers in response to being executed on the one or more computers; A step of extracting a plurality of output values from a plurality of neural networks (NN), a step of determining an action of one or a plurality of game characters based on the output value, and a game parameter after the action as the game parameter In the direction of increasing the evaluation value for the combination of the output values of each action and the step of calculating the evaluation value for the combination of the output values of each action corresponding to each neural network Applying a genetic algorithm to construct the plurality of neural networks (NN). And updating the parameter, the be to multiple generations repeatedly executed, configured to adjust the structure and parameters of the neural network (NN) using the neuro evolution.

  A system according to an embodiment of the present invention includes a step of extracting a game parameter as an input value to the one or more computers in response to being executed on the one or more computers, and based on the input values A step of extracting a plurality of output values from a plurality of neural networks (NN), a step of determining an action of one or a plurality of game characters based on the output value, and a game parameter after the action as the game parameter In the direction of increasing the evaluation value for the combination of the output values of each action and the step of calculating the evaluation value for the combination of the output values of each action corresponding to each neural network Applying the genetic algorithm to the plurality of neural networks (NN) Updating the structure and parameters, by a multiple generations repeatedly performed, configured to adjust the structure and parameters of the neural network (NN) using the neuro evolution.

  A program according to an embodiment of the present invention includes a step of extracting a game parameter as an input value to one or a plurality of computers in response to being executed on the one or a plurality of computers, and based on the input value A step of extracting a plurality of output values from a plurality of neural networks (NN), a step of determining an action of one or a plurality of game characters based on the output value, and a game parameter after the action as the game parameter In the direction of increasing the evaluation value for the combination of the output values of each action and the step of calculating the evaluation value for the combination of the output values of each action corresponding to each neural network And applying a genetic algorithm to the plurality of neural networks (NN Updating the structure and parameters of the be to multiple generations repeatedly executed, configured to adjust the structure and parameters of the neural network (NN) using the neuro evolution.

  According to various embodiments of the present invention, even in a game in which a battle state changes every moment by the action of one or more game characters during one battle, a plurality of actions for outputting the action of one or more game characters in the game are output. By updating the structure and parameters of the neural network (NN) using the neuro evolution, it becomes possible to contribute to more appropriate and natural determination of the action of the game character.

The lineblock diagram showing roughly the composition of system 1 concerning one embodiment of the present invention. The block diagram which shows roughly the function which the system 1 in one Embodiment has. The figure which shows an example of the flow which adjusts the structure and parameter of a neural network (NN) in one Embodiment. The figure which shows what plotted the example of the relationship between the number of generations in the case of a NEAT method, and an evaluation value. The figure which shows what plotted the example of the relationship between the number of generations at the time of the update of the neural network (NN) in one Embodiment, and an evaluation value. The figure which shows what plotted the example of the relationship between the number of generations at the time of the update of the neural network (NN) in one Embodiment, and an evaluation value.

  FIG. 1 is a configuration diagram schematically showing a configuration of a system 1 according to an embodiment of the present invention. As illustrated, the system 1 according to an embodiment includes a server 10 and a plurality of terminal devices 30 connected to the server 10 via a communication network 20 such as the Internet. To provide electronic commerce services. In addition, the system 1 according to the embodiment includes a service for providing various digital contents other than games such as games using characters, electronic books, moving image contents, and music contents, text chat (mini mail), circles, and avatars. Various Internet services such as a communication platform (SNS platform) service that realizes communication functions between various users such as a diary, a message board, and greetings can be provided to the user of the terminal device 30.

  The server 10 in one embodiment is configured as a general computer, and as illustrated, a CPU (computer processor) 11, a main memory 12, a user I / F 13, a communication I / F 14, and storage (storage). Device) 15, and these components are electrically connected to each other via a bus 17. The CPU 11 loads an operating system and various other programs from the storage 15 into the main memory 12 and executes instructions included in the loaded programs. The main memory 12 is used for storing a program executed by the CPU 11, and is configured by a DRAM or the like, for example. In addition, the server 10 in one embodiment may be configured using a plurality of computers each having a hardware configuration as described above. The CPU (computer processor) 11 described above is merely an example, and it goes without saying that a GPU (graphics processing unit) may be used instead. How to select the CPU and / or GPU can be appropriately determined in consideration of desired cost or efficiency. Hereinafter, the CPU 11 will be described as an example.

  The user I / F 13 includes, for example, an information input device such as a keyboard and a mouse that accepts an operator input, and an information output device such as a liquid crystal display that outputs a calculation result of the CPU 11. The communication I / F 14 is implemented as hardware, firmware, communication software such as a TCP / IP driver or a PPP driver, or a combination thereof, and is configured to be able to communicate with the terminal device 30 via the communication network 20.

  The storage 15 is composed of, for example, a magnetic disk drive, and stores various programs such as a control program for providing various services. The storage 15 can also store various data for providing various services. Various data that can be stored in the storage 15 may be stored in a database server or the like that is physically separate from the server 10 that is communicably connected to the server 10.

  In one embodiment, the server 10 also functions as a web server that manages a website composed of a plurality of hierarchical web pages, and can provide various services to the user of the terminal device 30 via the website. . The storage 15 can also store HTML data corresponding to this web page. HTML data is associated with various image data, and various programs described in a script language such as JavaScript (registered trademark) can be embedded.

  In one embodiment, the server 10 can provide various services via an application (program) executed on the terminal device 30 in an execution environment other than the web browser. Such applications can also be stored in the storage 15. This application is created using a programming language such as Objective-C or Java (registered trademark). The application stored in the storage 15 is distributed to the terminal device 30 in response to the distribution request. Note that the terminal device 30 can also download such an application from a server other than the server 10 (a server that provides an application market) or the like.

  As described above, the server 10 can manage websites for providing various services, and distribute web pages (HTML data) constituting the websites in response to requests from the terminal device 30. In addition, as described above, the server 10 is an application that is executed in the terminal device 30 in place of or in addition to the provision of various services using such a web page (web browser). Various services can be provided based on communication. Regardless of which aspect of the service is provided, the server 10 can transmit and receive various data (including data necessary for screen display) necessary for providing various services to and from the terminal device 30. Moreover, the server 10 can store various data for each identification information (for example, user ID) for identifying each user, and can manage the provision status of various services for each user. Although detailed description is omitted, the server 10 may have a function of performing user authentication processing, billing processing, and the like.

  The terminal device 30 according to the embodiment is an arbitrary information processing device that displays a web page of a website provided by the server 10 on a web browser and implements an execution environment for executing an application. , A wearable device, a personal computer, a game-dedicated terminal, and the like, but are not limited thereto.

  The terminal device 30 is configured as a general computer, and as shown in FIG. 1, a CPU (computer processor) 31, a main memory 32, a user I / F 33, a communication I / F 34, and a storage (storage device) 35. These components are electrically connected to each other via a bus 37.

  The CPU 31 loads an operating system and various other programs from the storage 35 into the main memory 32 and executes instructions included in the loaded programs. The main memory 32 is used for storing a program executed by the CPU 31, and is configured by, for example, a DRAM or the like.

  The user I / F 33 includes, for example, an information input device such as a touch panel that accepts user input, a keyboard, a button, and a mouse, and an information display device such as a liquid crystal display that outputs a calculation result of the CPU 31. The communication I / F 34 is implemented as hardware, firmware, communication software such as a TCP / IP driver or a PPP driver, or a combination thereof, and is configured to be able to communicate with the server 10 via the communication network 20.

  The storage 35 is composed of, for example, a magnetic disk drive, a flash memory, or the like, and stores various programs such as an operating system. The storage 35 can store various applications received from the server 10.

  The terminal device 30 includes, for example, a web browser for interpreting an HTML file (HTML data) and displaying the screen, and interprets and receives the HTML data acquired from the server 10 by the function of the web browser. A web page corresponding to the HTML data thus displayed can be displayed. The web browser of the terminal device 30 can incorporate plug-in software that can execute various types of files associated with HTML data.

  When the user of the terminal device 30 uses a service provided by the server 10, for example, HTML data, an animation instructed by an application, an operation icon, or the like is displayed on the terminal device 30. The user can input various instructions using the touch panel of the terminal device 30 or the like. The instruction input from the user is transmitted to the server 10 via the function of the application execution environment such as the web browser of the terminal device 30 and NgCore (trademark).

  Next, the function which the system 1 in one embodiment comprised in this way has is demonstrated. As described above, the system 1 according to an embodiment can provide various Internet services to the user, and in particular, can provide a game distribution service. Hereinafter, the function of the system 1 according to an embodiment will be described by taking a function of providing a game distribution service as an example.

  FIG. 2 is a block diagram schematically showing functions of the system 1 (the server 10 and the terminal device 30). First, the function which the server 10 in one Embodiment has is demonstrated. As illustrated, the server 10 includes an information storage unit 41 that stores various information, and a character action control unit 42 that determines the action of the game character in one embodiment. These functions are realized by the cooperative operation of hardware such as the CPU 11 and the main memory 12 and various programs and tables stored in the storage 15, for example, instructions included in the loaded program This is realized by the CPU 11 executing. 2 may be realized by the terminal device 30 or may be realized by the cooperation of the server 10 and the terminal device 30.

  The information storage unit 41 in one embodiment is realized by the storage 15 or the like, and as shown in FIG. 2, various statuses of enemies and teammate characters (collectively referred to as game characters) in the game, actionable characters, A game parameter management table 41a for managing game parameters such as an action that can be performed, a flag of a character that has recently acted, a technique of a character that has recently acted, and an evaluation value when each action is selected based on the game parameter A behavior evaluation value management table 41b.

  Next, the function of the character action control part 42 which determines the action of the game character in one Embodiment is demonstrated. The character action control unit 42 has various statuses of enemies and teammate characters stored in the game parameter management table 41a of the information storage unit 41, an actionable character, an actionable technique, a flag of a character that has recently acted, and an action of the most recent character. The action of each game character is selected and determined based on game parameters such as the character's skill. More specifically, the character action control unit 42 extracts a game parameter of the game parameter management table 41a, inputs this as an input value to one or more layers of neural network, and acts as an output of the character that can act as an output. The output value corresponding to this is extracted, and the action with the highest output value is determined from these. In this way, for example, the character action control unit 42 can extract a combination of actions of the teammate characters until one battle is completed. In this example, each action of the teammate character until the end of one battle is extracted as an output value, but it may be an enemy character, a teammate character, or a part thereof. Moreover, although it was made until 1 battle was complete | finished, if it is a state which can calculate other evaluation values, it will not be restricted to this.

  Furthermore, when the character action control unit 42 extracts the combination of each action of the teammate character until one battle ends, the character action control unit 42 evaluates the combination of the output values by the method described in detail below, and determines the evaluation value. calculate. Since an evaluation value for a combination of output values is extracted for each neural network, if there are a plurality of neural networks, evaluation values corresponding to the number of neural networks are extracted. Normally, about 100 individuals, that is, about 100 neural networks are prepared, and as a result, about 100 evaluation values are extracted, but the number is arbitrary and can be determined as appropriate. The individual may be configured by a convolution network (CNN). The character action control unit 42 applies a genetic algorithm based on the evaluation value calculated for the combination of the extracted output values, and performs a habit so that a more optimal individual survives, that is, a more optimal neural network. Update to By repeating this for a plurality of generations, for example, about 100 generations, the neural network evolves with a genetic algorithm. How many generations are repeated can be appropriately changed according to various conditions. Here, the character action control unit 42 optimizes the structure and parameters (also referred to as weighting) of the neural network by using a NEAT method classified as one of the neuro evolutions. In this way, the character action control unit 42 basically survives the structure and parameters of the neural network that can extract a combination of output values with higher evaluation values, and repeats these processes over a plurality of generations. Thus, an optimal neural network is searched. Here, existing genetic algorithms, neuro evolution, and basic methods of NEAT are not described in detail, but those skilled in the art can appropriately apply these methods to the system according to the embodiment.

  FIG. 3 shows this flow. First, a game parameter (input value) is extracted (step 110), and this is input to a plurality of initial neural networks to extract an output value of each action (step 120). A specific action of the game character is selected / determined (step 130), the game parameter is updated to be the game parameter after the action (step 140), and these steps are repeated until one game battle is completed. An evaluation value of a combination of output values of each action of the character is calculated (step 150), and a genetic algorithm is applied based on the evaluation value to update the structure and parameters of the neural network (individual) (step 160). The structure and parameters of the neural network (individual) are evolved by repeating these for multiple generations. Hereinafter, a method of learning the structure and parameters of a neural network using neuro evolution in order to determine a more optimal action of the game character in the invention according to an embodiment will be described more specifically.

  First, the neural network of an individual subject to reinforcement learning may be an arbitrary network having a single-layer or multiple-layer neural network (including CNN) structure. Here, an individual can be defined as the structure of a neural network and the weight between nodes. Hereinafter, it is expressed as the structure and parameters of a neural network (NN). In the initial stage, the individuals are randomly generated based on the initial network in which the number of input and output nodes is set.

  Usually, about 100 individuals are prepared, and a trial of the game is performed for these individuals. The number of individuals to be prepared is arbitrary, and can be appropriately determined in consideration of various factors such as time, system load, and calculation efficiency. In reinforcement learning, these multiple individuals are evaluated at the same time, but in neuro evolution, it is important to secure a search range within a certain range so as not to fall into a local solution. Individuals are divided into a plurality of races (hereinafter referred to as groups), so that the genetic algorithm is used only within the same race, and this is not performed between races (called “Nitching”).

  As a method of grouping, based on the NEAT method, one individual and all remaining individuals are compared, and the degree of similarity (calculated by the number of nodes and the number of edges in the neural network, also referred to as distance) is below a threshold value. The individuals are assigned to the same group (race), and the same procedure is repeated for individuals who have not entered the group (race), thereby dividing the race. As a result, all individuals belong to any group (race). For example, when dividing into three races, it is necessary to further divide individuals whose similarity is above the above threshold into two, so a threshold is provided separately, and individuals whose similarity is below this threshold Is the second group (race), and individuals whose similarity is higher than the threshold is the third group (race). The lower this threshold is, the more groups (racial groups) can be divided, and it is possible to avoid falling into a local solution. However, if this threshold is made too low, a large number of groups (racial groups) with 1 population will be generated. It becomes.

  Hereinafter, as an embodiment, a command battle game in which a plurality of teammate characters and enemy characters battle each other will be described as an example. In the game of the present embodiment, the teammate character and the enemy character have action gauges, and the action gauges increase with the passage of time. Then, a character whose action gauge is full is in an actionable state. The game progresses when the player selects a command for the ally character in the actionable state. Further, for an enemy character that is in an actionable state, a command is determined according to a predetermined condition (for example, random) set in the game program. In a trial of a game using an individual, the game state at a certain time is parameterized and input as an input value to the individual, and a command is selected according to an output value from the individual to advance the game. As described above, various statuses of enemies and teammate characters (collectively referred to as game characters) in the game, actionable game characters, actionable techniques, the flag of the most recently acted game character, and the game that recently acted Character skills are defined as game parameters.

  Next, an example of this input value will be described more specifically. As an input value to the neural network (individual), for example, (1) a command selectable flag for all game characters at a certain time (a time when any game character can act), and (2) all game characters Enter the status change rate. The former (1) command selectable flag for all game characters means an available flag (0 or 1) for each command of all game characters (ally characters and enemy characters).

In the game of the present embodiment, there are teammate character 1, teammate character 2, teammate character 3, teammate character 4, and enemy character, and each command has five commands (attack, defense, skill 1, skill 2, and special move). Is set. When only the ally character 1 can act at a certain time and the ally character 1 can select attack, defense, skill 1, and skill 2,
Allied character 1_attack: 1
Ally character 1_defense: 1
Ally character 1_skill 1: 1
Ally character 1_skill 2: 1
Allied Character 1_ Special Move: 0
Ally character 2_attack: 0
Ally character 2_defense: 0
Ally character 2_skill 1: 0
Ally character 2_skill 2: 0
Allied Character 2_ Special Moves: 0
... (omitted) ...
Enemy Character_ Special Moves: 0
25 (because each of the five game characters has five commands).

At that time, by adjusting the input value in consideration of the command selected in the past, the same command can be adjusted so as not to be selected continuously. For example, the command selected in the past is stored for each game character, and the command selected in the previous action is multiplied by 0.1. Further, the command selected at the time of the second previous action is multiplied by 0.2, and the command selected at the time of the third previous action is multiplied by 0.3. If the command selected by the ally character 1 in the past three is Skill 1, Skill 2, and Attack from the oldest, the input value is the ally character 1_attack: 0.1
Ally character 1_defense: 1
Ally character 1_skill 1: 0.3
Ally character 1_skill 2: 0.2
Allied Character 1_ Special Move: 0
And

Next, the latter (2) the status change rate of all game characters is the initial value of various parameters (HP, attack power, defense power, magic attack power, magic defense power) of all game characters (friend characters and enemy characters). The rate of change from Parameters other than HP have an initial value of 0 as a device for making it easier for the AI to recognize the buff / debuff state. The HP is set so that 50% of the initial value is 0 (range is -0.5 to 0.5). For example, a game character whose HP is full (initial value) and whose attack power is double is
HP: 0.5
Attack power: 1
Defense: 0
Magical attack power: 0
Magic Defense: 0
It becomes. Thereby, the learning efficiency of the neural network can be increased.

  It is also possible to further input the above-mentioned game parameter “flag of the game character that has acted most recently”. In a game battle in the game FRK, there is a “buff / debuff” effect that lasts for a certain period of time, and it is desirable to continue buffing at regular intervals. Can be avoided. Here, “buff” refers to an action that improves the ability value of the player character, such as a spell that doubles the attack power of the game character. “Debuff” means the opposite. By further inputting “the flag of the game character that acted most recently” as an input value, “buff”, “another action”, “re-buffing (at the timing when the first buff goes off) are output as output values described later. ”Was confirmed. It should be noted that normalization processing can be performed without inputting in-game values as input for status values. For example, the initial value of the buffed or debuffed status is 0, and the rate of change is input. This is to make it easier for the AI to recognize the buff / debuff state. As another example, HP sets a value of 50% to 0 and inputs a difference. This is because there is an enemy character that changes the action when the HP falls below a certain ratio, or when the player psychology falls below 50%, the recovery action is often taken.

Next, examples of output values will be described more specifically. Numerical values are extracted from the neural network (individual) as output values for all commands of all ally characters. If four teammate characters have five commands each, there are 20 nodes. In each game character, a command corresponding to the node having the largest numerical value is selected. Thereby, the command of the character that can act at a certain time is determined.
For example, the output value of the ally character 1 is
Allied character 1_attack: 0.8
Ally character 1_defense: 0.5
Ally character 1_skill 1: 0.4
Ally character 1_skill 2: 0.1
Allied Character 1_ Special Move: 0
In this case, the “attack” having the largest output value is selected. When the output values of all commands are 0, it means that the game character cannot act at that time.

  In the game battle, each action of the game character is determined by repeating this input and output. Usually, a plurality of actions of a plurality of game characters are combined and a game battle proceeds, and continues until the battle ends. The end of the battle is usually intended, for example, when the HP of the enemy game character is 0, or the HP of all the friend game characters is 0, but in other cases where the game battle can be evaluated. It doesn't matter.

Normally, a plurality of game characters respectively select a plurality of actions before the end of the above-described game battle. The evaluation of the neural network (individual) is performed for the combination of output values of each action. Usually, it will be performed about the combination of the output value of each action until a battle is complete | finished. The evaluation is calculated based on the following four parameters. In addition, although the following parameter uses the average which performed the game battle 3 times using the same neural network (individual), for example, other methods may be used.
(1) Number of actions of allied characters until the end of the game battle (ability_count)
(2) Remaining HP (boss_hp_sum) of enemy game characters at the end of the game battle
(3) Damage taken by allied game characters before the end of the game battle (damage_ratio)
(4) Number of times the teammate game character died before the end of the game battle (dead_count)
Based on these parameters, the following setting can be made as a method for calculating the evaluation value. This setting can be changed as appropriate. In the following example, the code “sum” corresponds to the evaluation value, and the maximum evaluation value is 0.
return sum ((-1.0 * ability_count / max_ability_count, -1.0 * boss_hp_sum / boss_max_hp_sum * 2, -1.0 * damage_ratio * 2, (-1.0 * dead_count / max_dead_count) * 4))
The max value can be set as appropriate, and each parameter takes its ratio. Further, since the number of deaths is the most important parameter, it is weighted four times.

  In this way, the structure and parameter evaluation values of each neural network are calculated for each generation, and the neural network (individual) is updated in the direction of evolution according to the NEAT genetic algorithm. The number of generations may be 100 generations or others. The method of evolving an individual when moving to the next generation is not described in detail, but by multiplying individuals above a plurality of neural networks (individuals), for example, individuals whose evaluation values are within 10 from the top A new individual to be the next generation may be generated. The neural network (individual) update method can be set as appropriate, and is not intended to be limited in any way. Moreover, when a plurality of neural networks (individuals) are divided into a plurality of groups (race), the average value of the evaluation values of the neural networks (individuals) belonging to a plurality of groups (race) in each generation is calculated, This is the evaluation value for each group of each generation. In some cases, the maximum evaluation value of a neural network (individual) belonging to a plurality of groups (race) in each generation may be used.

  Next, the relationship between groups when a plurality of neural networks (individuals) are divided into a plurality of groups (racial groups) will be described. When moving to the next generation, the ratio of each group (race) is changed as follows. First, an overall average value is calculated from the evaluation values of all groups (race) in the generation. Then, the evaluation value of each group (race) is compared with the overall average value, the group (race) with an evaluation value higher than the overall average value increases its proportion, and the group with an evaluation value lower than the overall average value is Reduce the percentage. At that time, the number of individuals of various groups is adjusted so that the total number of individuals is constant. The increase / decrease of the ratio is, for example, usually about ± 10%, but is not limited thereto and can be set as appropriate.

  There is an exceptional handling method for the relationship between groups when a plurality of neural networks (individuals) are divided into a plurality of groups (racial groups). In NEAT, this is called staging. This is to remove the group (race) when the average evaluation value in each group (race) is not improved continuously for X generations. The basic idea of staging is the same in one embodiment, but if this is incorporated in a game where the game parameters change every moment as in the game of this embodiment, even if about 500 generations are repeated. It was confirmed that there was a problem that the average evaluation value of the group (race) was not improved at all. FIG. 4 shows this, in which the horizontal axis indicates the generation, the vertical axis indicates the evaluation value (fitness), and the average evaluation value and the maximum evaluation value of the group (race) are plotted. As shown in the figure, the maximum evaluation value of the group (race) randomly increases and decreases, and the average evaluation value of the group (race) does not show any improvement even after 400 to 500 generations. This is because the game of this embodiment is a highly random game, and the evaluation value of the group (race) tends to vibrate. Therefore, the condition of the stagnation is not satisfied, and the group (race) trap is This is thought to be due to the fact that virtually nothing is done.

  In view of such a problem, in one embodiment, a modification was made to the conditions for staging in NEAT. Specifically, the average evaluation value in the group (race) is not assumed to be improved continuously for X generations. For example, the largest evaluation value among the plurality of neural networks (NN) in each group of the plurality of groups. The condition is that the evaluation value is not updated so as to be high during a predetermined generation (such as 15 generations). Moreover, it is good also as conditions that the average value of the evaluation value of several neural network (NN) in each group of several groups is not updated so that it may become high during a predetermined generation. As shown in FIG. 5, it was confirmed that resetting can be performed by performing such a condition change, as shown in FIG. 5, when the race is an excellent race and the evaluation value does not show a certain improvement. . Similar to FIG. 4, FIG. 5 shows the generation on the horizontal axis, the evaluation value (fitness) on the vertical axis, and plots the average evaluation value and maximum evaluation value of the group (race). As shown in the figure, after both the maximum evaluation value and the average evaluation value of the group (race) rise, the excellent group (race) becomes dominant, but the evaluation value cannot achieve a certain degree of improvement. A group (race) is formed and it is shown how these are repeated. Even in the case of a highly random game such as the game of this embodiment in which the evaluation value of the group (race) is highly oscillating, the group (race) trap is generated by correcting the conditions of the stagnation It was confirmed that it can be made.

  However, the best group (race) in about 120 generations was reset, and after that, one group (race) occupied the whole, and after this continued for several tens of generations, it was replaced with another group (race). Another issue has been identified that does not show an overall improvement. Therefore, in one embodiment, not only the conditions for the staging but also the effects thereof are corrected. Specifically, when the maximum evaluation value among the evaluation values of the plurality of neural networks (NN) in each group of the plurality of groups is not updated so as to be higher during a predetermined generation (for example, 15 generations), the group The number of neural networks (NN) of the network is reduced so as to decrease at a rate larger than 10% (for example, about 50%). FIG. 6 shows the results when the number of neural networks (NN) in the group that satisfies the condition is reduced by about 50%. Similar to FIGS. 4 and 5, FIG. 6 also shows the generation on the horizontal axis and the evaluation value (fitness) on the vertical axis, and plots the average evaluation value and maximum evaluation value of the group (race). In this way, the average of the evaluation values of the group (race) is steadily improved and the maximum evaluation value is improved without deteriorating while repeating the stagnation by correcting the stagnation condition and its effect. confirmed. This is because a group (race) whose evaluation value is not updated is gradually deceived, another group (race) is newly born, and diversity of the group (race) is ensured, while the neural network (individual) as a whole This is considered to indicate that has evolved.

  In this way, even in games where game parameters change every moment, such as game battles, the neural network structure and parameters are reinforced by using a genetic algorithm, and the neural network structure and parameters are updated. By doing so, it becomes possible to determine a more appropriate action of the enemy character or ally character, to enjoy more naturally without getting tired of the progress of the game, and to enhance the appeal to the entire game.

  In the above, the function which the server 10 has was demonstrated. Next, functions of the terminal device 30 in the embodiment will be described. As shown in FIG. 2, the terminal device 30 includes an information storage unit 51 that stores various information, and a terminal-side control unit 52 that executes control for displaying image information on the terminal side in one embodiment. Have. These functions are realized by the cooperation of hardware such as the CPU 31 and the main memory 32, and various programs and tables stored in the storage 35. For example, instructions included in the loaded program This is realized by the CPU 31 executing. In addition, part or all of the functions of the terminal device 30 illustrated in FIG. 2 can be realized by the cooperation of the server 10 and the terminal device 30, or can be realized by the server 10.

  The information storage unit 51 in the embodiment is realized by the main memory 32 or the storage 35. The terminal-side control unit 52 in one embodiment controls the execution of various terminal-side processes such as selection of actions by the user character and display of received game screen information. For example, when the user selects an action with a user character, the terminal-side control unit 52 transmits the action to the server 10 or when the game parameter changes as a result of the action of an ally character or enemy character, The changed game parameters can be received from the server 10 and displayed.

  The processes and procedures described in this specification are implemented by software, hardware, or any combination thereof other than those explicitly described in the embodiments. More specifically, the processes and procedures described in this specification are performed by mounting logic corresponding to the processes on a medium such as an integrated circuit, a volatile memory, a nonvolatile memory, a magnetic disk, or an optical storage. Realized. Further, the processes and procedures described in this specification can be implemented as a computer program and executed by various computers.

  Even if the processes and procedures described herein are described as being performed by a single device, software, component, or module, such processes or procedures may be performed by multiple devices, multiple software, multiple Component and / or multiple modules. In addition, even though the data, tables, or databases described herein are described as being stored in a single memory, such data, tables, or databases are provided on a single device. Alternatively, the data can be distributed and stored in a plurality of memories or a plurality of memories arranged in a plurality of devices. Further, the software and hardware elements described herein may be implemented by integrating them into fewer components or by decomposing them into more components.

  In the present specification, when the constituent elements of the invention are described as one or a plurality, or when they are described without being limited to one or a plurality of cases, they should be understood separately in context. The component may be either singular or plural.

DESCRIPTION OF SYMBOLS 10 Server 20 Communication network 30 Terminal device 41 Information storage part 42 Character action control part 51 Information storage part 52 Terminal side control part

Claims (11)

A method of adjusting the structure and parameters of a plurality of neural networks (NN) that output the action of one or more game characters in a game using neuro evolution,
In response to being executed on one or more computers,
Extracting game parameters as input values;
Extracting a plurality of output values from a plurality of neural networks (NN) based on the input values;
Determining an action of one or more game characters based on the output value;
Updating the game parameter to the game parameter after the action;
Repeating these steps, calculating an evaluation value for a combination of output values of each action corresponding to each neural network;
Updating a structure and parameters of the plurality of neural networks (NN) by applying a genetic algorithm in a direction in which an evaluation value with respect to a combination of output values of the actions increases.
The all SANYO to multiple generations repeatedly executed,
Dividing the neural network (NN) into a plurality of groups using a predetermined threshold, and applying the genetic algorithm to update the structure and parameters of the neural network (NN) only within the same group;
Neuroevolution characterized by dividing a plurality of neural networks (NN) into a plurality of groups based on a structure and parameter similarity between the plurality of neural networks (NN), with a predetermined threshold relating to the similarity as a boundary. To adjust the structure and parameters of a neural network (NN) using An average value of the evaluation values of the neural network (NN) of each group in the same generation is compared between the plurality of groups, and the average value is higher than the average value of the evaluation values of the entire plurality of groups Reducing the proportion of groups having an average value of evaluation values lower than the average value of the evaluation values of the entire plurality of groups, and reducing the ratio of the plurality of groups in the next generation. the method according to claim 1, characterized in that varying the ratio of number occupied in the neural network (NN) in each group of the group. An average value of the evaluation values of the neural network (NN) of each group in the same generation is compared between the plurality of groups, and the average value is higher than the average value of the evaluation values of the entire plurality of groups Increasing the ratio of groups having an average value of about 10%, reducing the ratio of groups having an average value of evaluation values lower than the average value of the evaluation values of the plurality of groups as a whole by about 10%, The method according to claim 2 , wherein a ratio occupied by the number of neural networks (NN) of each group of the plurality of groups in the next generation is changed. When the maximum evaluation value among the evaluation values of the plurality of neural networks (NN) in each group of the plurality of groups is not updated so as to be high during a predetermined generation, the number of neural networks (NN) of the group is determined. 4. A method according to claim 2 or 3 , characterized in that the reduction is carried out at a rate greater than 10%. When the maximum evaluation value among the evaluation values of the plurality of neural networks (NN) in each group of the plurality of groups is not updated so as to be high during a predetermined generation, the number of neural networks (NN) of the group is determined. 2. The method according to claim 2 or 3 , characterized by halving.
The method described in 1. Wherein a plurality of neural networks (NN), Method according to one of claims 1 to 5, which comprises using about 100 of the neural network (NN). Any one method according to claims 1 to 6, characterized in that repeated about 100 generations the step of the update. The evaluation value is determined based on at least one of a game character action count, a game character remaining HP, a total damage received by the game character, a game character death count, or a combination thereof. The method according to any one of claims 1 to 7 . The evaluation value is an evaluation value for a combination of output values extracted when each step is repeated until one battle of the game is completed.
Any one process as claimed in to 8. A system for adjusting the structure and parameters of a plurality of neural networks (NN) that output the action of one or more game characters in a game using neuro evolution,
In response to being executed on one or more computers,
Extracting game parameters as input values;
Extracting a plurality of output values from a plurality of neural networks (NN) based on the input values;
Determining an action of one or more game characters based on the output value;
Updating the game parameter to the game parameter after the action;
Repeating these steps, calculating an evaluation value for a combination of output values of each action corresponding to each neural network;
Updating a structure and parameters of the plurality of neural networks (NN) by applying a genetic algorithm in a direction in which an evaluation value with respect to a combination of output values of the actions increases.
Is repeatedly executed for multiple generations ,
Dividing the neural network (NN) into a plurality of groups using a predetermined threshold, and applying the genetic algorithm to update the structure and parameters of the neural network (NN) only within the same group;
Neuroevolution characterized by dividing a plurality of neural networks (NN) into a plurality of groups based on a structure and parameter similarity between the plurality of neural networks (NN), with a predetermined threshold relating to the similarity as a boundary. To adjust the structure and parameters of a neural network (NN) using A program for adjusting the structure and parameters of a plurality of neural networks (NN) that output the actions of one or more game characters in a game using neuro evolution,
In response to being executed on one or more computers,
Extracting game parameters as input values;
Extracting a plurality of output values from a plurality of neural networks (NN) based on the input values;
Determining an action of one or more game characters based on the output value;
Updating the game parameter to the game parameter after the action;
Repeating these steps, calculating an evaluation value for a combination of output values of each action corresponding to each neural network;
Updating a structure and parameters of the plurality of neural networks (NN) by applying a genetic algorithm in a direction in which an evaluation value with respect to a combination of output values of the actions increases.
Is repeatedly executed for multiple generations ,
Dividing the neural network (NN) into a plurality of groups using a predetermined threshold, and applying the genetic algorithm to update the structure and parameters of the neural network (NN) only within the same group;
Neuroevolution characterized by dividing a plurality of neural networks (NN) into a plurality of groups based on a structure and parameter similarity between the plurality of neural networks (NN), with a predetermined threshold relating to the similarity as a boundary. A program for adjusting the structure and parameters of a neural network (NN) using

Images