JP7572815B2 - Test support device and program - Google Patents

Description

The present invention relates to a test support device and a program.

Traditionally, product testing of games, user interfaces, etc. has been performed in a virtual environment constructed on a computer, and the parameters obtained from the testing have been applied to the configuration of the real environment to improve the efficiency of product design. In recent years, with the improvement of computer computing power and advances in artificial intelligence (AI) technology, the environment in which product testing in virtual environments can be utilized is becoming more and more complete.

For example, Patent Document 1 proposes that a computer system using a statistically processed database of sensitivities of standard people be used to design equipment that matches the sensitivities and body shape of the individual user.

Japanese Patent Application Publication No. 9-218885

For example, when testing an interactive system in a virtual environment, it is possible to have an AI program act as an agent to carry out the testing. However, when testing takes into account emotional factors such as ease of learning and entertainment value, it is still difficult for a computer to make an appropriate evaluation. For this reason, human work still accounts for a large portion of this type of testing, and there is a demand for improvements in this area.

The present invention was made against the background of the above problems, and provides a test support device and program that can reduce the human workload during testing when optimizing a system taking into account emotional factors.

According to one aspect of the present invention, there is provided a test support device (10) including: a simulation unit (21) that simulates a system to be tested in a virtual environment and is capable of setting parameters related to the system; a learning unit (22) that tests the system simulated by the simulation unit (21) based on the set parameters and performs machine learning using the results of the test; a first calculation unit (26) that calculates a first evaluation value according to the learning speed of the learning unit (22); a second calculation unit (27) that calculates a second evaluation value according to the diversity of actions selected by the learning unit (22) during the test; and an output processing unit (28) that displays the first evaluation value and the second evaluation value corresponding to the set parameters on a display device (19).

According to another aspect of the present invention, a program is provided that causes a computer (11) to execute the steps of: having a learning unit (22) test a system simulated in a virtual environment based on set parameters, and using the test results to perform machine learning in the learning unit (22) (S4); determining a first evaluation value according to the learning speed of the learning unit (22); determining a second evaluation value according to the diversity of actions selected by the learning unit (22) during the test (S6); and displaying on a display device (19) the first evaluation value and the second evaluation value corresponding to the set parameters (S10).

The present invention makes it possible to reduce the human workload during testing when optimizing a system by taking into account emotional factors.

FIG. 2 is a block diagram showing an example of a hardware configuration of the test support device. 2 is a block diagram showing an example of a software configuration of the test support device; 13 is a flowchart showing an example of the operation of the test support device. FIG. 4 is a diagram illustrating an example of a learning curve of a learning section. FIG. 13 is a diagram illustrating an overview of processing by a second calculation unit.

Below, an embodiment of a test support device and a program of the present invention will be described with reference to the drawings. The configuration described below is an example (representative example) of the present invention, and is not limited to this.

First Embodiment
In the first embodiment, a test play of a card game will be described as an example of parameter adjustment in an interactive system. Specifically, in the first embodiment, an AI program that performs machine learning in a virtual environment is made to test play the card game, and a parameter P of the card game is adjusted based on the results of the test play of the AI program. This makes it possible to create rules for a card game with good game balance.

In this card game, multiple cards that are different from each other are used. Each card is assigned a type of card that corresponds to a suit, like playing cards, and a number that determines the order of the card within that type.

(Hardware configuration of the test support device)
FIG. 1 is a block diagram showing an example of a hardware configuration of a test support device according to the first embodiment.

The test support device 10 is a computer that includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage device 14, an input I/F 15, and a display I/F 16. Each element of the test support device 10 is connected to each other via a bus 17.

The CPU 11 is a processor that performs various types of arithmetic processing according to programs stored in the ROM 12 or the RAM 13. The ROM 12 is a non-volatile storage area in which programs such as the BIOS are stored. The RAM 13 is a volatile storage area that is used as a temporary storage area when the CPU 11 performs various types of arithmetic processing.

The storage device 14 is a non-volatile large-capacity storage medium, such as a hard disk drive (HDD) or a solid-state drive (SSD). The storage device 14 stores an operating system (OS), programs that realize the various functions of the test support device, and data used by the programs.

The input I/F 15 is an interface that is connected to an external input device 18 and accepts input from the input device 18. Examples of the input device 18 include a keyboard and a pointing device.

The display I/F 16 is an interface that is connected to an external display device 19 and is responsible for output to the display device 19. An example of the display device 19 is a monitor device such as a liquid crystal display.

After booting up the test support device 10, the CPU 11 executes the BIOS and loads the OS from the storage device 14 into the RAM 13 in an executable manner. In accordance with the operation of the OS, the CPU 11 loads software modules of the test support device 10 from the storage device 14 into the RAM 13 in an executable manner as required. The loaded software modules then operate when the CPU 11 executes a program.

(Software configuration of the test support device)
FIG. 2 is a block diagram showing an example of the software configuration of the test support device in the first embodiment.

The test support device 10 includes, as software modules, a simulation unit 21, a learning unit 22, an input processing unit 23, a learning recording unit 24, a history recording unit 25, a first calculation unit 26, a second calculation unit 27, and an output processing unit 28.

The simulation unit 21 is a simulator that simulates an interactive system to be tested in a virtual environment. The simulation unit 21 returns an output of the system simulated in the virtual environment in response to an input from the learning unit 22. For example, in the first embodiment, the simulation unit 21 is configured with a game program that executes a card game.
The simulation unit 21 can change the set value of the parameter P designated for the system under test.

The learning unit 22 is a machine learning software module that functions as an AI player of the system to be tested. In the first embodiment, the learning unit 22, as a reinforcement learning agent, selects an action on the system to be tested according to an action value function. Here, when the agent acts, the state of the environment changes and a reward is obtained. Reinforcement learning of the learning unit 22 is performed by learning a policy that maximizes the sum of the rewards. Note that the algorithm applied in the reinforcement learning may be, for example, any of the Monte Carlo method, SARSA, and Q-learning.
The learning unit 22 also has a function of acquiring, from the simulation unit 21, the setting values of the parameters applied by the simulation unit 21.

In the example of FIG. 2, for simplicity, a configuration having one learning unit 22 is shown. However, the test support device 10 may have two or more independent learning units 22.

The input processing unit 23 receives the specification of the parameters to be adjusted from the user via the input device 18. In addition, when the user is one of the players in an interactive system, the input processing unit 23 outputs the user's operation information from the input device 18 to the simulation unit 21.

The learning record unit 24 records, in the storage device 14, information on a learning curve that indicates the progress of reinforcement learning by the learning unit 22. The learning curve indicates the correspondence between the number of tests and the test results. For example, the learning record unit 24 in the first embodiment records, as the learning curve information, information indicating the number of test plays of the game and the progress of the winning percentage of the learning unit 22.

The history recording unit 25 records information about the behavior history during testing by the learning unit 22 in the storage device 14. The history recording unit 25 records the history of the actions selected by the learning unit 22 in each game play in chronological order, and accumulates the history information in the storage device 14 each time a test play is performed.

The first calculation unit 26 calculates the first evaluation value V1 according to the speed of reinforcement learning of the learning unit 22. For example, in the first embodiment, the first calculation unit 26 uses the learning curve of the learning record unit 24 to calculate the first evaluation value V1 from the speed of reinforcement learning as an index for evaluating the difficulty level of the game.

The second calculation unit 27 calculates a second evaluation value V2 according to the diversity of the actions selected by the learning unit 22. For example, in the first embodiment, the second calculation unit 27 extracts a strategy for each scene from the action history of the history recording unit 25, and calculates a second evaluation value V2 as an index for evaluating the complexity of the strategy during the game based on the strategy.

The output processing unit 28 controls the display device 19 to display the first evaluation value V1 and the second evaluation value V2 corresponding to the set value of the parameter P. The output processing unit 28 may display the learning curve (see FIG. 4) of the learning record unit 24 in association with the first evaluation value V1. The output processing unit 28 may also display details of the behavior history and a state transition diagram of the behavior (not shown) in association with the second evaluation value V2.

(Example of operation of the test support device)
3 is a flow chart showing an example of the operation of the test support device 100. The process of FIG.

In S1, the input processing unit 23 accepts the specification of a parameter P to be adjusted in the system (card game). The type and adjustment range of the parameter P to be adjusted are determined by the user. In S1 of the first embodiment, the parameter P to be adjusted is specified from, for example, the type of card in the card game, the numerical range of the card, etc. Note that the parameter P to be adjusted may be one type, or a combination of multiple types.

In S2, the simulation unit 21 determines the initial value to be applied to the test as the setting value of the specified parameter P.

In S3, the learning unit 22 acquires the setting value of the parameter P from the simulation unit 21. As a result, the parameter P becomes known in the learning unit 22, and the learning unit 22 becomes able to select an action according to the current setting of the simulation unit 21.

In S4, the learning unit 22 test plays the game as a player via the simulation unit 21, and performs reinforcement learning. For example, the test play of the game during reinforcement learning can take various forms, such as a match between a person and the learning unit 22, a match between multiple different learning units 22, or a match between the learning unit 22 and a player program without a learning function. In addition, the number of players competing in one test play can be changed as appropriate.

At this time, the history recording unit 25 records the behavioral history of the learning unit 22 during the test. In addition, the learning recording unit 24 records the winning percentage of the learning unit 22 in association with the number of test plays of the game. This makes it possible to calculate a learning curve that indicates the progress of the number of test plays of the game and the winning percentage of the learning unit 22.

In S5, the first calculation unit 26 uses the learning curve of the learning record unit 24 to determine the first evaluation value V1 according to the speed of reinforcement learning of the learning unit 22 (how few plays are required to reach a predetermined win rate). For example, if the number of plays required to reach a predetermined win rate is small, the first evaluation value V1 will indicate a low value, and if the number of plays required to reach a predetermined win rate is large, the first evaluation value V1 will indicate a high value.

Here, FIG. 4 shows an example of a learning curve of the learning unit 22. The horizontal axis of FIG. 4 is the number of test plays of the game pn, and the vertical axis of FIG. 4 is the win rate of the game wp. Furthermore, curve C1 of FIG. 4 shows an example of a learning curve when the game difficulty level is low, and curve C2 of FIG. 4 shows an example of a learning curve when the game difficulty level is higher than that of C1.

In cases where the learning unit 22 performs reinforcement learning, the learning of the learning unit 22 progresses as the number of test plays pn increases. In general, the winning percentage wp of the learning unit 22 tends to improve as the number of test plays pn increases.

In addition, when the game difficulty level is low, it is easy for the learning unit 22 to learn, but when the game difficulty level is high, it becomes difficult for the learning unit 22 to learn. In other words, when the game difficulty level is low, curve C1 reaches the win rate wp of the specified threshold Th with a relatively small number of test plays. On the other hand, when the game difficulty level is high, curve C2 reaches the win rate of the threshold Th with a larger number of test plays than curve C1.

In this way, the speed of reinforcement learning of the learning unit 22 (the number of test plays required to reach a predetermined winning rate) is highly correlated with the difficulty of the game. Therefore, it can be seen that the first evaluation value V1 can be used as an index for evaluating the difficulty of the game. For example, if the first evaluation value V1 is low, the game can be evaluated as having a low difficulty, being monotonous, and being easily boring, and if the first evaluation value V1 is high, the game can be evaluated as having a high difficulty, being complex, and not easily boring. Note that if the first evaluation value V1 is extremely high, the game may be evaluated as being too complex and difficult to get into.

In S6, the second calculation unit 27 uses the behavior history in the history recording unit 25 to determine a second evaluation value V2 according to the diversity of the behaviors selected by the learning unit 22. For example, the more diverse the behaviors selected by the learning unit 22, the higher the second evaluation value V2.

FIG. 5 is a diagram showing an outline of the process of the second calculation unit 27. As shown in FIG.
The second calculation unit 27 first refers to the multiple behavioral histories 25a recorded by the history recording unit 25. The second calculation unit 27 sets a state space from the multiple behavioral histories 25a so that each scene _Sx has an appropriate size, and then extracts a policy π for each scene _Sx in a state where reinforcement learning has progressed to a certain extent. The policy π in the scene _Sx is expressed by the following formula (1).
π(S _x , a _y )=z...(1)

In formula (1), "a _y " denotes one of the actions that the agent can select in the scene S _x , and "z" denotes the probability that the agent selects the action a _y in the scene S _x .

Then, the second calculation unit 27 calculates a second evaluation value V2 based on two features, namely, the total number of scenes _Sx and the probability distribution of actions included in the policy π of each scene _Sx .

Here, since the more the number of scenes _Sx increases, the more diverse the situations that arise during play become, and the more complicated the game tends to become, the second calculation unit 27 increases the value of the second evaluation value V2 in accordance with the total number of scenes _Sx .

Furthermore, if the actions a that can be selected in the scene _Sx are diverse when reinforcement learning has progressed to a certain extent, the actions in the scene _Sx are less likely to be standardized, that is, the actions during the game become complex. Therefore, the second calculation unit 27 adjusts the second evaluation value V2 based on the probability distribution for each action included in the policy π. For example, in a case where the probabilities of multiple actions in the policy π are each equal to or greater than a predetermined value, the second calculation unit 27 sets the second evaluation value V2 higher than a case where only one action in the policy π has a high probability. The diversity of the actions a in the policy π may be obtained using, for example, the distribution of the values of the π matrix or the amount of compressed information obtained by spectral decomposition or the like.

The second evaluation value V2 obtained in the above manner can be used as an indicator of the fun factor of the game (or the strategic level of the game) resulting from the complexity of the game's development.

In S7, the test support device 10 judges whether the test end condition is satisfied with the current setting values. For example, the test support device 10 judges that the test end condition is satisfied with the current setting values when the state of the reinforcement learning of the learning unit 22 satisfies a predetermined convergence condition or when the number of plays of the learning unit 22 reaches a certain number or more.
If the test end condition is met (Yes in S7), the process proceeds to S8. On the other hand, if the test end condition is not met (No in S7), the process returns to S4 and the above process is repeated.

In S8, the test support device 10 determines whether testing has been performed in the entire test range of the specified parameter P. If testing has been performed in the entire test range (Yes in S8), the process proceeds to S10. On the other hand, if testing has not been performed in the entire test range (No in S8), the process proceeds to S9.

At S9, the simulation unit 21 updates the set value of the parameter P to the next value. After that, the process returns to S4 and the above process is repeated. As a result, reinforcement learning by the learning unit 22 is sequentially performed over the entire test range of the parameter P, and a first evaluation value V1 and a second evaluation value V2 are obtained for each set value over the entire range of the parameter P.

In S10, the output processing unit 28 displays the test results of the parameter P on the display device 19. In displaying the test results in S10, for example, a first evaluation value V1 and a second evaluation value V2 are displayed in association with each set value of the entire test range of the parameter P. This allows the user to select an optimal value for the parameter P by referring to the first evaluation value V1 and the second evaluation value V2. In addition, the display of the test results in S10 allows the user to qualitatively grasp the overall tendency of the parameter P in the parameter space.
Then, the process of FIG. 3 ends.

As described above, in the first embodiment, the learning unit 22 tests the system simulated by the simulation unit 21 based on the set value of the parameter P, and performs reinforcement learning using the results of the test (S4). The first calculation unit 26 calculates a first evaluation value V1 according to the speed of reinforcement learning of the learning unit 22 (S5). The second calculation unit 27 calculates a second evaluation value V2 according to the diversity of actions selected by the learning unit 22 during the test (S6). Then, the output processing unit 28 causes the display device 19 to display the first evaluation value V1 and the second evaluation value V2 corresponding to the set value of the parameter P (S10).

In the first embodiment, the simulation unit 21 uses the learning unit 22 to test the system to be tested, which makes it possible to significantly reduce the number of cases where testing is performed in a real environment. In addition, because the testing in the simulation unit 21 is performed by the learning unit 22, the effort required for human test play can be significantly reduced.

In addition, in the first embodiment, a first evaluation value V1 and a second evaluation value V2 corresponding to the setting value of the parameter P are presented to the user through reinforcement learning by the learning unit 22. Since the above evaluation values V1 and V2 are quantitative indexes that vary according to the setting value of the parameter P, the use of the evaluation values V1 and V2 makes it easy for the user to objectively evaluate the quality of the parameter settings of the system.

Therefore, according to the test support device 10 of the first embodiment, in optimizing a system taking into account emotional factors, it is possible to reduce the human workload during test play and evaluation of test results.

Second Embodiment
The second embodiment is a modified example of the first embodiment, and describes an example of parameter adjustment related to a human-machine interface (HMI) of a mechanical device. The hardware configuration and software configuration of the test support device 10 in the second embodiment, as well as the operation example, are all similar to those in the first embodiment. Therefore, in the description of the second embodiment, differences from the first embodiment will be described, and any overlapping description will be omitted.

In the second embodiment, as an example, we consider a case in which an estimate of an arbitrary parameter P is obtained by testing in a virtual environment in the preliminary stage of designing an HMI for driving an automobile.

The simulation unit 21 in the second embodiment is configured as an automobile simulator that simulates the behavior of an automobile via a driving HMI in a virtual environment. Elements of the driving HMI include, for example, the steering wheel, brakes, accelerator, shift lever, etc. of the automobile. The simulation unit 21 is capable of changing the setting values of parameters P (for example, the responsiveness of the steering wheel, brakes, accelerator, shift lever, etc.) related to the steering wheel, brakes, accelerator, shift lever, etc. of the automobile.

The learning unit 22 in the second embodiment functions as a driver of the automobile simulator of the simulation unit 21, and performs reinforcement learning based on the driving results. Note that the learning record unit 24 in the second embodiment records, as information on the learning curve, for example, the number of times the learning unit 22 has driven and information showing the progress of the score that evaluates the accuracy of the driving.

The operation of the first calculation unit 26 in the second embodiment is similar to that in the first embodiment, but the meaning of the first evaluation value V1 acquired in S5 of FIG. 3 is different as follows. In the case of an HMI for a mechanical device, the easier the reinforcement learning is, the simpler the operation is, and the easier it is to use, and the better the HMI can be considered to be. Therefore, in the second embodiment, when the speed of reinforcement learning of the learning unit 22 is fast (when the first evaluation value V1 is low), it is sufficient to evaluate the current settings as being better than when the speed of reinforcement learning is slow (when the first evaluation value V1 is high).

Similarly, the operation of the second calculation unit 27 in the second embodiment is the same as that in the first embodiment, but the meaning of the second evaluation value V2 acquired in S6 in Fig. 3 is different as follows. In the case of an HMI of a mechanical device, as the total number of scenes _Sx increases and as the number of types of actions to be selected in each scene increases, more complex operations and situational judgments become necessary, and the difficulty of operating the HMI increases. Therefore, in the second embodiment, the cases where the actions selected by the learning unit 22 are diverse and the second evaluation value V2 is high should be evaluated lower than the cases where the second evaluation value V2 is low.

In the second embodiment, the display in S10 of FIG. 3 is the same as that in the first embodiment. This makes it possible to obtain the same effect as in the first embodiment when adjusting the parameters of the HMI of a machine device such as in the second embodiment.

<Modifications of the embodiment>
A program for implementing one or more of the functions of the above-described embodiments may be provided to the device via a network. Also, the configuration of the above-described embodiments may be implemented using a hardware circuit (e.g., ASIC) for implementing one or more of the functions.

In addition, the cases in which parameter adjustment tests are performed using the test support device 10 are not limited to those in the above embodiment. For example, tests may be performed using the test support device 10 on the HMI of a mechanical device other than an automobile. In addition, for example, it is also possible to test the parameter P of additional content for a game, and adjust the game balance with the additional content introduced using the test support device 10.

In the above embodiment, a configuration has been described in which an AI player playing in the simulation unit 21 performs reinforcement learning. However, depending on the problem situation, such as the configuration of the test target and the range of the parameter P, a rule-based agent may be used as the AI player. When a rule-based agent is used, it is possible to significantly reduce the learning time.

In the above embodiment, an example has been described in which reinforcement learning of the learning unit 22 is performed over the entire test range of the parameter P. However, the test support device 10 may search for a parameter P that brings the first evaluation value V1 and the second evaluation value V2 into a desired state, and end the test when the search for the parameter P that satisfies the condition is completed.

The above describes preferred embodiments of the present invention, but the present invention is not limited to these embodiments, and various modifications and variations are possible within the scope of the gist of the invention.

Reference Signs List 10: Test support device, 11: CPU, 14: Storage device, 19: Display device, 21: Simulation unit, 22: Learning unit, 26: First calculation unit, 27: Second calculation unit, 28: Output processing unit

Claims (7)

A simulation unit (21) that can simulate a system to be tested in a virtual environment and set parameters related to the system;
a learning unit (22) that tests the system simulated by the simulation unit (21) based on the set parameters and performs machine learning using the results of the test;
a first calculation unit (26) that calculates a first evaluation value in accordance with a learning speed of the learning unit (22);
a second calculation unit (27) that calculates a second evaluation value according to the diversity of actions selected by the learning unit (22) during a test;
an output processing unit (28) for displaying on a display device (19) the first evaluation value and the second evaluation value corresponding to the set parameters;
A test support device (10) comprising: The test support device (10) according to claim 1, wherein the learning unit (22) performs reinforcement learning using the results of the test. The test support device (10) according to claim 2, wherein the first calculation unit (26) determines the first evaluation value based on a learning curve of reinforcement learning by the learning unit (22). The device further includes a recording unit (25) for recording a behavior history during a test performed by the learning unit (22),
The test support device (10) according to claim 2 or 3, wherein the second calculation unit (27) extracts a measure for each scene from the action history, and calculates the second evaluation value based on the measure. The test support device (10) according to claim 4, wherein the second calculation unit (27) determines the second evaluation value based on at least one of the number of scenes or a distribution of probabilities for the actions of the policy. The learning unit (22) tests the system under a plurality of conditions each having a different value of the parameter,
The test support device (10) according to any one of claims 1 to 5, wherein the output processing unit (28) causes the display device (19) to display the first evaluation value and the second evaluation value corresponding to each of the plurality of conditions. A step (S4) of having a learning unit (22) test the system simulated in the virtual environment based on the set parameters, and performing machine learning of the learning unit (22) using the results of the test;
A step (S5) of calculating a first evaluation value according to a learning speed of the learning unit (22);
A step (S6) of obtaining a second evaluation value according to the diversity of actions selected by the learning unit (22) during testing;
a step (S10) of displaying on a display device (19) the first evaluation value and the second evaluation value corresponding to the set parameters;
A program for causing a computer (11) to execute the above.

Images