JP7601241B2 - Support device, support method, and support program - Google Patents

Description

The present invention relates to an assistance device, an assistance method, and an assistance program.

When using real-world environments for reinforcement learning of models, care must be taken to avoid irreparably damaging the environment.

For example, reinforcement learning is used to control self-driving cars and robots. However, during reinforcement learning in real-world environments, it is possible that the robot's hardware may be damaged or that the car may have a collision.

In response to this, non-patent document 1 discloses a reinforcement learning technique based on Lyapunov's method that rewards safe behavior.

In addition, non-patent document 2 discloses a technology called CARL (Cautious Adaptation For Reinforcement Learning), in which uncertainties regarding transition dynamics and catastrophic states are trained in a sandbox, and in the real environment, the agent learns to avoid catastrophic states.

Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzman, Mohammad Ghavamzadeh, "A Lyapunov-based Approach to Safe Reinforcement Learning", arXiv:1805.07708v1 [cs.LG] 20 May 2018 Jesse Zhang, Brian Cheung, Chelsea Finn, Sergey Levine, Dinesh Jayaraman, "Cautious Adaptation For Reinforcement Learning in Safety-Critical Settings", arXiv:2008.06622v1 [cs.LG] 15 Aug 2020

However, conventional technologies have the problem that it can be difficult to perform reinforcement learning to support business operations at low cost and easily.

In reinforcement learning to support business operations, an agent is made to try and error various actions in the business environment and learn the optimal action. In this case, it is necessary to prepare the environment in which the agent can take action.

For example, in business, it is often unclear what should be defined as safe behavior. For this reason, it is difficult to apply the technology in Non-Patent Document 1 to business and design appropriate rewards.

In addition, preparing a simulation environment such as the sandbox described in non-patent document 2 is very costly, and it is difficult to prepare a simulation environment that is optimal for learning about a business.

In order to solve the above-mentioned problems and achieve the objectives, the support device is characterized by having an acquisition unit that acquires environmental information, which is information about the environment in a task, and behavioral information, which is information about behavior in the task, an extraction unit that associates and extracts the environmental information and the behavioral information, and a design unit that designs an environment and reward in reinforcement learning of a model that performs the task based on the combination of the environmental information and the behavioral information associated by the extraction unit.

According to the present invention, reinforcement learning to support business operations can be performed easily and at low cost.

FIG. 1 is a diagram illustrating an example of the configuration of a support device according to a first embodiment. FIG. 2 is a diagram showing an example of the environmental information and the behavioral information. FIG. 3 is a diagram showing an example of the environmental information and the behavioral information. FIG. 4 is a diagram showing an example of the environmental information and the behavioral information. FIG. 5 is a diagram showing an example of the environmental information and the behavioral information. FIG. 6 is a diagram showing an example of an environment design and an action design. FIG. 7 is a flowchart showing the flow of the acquisition process. FIG. 8 is a flowchart showing the flow of the extraction process. FIG. 9 is a flowchart showing the flow of the learning process. FIG. 10 is a flowchart showing the flow of the execution process. FIG. 11 is a diagram illustrating an example of a computer that executes the assistance program.

Below, the embodiments of the support device, support method, and support program according to the present application are described in detail with reference to the drawings. Note that the present invention is not limited to the embodiments described below.

Conventionally, reinforcement learning of models to support business operations is performed in an environment in which the business operations are actually carried out. In such cases, it is conceivable that adverse effects on the business operations may occur. On the other hand, according to an embodiment, reinforcement learning of models to support business operations can be performed while preventing adverse effects on the business operations.

In the embodiments, the tasks include all tasks performed by humans. For example, the tasks include inputting data into a terminal device such as a personal computer (PC), responding to customer inquiries by voice or text, inspecting equipment, etc.

For example, a model that automates input operations into a terminal device automatically performs operations on the terminal device that mimic human input operations based on a captured image of the screen displayed by the terminal device. The model may also use, for example, a neural network.

[Configuration of the first embodiment]
First, the configuration of the support device according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of the support device according to the first embodiment.

As shown in FIG. 1, the support system 1 has a support device 10 and a terminal device 20. The support device 10 is connected to the terminal device 20. The terminal device 20 is a device such as a PC that a worker uses to perform work related to his or her job. The support device 10 may also be connected to a camera, microphone, or wearable device carried by the worker.

Here, we will explain each part of the support device 10. As shown in Figure 1, the support device 10 has an input/output unit 11, a memory unit 12, and a control unit 13.

The input/output unit 11 is an interface for inputting and outputting data. For example, the input/output unit 11 is a NIC (Network Interface Card). The input/output unit 11 can send and receive data with other devices.

The input/output unit 11 may also be connected to an input device such as a mouse or a keyboard. The input/output unit 11 may also be connected to an output device such as a display or a speaker.

The storage unit 12 is a storage device such as a hard disk drive (HDD), a solid state drive (SSD), an optical disk, etc. The storage unit 12 may be a semiconductor memory in which data can be rewritten, such as a random access memory (RAM), a flash memory, or a non-volatile static random access memory (NVSRAM).

The memory unit 12 stores the OS (Operating System) and various programs executed by the support device 10. The memory unit 12 stores learning information 121 and model information 122.

Learning information 121 is information for performing reinforcement learning. Learning information 121 includes rewards and environments in reinforcement learning.

Model information 122 is information such as parameters for constructing a model that supports business operations. If the model is a neural network, model information 122 includes weights and biases for each node.

The model receives input of information indicating the work environment and outputs information related to the behavior. The assistance device 10 assists with the work based on the output information.

The control unit 13 controls the entire support device 10. The control unit 13 is, for example, an electronic circuit such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), or GPU (Graphics Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

The control unit 13 also has an internal memory for storing programs and control data that define various processing procedures, and executes each process using the internal memory. The control unit 13 also functions as various processing units by the operation of various programs. For example, the control unit 13 has an acquisition unit 131, an extraction unit 132, a design unit 133, a learning unit 134, and an execution unit 135.

The acquisition unit 131 acquires environmental information, which is information regarding the environment in business, and behavioral information, which is information regarding behavior in business.

The acquisition unit 131 can acquire information on the behavior of a worker performing a task as behavioral information, and acquire information on the environment related to the worker as environmental information. The acquisition unit 131 can also acquire the content of an operation performed by the worker on the terminal device 20 as behavioral information, and acquire the state of the terminal device 20 that changes in response to the operation performed by the worker as environmental information.

For example, the acquisition unit 131 acquires an image captured by a camera in the direction of the worker's viewpoint and audio around the worker collected by a microphone as environmental information.

Also, for example, the acquisition unit 131 acquires information such as images and audio output from a terminal device 20 operated by a worker as environmental information.

The image acquired by the acquisition unit 131 may be a still image such as a captured image, or may be a moving image. The acquisition unit 131 may also acquire the collected voice as an audio file, or may acquire text obtained by converting the collected voice.

For example, the acquisition unit 131 acquires information on the worker's physical movements sensed by a sensor attached to the worker and the worker's speech collected by a microphone as behavioral information.

In addition, for example, the acquisition unit 131 acquires, as behavioral information, the content of an operation performed by the worker on the terminal device 20. The content of the operation includes the time when the keyboard was pressed, the type of key pressed, the trajectory of the mouse movement, the position and time when the mouse was clicked, etc.

At this time, the acquisition unit 131 can acquire the operation content of the worker on the terminal device 20 as behavioral information, and acquire a captured image of the screen of the terminal device 20 that changes in response to the operation as environmental information.

Furthermore, the acquisition unit 131 may acquire information identifying the application or window being operated by the worker on the terminal device 20 as behavioral information together with the operation content.

The extraction unit 132 extracts the environmental information and the behavioral information in association with each other. In other words, the extraction unit 132 extracts a combination of the environmental information and the behavioral information acquired by the acquisition unit 131.

The extraction unit 132 can extract behavioral information relating to behavior in business in association with environmental information relating to at least one of the environment before the behavior was taken and the environment influenced by the behavior.

The correspondence between environmental information and behavioral information will be explained using Figures 2 to 5. Figures 2, 3, 4 and 5 are diagrams showing examples of environmental information and behavioral information.

Figures 2 and 3 show examples of environmental information and behavioral information related to work using the terminal device 20.

In the example of FIG. 2, the extraction unit 132 extracts a captured image 51a and an operation content 52a in association with each other. The captured image 51a corresponds to environmental information. The operation content 52a corresponds to behavioral information.

The worker inputs information into various fields and clicks (presses) button 511a with the mouse. In this case, operation content 52a includes information indicating that the type of operation event is a click, and the coordinates of the cursor when the click is performed.

In this case, for example, the environment before the action is taken is a captured image of the screen before button 511a is pressed, while the environment affected by the action is a captured image of the screen to which the action transitions after button 511a is pressed.

The extraction unit 132 may extract the entire captured image 51a as environmental information, or may extract only a part of the captured image 51a, that is, the button 511a that was the target of the operation, as environmental information.

Note that the capture image 51a in Fig. 2 is an image obtained by cutting out a part of the screen displayed on the display. On the other hand, the environmental information may be an image obtained by capturing the entire screen displayed on the display, including the task bar of the OS and the toolbar of the browser or a specified application.

In the example of FIG. 3, the extraction unit 132 extracts a captured image 51b and an operation content 52b in association with each other. The captured image 51b corresponds to environmental information. The operation content 52b corresponds to behavioral information.

To input "yokosuka" in the Roman alphabet, the worker first types (presses) the "y" key on the keyboard in text box 511b, and then types the "o" key on the keyboard.

In this case, operation content 52b includes the type of operation event being the pressing of the "o" key.

In this case, for example, the environment before the action is taken is a captured image of the screen before the "o" key is pressed. On the other hand, the environment affected by the action is a captured image of the screen after the "o" key is pressed and "yo" is entered in text box 511b. Note that in romaji input, the hiragana "yo" is displayed by entering "o" after "y."

The extraction unit 132 may extract the entire captured image 51b as environmental information, or may extract only a part of the captured image 51b, that is, the text box 511b that was the target of the operation, as environmental information.

Note that the capture image 51b in Fig. 3 is an image obtained by cutting out a part of the screen displayed on the display. On the other hand, the environmental information may be an image obtained by capturing the entire screen displayed on the display, including the task bar of the OS and the toolbar of the browser or a specified application.

The extraction unit 132 may extract a series of multiple pieces of environmental information by associating one piece of behavioral information with the plurality of pieces of environmental information. For example, the extraction unit 132 may extract, as environmental information, capture images of multiple frames of a screen in a chronological order until a predetermined operation content occurs on the terminal device 20.

In addition, the extraction unit 132 may further extract environmental information regarding an environment whose similarity to the environment corresponding to the extracted environmental information is equal to or greater than a threshold value, by associating it with behavioral information.

For example, in addition to the captured image 51b, a past captured image similar to the captured image 51b may be extracted in association with the operation content 52b.

For example, the number of words that appear in common in two captured images is taken as the similarity. The extraction unit 132 then considers captured images whose similarity is equal to or greater than a threshold to be similar.

In other words, the extraction unit 132 extracts not only environmental information at the moment when the behavior indicated by the behavior information was performed, but also similar environmental information from the past.

Figure 4 shows an example of environmental information and behavioral information related to telephone response work.

In the example of FIG. 4, the extraction unit 132 extracts voice 51c of a telephone inquiry from a customer and voice 52c of the operator's response in association with each other. Voice 51c corresponds to environmental information. Voice 52c corresponds to behavioral information.

At this time, the extraction unit 132 may extract text transcribed from the audio instead of the audio.

In this case, for example, the environment before the action is taken is the voice of the customer's telephone inquiry 51c. On the other hand, the environment influenced by the action is the voice uttered by the customer in response to the voice of the operator's response 52c.

Figure 5 shows an example of environmental information and behavioral information related to equipment inspection work.

In the example of FIG. 5, the extraction unit 132 extracts an image 51d from the viewpoint of a moving worker and a position 52d of a destination to which the worker has moved in association with each other. The image 51d corresponds to environmental information. The position 52d corresponds to behavioral information.

In this case, for example, the environment before the action is taken is the image 51d from the viewpoint of the worker while he is moving, while the environment affected by the action is the image from the viewpoint of the worker after he has moved.

The design unit 133 designs the environment and rewards in reinforcement learning of the model that performs the task based on the combination of environmental information and behavioral information associated by the extraction unit 132.

The design unit 133 assumes that the behavior indicated by the behavioral information is the "correct behavior" and designs the system so that a reward is given if the agent performs a similar behavior in the environment in which the behavior was performed.

Figure 6 is a diagram showing an example of environmental design and behavioral design. As shown in Figure 6, the design unit 133 performs environmental design and reward design.

For example, if clicking a specific button is the "right action," the design unit 133 designs the system so that a positive reward is given to clicking on that button and moving the cursor onto that button. On the other hand, the design unit 133 designs the system so that a negative reward (penalty) is given to clicking anywhere other than on the button.

Furthermore, the design department 133 designs the system so that when the same operation as that performed by the worker, i.e., clicking a button, is performed, the environment transitions to a captured image after the operation, and when an operation other than clicking a button is performed, the environment does not transition and the agent re-executes the operation on the same screen.

Here, the extraction unit 132 extracts both environmental information related to the environment before the action is taken and environmental information related to the environment influenced by the action. At this time, the design unit 133 designs the environment before the action is taken to be presented to the agent, and transitions to the environment influenced by the action when the agent takes the "correct action."

In addition, the design unit 133 stores the design contents in the memory unit 12 as learning information 121.

The learning unit 134 constructs a learning environment for performing reinforcement learning of the model according to the learning information 121. Furthermore, the learning unit 134 updates the model information 122 based on the results of having the agent perform an action in the learning environment.

If the task involves the use of the terminal device 20, the learning unit 134 presents the agent with a captured image of the screen of the terminal device 20 as the environment, and allows the agent to select an action to be taken on the captured image (such as clicking or moving the cursor).

If the task involves movement by walking, the learning unit 134 presents the agent with a moving image from the viewpoint of the worker while walking or a still image extracted from the moving image as the environment, and allows the agent to select the direction in which to move.

In this way, by using environmental information about the worker's work environment as a learning environment and having the agent act in that environment, it is possible to learn without affecting actual work.

In addition, an agent is a simulated entity that selects actions based on the output of a model constructed from model information 122.

The learning unit 134 may also perform learning based on environmental information and behavioral information about a single worker. In this case, it is expected that behavior that reflects the characteristics of each worker can be learned.

On the other hand, the learning unit 134 may also perform learning by combining environmental information and behavioral information about multiple workers. In this case, it is expected that more efficient learning of work procedures can be achieved.

The execution unit 135 generates a sequence of tasks-related actions using a model that has undergone reinforcement learning based on the environment and rewards designed by the design unit 133.

For example, the execution unit 135 inputs environmental information from actual business operations into a learned model constructed from the model information 122 and identifies an action based on the output obtained.

Specifically, the execution unit 135 generates a sequence of actions from environmental information during the worker's work using a learned model, and provides support for the work based on the generated sequence of actions.

Work support may involve directly performing tasks or providing workers with actions to take in their work.

For example, the execution unit 135 automatically inputs information into the fields based on a captured image of the screen of the terminal device. In addition, for example, the execution unit 135 may infer the next task to be performed from the viewpoint video of the worker and provide information about the inferred task by voice.

[Processing of the First Embodiment]
7 is a flowchart showing the flow of the acquisition process. As shown in FIG 7, if the worker has not finished the work (step S101, No), the acquisition unit 131 acquires environmental information about the worker's working environment (step S102).

Then, if the worker takes action (step S103, Yes), the acquisition unit 131 acquires the worker's action information (step S104).

If the worker does not take any action (step S103, No), the acquisition unit 131 returns to step S101.

Here, if the worker has finished the work (step S101, Yes), the acquisition unit 131 terminates the processing.

Figure 8 is a flowchart showing the flow of the extraction process. If no corresponding environment has been extracted for all of the behavioral information acquired by the acquisition unit 131 (step S201, No), the extraction unit 132 determines the behavioral information to be targeted (step S202).

Then, the extraction unit 132 extracts environmental information corresponding to the targeted behavioral information (step S203).

If the extraction unit 132 extracts corresponding environments for all the behavioral information acquired by the acquisition unit 131 (step S201, Yes), the extraction unit 132 terminates the processing.

Figure 9 is a flowchart showing the flow of the learning process. Here, it is assumed that the reward and environment for reinforcement learning have already been designed by the design unit 133.

As shown in FIG. 9, if the acquired environmental information does not generate behavior similar to that of the worker (step S301, No), the learning unit 134 determines the target environmental information (step S302).

The learning unit 134 uses the environmental information as a target and as an environment for reinforcement learning to learn the actions to be taken by trial and error (step S303). The learning unit 134 updates the model information 122 based on the results of the learning.

If it becomes possible to generate behavior similar to that of the worker based on the acquired environmental information (step S301, Yes), the learning unit 134 terminates the processing.

Figure 10 is a flowchart showing the flow of the execution process. Here, the execution unit 135 constructs a trained model from the model information 122.

As shown in FIG. 10, if the worker has not finished the work (step S401, No), the execution unit 135 acquires the worker's environmental information (step S402).

Then, if the execution unit 135 can generate an appropriate behavior sequence for the environment (step S403, Yes), it executes the behavior sequence generated using the model (step S404).

If the execution unit 135 cannot generate an appropriate behavior sequence for the environment (step S403, No), it returns to step S401.

If the worker has finished the work (step S401, Yes), the execution unit 135 terminates the processing.

[Effects of the First Embodiment]
As described above, the acquisition unit 131 acquires environmental information, which is information about the environment in a task, and behavioral information, which is information about behavior in the task. The extraction unit 132 extracts the environmental information and the behavioral information in association with each other. The design unit 133 designs an environment and a reward in reinforcement learning of a model that executes a task, based on the combination of the environmental information and the behavioral information associated with each other by the extraction unit 132.

In this way, the support device 10 can design reinforcement learning based on the behavior and environment related to the work. As a result, according to the embodiment, reinforcement learning for supporting work can be performed easily and at low cost.

In addition, the acquisition unit 131 acquires information on the behavior of the worker performing the task as behavioral information, and acquires information on the environment related to the worker as the environmental information. In this way, by focusing on the behavior and environment of the worker, it is possible to easily acquire behavioral information and environmental information.

In addition, the acquisition unit 131 acquires the content of the operation of the terminal device 20 by the worker as behavioral information, and acquires the state of the terminal device 20 that changes in response to the operation by the worker as environmental information. This makes it possible to easily acquire environmental information related to the work using the terminal device.

In addition, the extraction unit 132 extracts first behavioral information related to behavior in a business and first environmental information related to at least one of the environment before the behavior was taken and the environment influenced by the behavior, in association with each other. This makes it easy to design reinforcement learning from the related behavioral information and environmental information.

In addition, the extraction unit 132 further extracts second environmental information related to an environment whose similarity to the environment is equal to or greater than a threshold value by associating it with the first behavioral information. In this way, by associating multiple pieces of environmental information similar to the behavioral information, the accuracy of reinforcement learning can be improved.

In addition, the execution unit 135 generates a sequence of actions related to the task using a model that has undergone reinforcement learning based on the environment and rewards designed by the design unit 133. This makes it possible to reduce human work and judgment related to the task.

[System configuration, etc.]
In addition, each component of each device shown in the figure is functionally conceptual, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or a part of it can be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc. Furthermore, each processing function performed by each device can be realized in whole or in part by a CPU (Central Processing Unit) and a program analyzed and executed by the CPU, or can be realized as hardware by wired logic. Note that the program may be executed not only by the CPU but also by other processors such as a GPU.

Furthermore, among the processes described in this embodiment, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.

[program]
In one embodiment, the support device 10 can be implemented by installing a support program that executes the above support processing as package software or online software on a desired computer. For example, the above support program can be executed by an information processing device, causing the information processing device to function as the support device 10. The information processing device here includes desktop or notebook personal computers. In addition, the information processing device also includes mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone System), as well as slate terminals such as PDA (Personal Digital Assistant), etc.

The support device 10 can also be implemented as a support server device that provides services related to the above support processing to a client, the client being a terminal device used by the user. For example, the support server device is implemented as a server device that provides a support service that takes action information and environmental information in a business as input and outputs a trained model for supporting the business. In this case, the support server device may be implemented as a web server, or may be implemented as a cloud that provides services related to the above support processing by outsourcing.

Figure 11 is a diagram showing an example of a computer that executes an assistance program. The computer 1000 has, for example, a memory 1010 and a CPU 1020. The computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. Each of these components is connected by a bus 1080.

The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM (Random Access Memory) 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the program that defines each process of the support device 10 is implemented as a program module 1093 in which computer-executable code is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing processes similar to the functional configuration of the support device 10 is stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

In addition, the setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in memory 1010 or hard disk drive 1090. Then, CPU 1020 reads out program module 1093 and program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as necessary, and executes the processing of the above-described embodiment.

Note that the program module 1093 and the program data 1094 are not limited to being stored in the hard disk drive 1090, but may be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and the program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or wide area network (WAN)). The program module 1093 and the program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

REFERENCE SIGNS LIST 1 Support system 10 Support device 11 Input/output unit 12 Storage unit 13 Control unit 20 Terminal device 51a, 51b Captured image 51c, 52c Audio 51d Video 52a, 52b Operation content 52d Position 121 Learning information 122 Model information 131 Acquisition unit 132 Extraction unit 133 Design unit 134 Learning unit 135 Execution unit 511a Button 511b Text box

Claims (6)

an acquisition unit that acquires environmental information, which is information about an environment in a business, and behavioral information, which is information about behavior in the business;
an extracting unit that extracts, in association with each other, first behavioral information related to the behavior in the business and first environmental information related to at least one of an environment before the behavior was taken and an environment influenced by the behavior, and further extracts, in association with the first behavioral information, second environmental information related to an environment having a similarity to the environment equal to or greater than a threshold ;
a design unit that designs an environment and a reward in reinforcement learning of a model that executes the task based on a combination of the first environmental information , the second environmental information, and the first behavior information associated by the extraction unit;
An assistance device comprising: The support device according to claim 1, characterized in that the acquisition unit acquires information about the behavior of a worker performing the task as the behavior information, and acquires information about an environment related to the worker as the environment information. The support device according to claim 2, characterized in that the acquisition unit acquires the content of the operation of the terminal device by the worker as the behavioral information, and acquires the state of the terminal device that changes in response to the operation by the worker as the environmental information. 4. The support device according to claim 1, further comprising an execution unit that generates a series of actions related to a task using a model that has undergone reinforcement learning based on the environment and rewards designed by the design unit. An assistance method executed by an assistance device, comprising:
An acquisition step of acquiring environmental information, which is information about an environment in a business, and behavioral information, which is information about behavior in the business;
an extraction step of extracting, in association with each other, first behavioral information related to the behavior in the business and first environmental information related to at least one of an environment before the behavior was taken and an environment influenced by the behavior, and further extracting, in association with the first behavioral information, second environmental information related to an environment having a similarity to the environment equal to or greater than a threshold ;
a design process of designing an environment and a reward in reinforcement learning of a model that executes the task based on a combination of the first environmental information , the second environmental information, and the first behavioral information that are associated by the extraction process;
Assistance method comprising: A support program for causing a computer to function as the support device according to any one of claims 1 to 4 .

Images