JP7793386B2 - Processing system and processing method - Google Patents

Description

The present invention relates to a processing system and a processing method.

In recent years, various technologies related to artificial intelligence (AI) that use behavioral models to determine behavior depending on the situation have been proposed. For example, Patent Document 1 discloses technology related to learning behavioral models through reinforcement learning.

US Patent Application Publication No. 2010/094786

By the way, it is conceivable that artificial intelligence technology could be utilized in workplaces such as factory production sites. Here, it is important to transfer the high level of skills possessed by experienced workers at the workplace. Skill transfer requires experienced workers to allocate time. Therefore, it is desirable to transfer skills at the workplace efficiently.

In light of these issues, the present invention aims to provide a processing system and processing method that enables efficient transfer of skills at the workplace.

To solve the above problem, the processing system determines behavior using a behavior model for determining behavior depending on the state, and the behavior model includes a first behavior model that is an imitation model that imitates the behavior of a first worker, and a second behavior model that is an imitation model that imitates the behavior of a second worker. The processing system includes a first determination unit that determines behavior using the first behavior model, a first learning unit that learns the first behavior model, and a selection unit that selects a specific state from a plurality of states. The first learning unit learns the first behavior model through reinforcement learning for each of the plurality of states, and the selection unit compares the first behavior model and the second behavior model after learning for each of the plurality of states and selects a specific state based on the results of the comparison.

To solve the above problem, the processing method determines behavior using a behavior model for determining behavior depending on a state, the behavior model including a first behavior model that is an imitation model that imitates the behavior of a first worker, and a second behavior model that is an imitation model that imitates the behavior of a second worker, and includes a first step of determining behavior using the first behavior model, a second step of learning the first behavior model, and a third step of selecting a specific state from a plurality of states, in which in the second step, the first behavior model is learned by reinforcement learning for each of the plurality of states, and in the third step, the first behavior model and the second behavior model after learning are compared for each of the plurality of states, and a specific state is selected based on the results of the comparison.

This invention makes it possible to efficiently transfer skills at the workplace.

1 is a schematic diagram illustrating a schematic configuration of a processing system according to an embodiment of the present invention. FIG. 2 is a block diagram showing an example of a functional configuration of a first processing device according to an embodiment of the present invention. FIG. 2 is a block diagram showing an example of a functional configuration of a second processing device according to an embodiment of the present invention. 1 is a flowchart showing an example of an overall flow of processing performed by a first robot, a second robot, and a first worker according to an embodiment of the present invention. FIG. 10 is a diagram showing an entity that executes a selection process of an operation status according to an embodiment of the present invention. 10 is a flowchart showing an example of a process flow in a work state selection process according to an embodiment of the present invention.

Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Dimensions, materials, and other specific values shown in these embodiments are merely examples to facilitate understanding of the invention and, unless otherwise specified, do not limit the present invention. Furthermore, in this specification and drawings, elements that have substantially the same function and configuration are designated by the same reference numerals to avoid redundant explanation, and elements that are not directly related to the present invention are not shown.

<Configuration of processing system>
The configuration of a processing system 1 according to an embodiment of the present invention will be described with reference to FIGS.

Figure 1 is a schematic diagram showing the general configuration of the processing system 1.

As shown in FIG. 1, the processing system 1 includes a first robot 10 and a second robot 20. The processing system 1 is utilized in a work site such as a factory production site. Below, an example is described in which the processing system 1 is utilized in a production process at a factory production site. The production process is a process for producing products. However, the work site in which the processing system 1 is utilized may also be a work site other than a production site where products are produced (for example, a site where product inspection is performed).

The first robot 10 and the second robot 20 are robots that use artificial intelligence to determine and execute actions. Specifically, the first robot 10 and the second robot 20 have behavioral models for determining actions depending on the state, and determine and execute actions using the behavioral models. The first robot 10 is equipped with a first processing device 11 that performs various processes related to the first robot 10. The first processing device 11 determines the actions of the first robot 10 using the first behavioral model π1 and causes the first robot 10 to execute the determined actions. The second robot 20 is equipped with a second processing device 21 that performs various processes related to the second robot 20. The second processing device 21 determines the actions of the second robot 20 using the second behavioral model π2 and causes the second robot 20 to execute the determined actions.

In a production process in which the processing system 1 is utilized, products are produced by multiple workers. These multiple workers include a first worker 30 and a second worker 40. The first worker 30 is an inexperienced worker with lower skills than the second worker 40. For example, the first worker 30 is a worker with fewer years of work experience than the second worker 40. The second worker 40 is a skilled worker with higher skills than the first worker 30. For example, the second worker 40 is a skilled worker with more years of work experience than the first worker 30.

The first robot 10 is capable of imitating the actions of the first worker 30. In the example shown in Figure 1, the first robot 10 is a humanoid robot. However, the first robot 10 may be a robot other than a humanoid robot.

The second robot 20 is capable of imitating the actions of the second worker 40. In the example of Figure 1, the second robot 20 is a humanoid robot. However, the second robot 20 may be a robot other than a humanoid robot.

In production processes that utilize the processing system 1, it is important to transfer the advanced skills of the experienced second worker 40 to the less experienced first worker 30. By utilizing the first robot 10 and the second robot 20, the processing system 1 can promote the improvement of the skills of the first worker 30 and enable the transfer of skills to be carried out efficiently, as described below.

Figure 2 is a block diagram showing an example of the functional configuration of the first processing device 11.

The first processing device 11 includes a CPU (Central Processing Unit), which is an arithmetic processing device; a ROM (Read Only Memory), which is a memory element that stores programs used by the CPU, calculation parameters, etc.; and a RAM (Random Access Memory), which is a memory element that temporarily stores parameters that change as the CPU executes.

As shown in FIG. 2, the first processing device 11 includes, for example, a first acquisition unit 11a, a first determination unit 11b, a first control unit 11c, a first learning unit 11d, a selection unit 11e, and a first memory unit 11f.

The first acquisition unit 11a acquires various information necessary for controlling the first robot 10. As described above, the behavior of the first robot 10 is determined using the first behavior model π1. The first acquisition unit 11a acquires information necessary for determining behavior using the first behavior model π1.

The first determination unit 11b determines the behavior to be performed by the first robot 10. Specifically, the first determination unit 11b determines the behavior of the first robot 10 using the first behavior model π1. The first behavior model π1 is stored in the first memory unit 11f and is an imitation model that imitates the behavior of the first worker 30. The first behavior model π1 is a function that, for example, when a state-action pair is input, outputs an evaluation index (degree of recommendation) for the input behavior.

The information input as a state to the first behavioral model π1 includes, for example, information about the production site (e.g., information indicating the type of work or information about the materials used in the work), information about the first robot 10 (e.g., information indicating the position and posture of the first robot 10), or information about the worker (e.g., information indicating the position and posture of the worker). This information is acquired by the first acquisition unit 11a. The first acquisition unit 11a can acquire information about the production site and information about the worker, for example, based on images captured by a camera provided on the first robot 10. The first acquisition unit 11a can also acquire information about the first robot 10 based on, for example, the output results of various sensors provided on the first robot 10.

For example, once the state is determined, several candidate actions are determined. The first determination unit 11b can determine the action of the first robot 10 by inputting each of these candidate actions into the first behavioral model π1 and comparing the evaluation indices obtained. Specifically, the first determination unit 11b determines the action that results in the highest evaluation index as the action to be performed by the first robot 10.

The first control unit 11c causes the first robot 10 to perform the behavior determined by the first determination unit 11b using the first behavioral model π1. For example, the first control unit 11c can control the first robot 10 by controlling an actuator such as a motor provided on the first robot 10.

The first learning unit 11d learns the first behavioral model π1. Details of the learning of the first behavioral model π1 by the first learning unit 11d will be described later.

The selection unit 11e selects a specific state from among multiple states. Specifically, as described below, the selection unit 11e selects from among multiple work states the optimal work state for promoting improvement in the skills of the first worker 30. Details of the work state selection process will be described later.

The work state is the state of the work environment, for example, the location and number of boxes in the task of organizing boxes. The following describes the case where the first worker 30 is organizing boxes. In this case, the first worker 30 can efficiently improve his or her skills in the task by organizing boxes in the work state selected by the selection unit 11e. However, the task that is the subject of the first worker 30's skill improvement may be a task other than organizing boxes.

The first memory unit 11f stores various information necessary for controlling the first robot 10. Specifically, the first memory unit 11f stores the first behavioral model π1. The first behavioral model π1 stored in the first memory unit 11f is updated in accordance with the learning performed by the first learning unit 11d.

Figure 3 is a block diagram showing an example of the functional configuration of the second processing device 21.

The second processing device 21 includes a CPU, which is an arithmetic processing device; a ROM, which is a memory element that stores programs used by the CPU and calculation parameters; and a RAM, which is a memory element that temporarily stores parameters that change as the CPU executes.

As shown in FIG. 3, the second processing device 21 includes, for example, a second acquisition unit 21a, a second determination unit 21b, a second control unit 21c, a second learning unit 21d, and a second memory unit 21e.

The second acquisition unit 21a acquires various information necessary for controlling the second robot 20. As described above, the behavior of the second robot 20 is determined using the second behavior model π2. The second acquisition unit 21a acquires information necessary for determining behavior using the second behavior model π2.

The second determination unit 21b determines the behavior to be performed by the second robot 20. Specifically, the second determination unit 21b determines the behavior of the second robot 20 using the second behavior model π2. The second behavior model π2 is stored in the second memory unit 21e and is an imitation model that imitates the behavior of the second worker 40.

The second behavioral model π2 is, for example, similar to the first behavioral model π1, a function that, when a state-action pair is input, outputs an evaluation index (degree of recommendation) for the input action. The second determination unit 21b can determine the action of the second robot 20 by inputting candidate actions according to the state into the second behavioral model π2 and comparing the evaluation indices obtained. Specifically, the second determination unit 21b determines the action that results in the highest evaluation index as the action to be performed by the second robot 20. Note that the information input as a state to the second behavioral model π2 is the same as the information input as a state to the first behavioral model π1, and is acquired by the second acquisition unit 21a in a manner similar to that acquired by the first acquisition unit 11a.

The second control unit 21c causes the second robot 20 to perform the behavior determined by the second determination unit 21b using the second behavioral model π2. For example, the second control unit 21c can control the second robot 20 by controlling an actuator such as a motor provided on the second robot 20.

The second learning unit 21d learns the second behavioral model π2. Details of the learning of the second behavioral model π2 by the second learning unit 21d will be described later.

The second memory unit 21e stores various information necessary for controlling the second robot 20. Specifically, the second memory unit 21e stores the second behavioral model π2. The second behavioral model π2 stored in the second memory unit 21e is updated in accordance with the learning performed by the second learning unit 21d.

<Operation of the Processing System>
The operation of the processing system 1 according to the embodiment of the present invention will be described with reference to FIGS.

As described above, the processing system 1 is used, for example, at a factory production site to promote the improvement of the skills of the first worker 30. In the processing system 1, the process for promoting the improvement of the skills of the first worker 30 is performed by the first robot 10, the second robot 20, and the first worker 30. Below, the overall flow of the process for promoting the improvement of the skills of the first worker 30 is explained with reference to Figures 4 and 5, and then the details of the work state selection process described below are explained with reference to Figure 6.

Figure 4 is a flowchart showing an example of the overall flow of processing performed by the first robot 10, the second robot 20, and the first worker 30. Step S101 in Figure 4 corresponds to the start of the processing flow shown in Figure 4.

When the processing flow shown in FIG. 4 starts, in step S102, the first behavioral model π1 is learned by imitation learning. The first behavioral model π1 is learned by the first learning unit 11d of the first robot 10.

In step S102, the first learning unit 11d learns the first behavioral model π1 through imitation learning based on the actions performed by the first operator 30. For example, in step S104 (described later) or after step S104 (described later), the actions of the first operator 30 when actually performing work are recorded. In the record of the first operator's 30's actions, the actions performed by the first operator 30 are linked to the state (e.g., the type of work) at the time the actions were performed. Imitation learning of the first behavioral model π1 is performed using the record of the first operator's 30's actions obtained in this manner. In imitation learning of the first behavioral model π1, the first learning unit 11d causes the first robot 10 to perform various actions and calculates a reward for each action performed by the first robot 10. Here, the first learning unit 11d calculates a reward such that the closer (more similar) the action performed by the first robot 10 in a certain state is to the action of the first operator 30 corresponding to that state, the higher the reward. The first learning unit 11d calculates this reward for each state. The reward obtained in this way corresponds to the evaluation index (degree of recommendation) of the behavior described above. The first learning unit 11d uses the relationship between the obtained state-action pair and reward to update the relationship between the input (state-action pair) and output (evaluation index (degree of recommendation) of the behavior) in the first behavioral model π1.

Following step S102, in step S103, a selection process is performed to select the optimal work state from among multiple work states in order to promote improvement in the skills of the first worker 30. Figure 5 is a diagram showing the entity that executes the work state selection process. As shown in Figure 5, the work state selection process is performed by the first robot 10 and the second robot 20. Details of the work state selection process will be described later.

Following step S103 in FIG. 4, in step S104, the first operator 30 performs work in the work state selected in the selection process of step S103. This allows the first operator 30 to efficiently improve his or her work skills. Note that if the first operator 30's skills do not improve quickly in step S104, the first operator 30 may continue to perform the work until his or her skills improve to a certain extent, or the work state assigned to the first operator 30 may be changed (modified) as appropriate.

In the processing flow shown in FIG. 4, after step S104, the process returns to step S102, and steps S102, S103, and S104 are repeated. As a result, the first behavioral model π1 is updated as the first operator 30 grows, and the process for promoting improvement in the skills of the first operator 30 (i.e., steps S102, S103, and S104) is repeated.

Figure 6 is a flowchart showing an example of the processing flow for selecting an operation status. Step S201 in Figure 6 corresponds to the start of the processing flow shown in Figure 6. Step S205 in Figure 6 corresponds to the end of the processing flow shown in Figure 6. The processing flow shown in Figure 6 corresponds to an example of the processing flow performed in step S103 in the processing flow shown in Figure 4 described above.

When the processing flow shown in FIG. 6 starts, in step S202, the first learning unit 11d of the first robot 10 learns the first behavioral model π1 through reinforcement learning in each of the multiple task states.

In step S202, several candidate work states are prepared in advance. The first learning unit 11d then learns the first behavioral model π1 for each of the prepared candidate work states. For ease of understanding, the following describes a case in which three candidate work states for the task of organizing boxes, namely, a first work state, a second work state, and a third work state, each with a different combination of box position and number, are prepared as candidate work states. However, the number of candidate work states may be other than three. Furthermore, the types of work states are not limited to this example.

For example, in the above example in which three task states are prepared as candidates, the first learning unit 11d first learns the first behavioral model π1 through reinforcement learning in the first task state. In the reinforcement learning for the first task state, the first learning unit 11d, for example, causes the first robot 10 to perform various actions in a simulated task state that mimics the actual first task state, and calculates a reward for each action performed by the first robot 10. Here, the first learning unit 11d calculates the reward so that the higher the efficiency of the action performed by the first robot 10 (for example, the shorter the task time or the higher the success rate). The first learning unit 11d calculates such a reward for each action. Then, the first learning unit 11d uses the obtained relationship between the action and the reward to update the relationship between the input (pair of state and action) and the output (evaluation index (recommended degree) of the action) in the first behavioral model π1. The first behavioral model π1 after learning through reinforcement learning for the first task state is stored, for example, in the first memory unit 11f of the first robot 10.

Next, the first learning unit 11d returns the first behavior model π1 to the behavior model before reinforcement learning for the first work state, and then learns the first behavior model π1 through reinforcement learning in the second work state. The details of reinforcement learning for the second work state are the same as those for the first work state, so a detailed explanation will be omitted. The first behavior model π1 after learning through reinforcement learning for the second work state is stored, for example, in the first memory unit 11f of the first robot 10.

Next, the first learning unit 11d returns the first behavior model π1 to the behavior model before reinforcement learning for the second work state, and then learns the first behavior model π1 through reinforcement learning in the third work state. The details of reinforcement learning for the third work state are similar to those for the first work state, so a detailed explanation will be omitted. The first behavior model π1 after learning through reinforcement learning for the third work state is stored, for example, in the first memory unit 11f of the first robot 10.

Following step S202, in step S203, the selection unit 11e of the first robot 10 compares the learned first behavioral model π1 with the learned second behavioral model π2 for each of a plurality of work states (in the above example, the first work state, the second work state, and the third work state).

As described above, the second behavioral model π2 is an imitation model that imitates the behavior of the second worker 40, and learning of the second behavioral model π2 is performed by the second learning unit 21d of the second robot 20. Specifically, the second learning unit 21d learns the second behavioral model π2 through imitation learning based on the behavior performed by the second worker 40. For example, imitation learning of the second behavioral model π2 is performed using a record of the behavior of the second worker 40, similar to imitation learning of the first behavioral model π1. In imitation learning of the second behavioral model π2, the second learning unit 21d causes the second robot 20 to perform various actions and calculates a reward for each action performed by the second robot 20. Here, the second learning unit 21d calculates the reward so that the closer (more similar) the action performed by the second robot 20 in a certain state is to the action of the second worker 40 corresponding to that state, the higher the reward. The second learning unit 21d calculates this reward for each state. The second learning unit 21d then uses the obtained relationship between the state-action pairs and the reward to update the relationship between the input (state-action pairs) and the output (evaluation index (degree of recommendation) of the action) in the second behavioral model π2.

In step S203, the selection unit 11e compares each of the learned first behavioral models π1 and the second behavioral model π2 obtained by learning each candidate task state in step S202. Specifically, the selection unit 11e calculates the similarity of each learned first behavioral model π1 to the second behavioral model π2. The similarity increases the more similar the learned first behavioral model π1 is to the second behavioral model π2. The similarity of the learned first behavioral model π1 to the second behavioral model π2 may be calculated, for example, by comparing the output results when various pieces of input information are input to each behavioral model, or by calculation using a method such as a correlation function.

For example, in the above example where three task states are prepared as candidates, the selection unit 11e first calculates the similarity of the first task state's first behavioral model π1 to the second behavioral model π2 after learning through reinforcement learning. Next, the selection unit 11e ... third task state's first behavioral model π1 to the second behavioral model π2 after learning through reinforcement learning.

Following step S203, in step S204, the selection unit 11e of the first robot 10 selects, from among the multiple work states (in the above example, the first work state, the second work state, and the third work state), the state in which the post-learning first behavioral model π1 has the highest similarity to the second behavioral model π2 as the optimal work state, and the processing flow shown in Figure 6 ends.

As described above, the second behavioral model π2 is an imitation model that imitates the behavior of the second operator 40, who is a skilled operator. Therefore, the higher the similarity of the learned first behavioral model π1 to the second behavioral model π2 (i.e., the more similar the learned first behavioral model π1 is to the second behavioral model π2), the closer the behavior of the first robot 10 using the learned first behavioral model π1 will be to the behavior of the second operator 40. Therefore, the state with the highest similarity is the optimal working state for promoting improvement in the skills of the first operator 30 and efficiently bringing the skills of the first operator 30 closer to those of the second operator 40.

As described above, in the processing system 1, the first learning unit 11d of the first robot 10 learns the first behavioral model π1 through reinforcement learning for each of a plurality of work states, and the selection unit 11e compares the learned first behavioral model π1 with the second behavioral model π2 for each of the plurality of work states and selects the optimal work state based on the results of the comparison. This makes it possible to select the optimal work state to promote improvement in the skills of the first worker 30, and to provide such work state to the first worker 30.

By having the first worker 30 work in such a working condition, the improvement of the first worker's skills can be promoted. This allows the first worker's skills to efficiently approach those of the second worker 40, who is an experienced worker. In this way, the processing system 1 can efficiently improve the skills of the first worker 30 while saving the second worker 40's time, thereby enabling the efficient transfer of skills at the work site.

As described above, the first learning unit 11d of the first robot 10 learns the first behavioral model π1 through imitation learning based on the behavior of the first operator 30 after the first operator 30 works in the work state selected by the selection unit 11e. Specifically, the first operator 30 works in the optimal work state selected in the processing flow of FIG. 6 described above (step S104 in FIG. 4). As a result, the first operator 30 grows. Thereafter, the first behavioral model π1 is learned through imitation learning (step S102 in FIG. 4). As a result, the first behavioral model π1 can be updated to imitate the behavior of the first operator 30 after growth.

In the above, an example has been described in which the second behavioral model π2 is an imitation model that imitates the behavior of a single second worker 40. However, the second behavioral model π2 is not limited to this type of behavioral model. For example, the second behavioral model π2 may be generated by preparing multiple different imitation models that imitate the behavior of multiple second workers 40 and integrating these multiple imitation models. For example, the arithmetic mean of the multiple imitation models may be generated as the second behavioral model π2. As described above, the second behavioral model π2 may be generated by integrating multiple imitation models that imitate the behavior of the second worker 40. This enables a wider range of skills to be passed on.

In the above example, the functional units of the first processing device 11 (i.e., the first acquisition unit 11a, the first determination unit 11b, the first control unit 11c, the first learning unit 11d, the selection unit 11e, and the first memory unit 11f) and the functional units of the second processing device 21 (i.e., the second acquisition unit 21a, the second determination unit 21b, the second control unit 21c, the second learning unit 21d, and the second memory unit 21e) are separated into separate robots. However, each of the functional units described above may be included in a single device. For example, the first robot 10 may include a simulator that internally calculates the processing performed by the second robot 20 described above. In this case, the second robot 20 may be omitted from the processing system 1.

<Effects of the treatment system>
The effects of the processing system 1 according to the embodiment of the present invention will be described.

In the processing system 1, the behavioral models include a first behavioral model π1, which is an imitation model that imitates the behavior of the first worker 30, and a second behavioral model π2, which is an imitation model that imitates the behavior of the second worker 40. The processing system 1 includes a first determination unit 11b that determines behavior using the first behavioral model π1, a first learning unit 11d that learns the first behavioral model π1, and a selection unit 11e that selects a specific state (in the above example, the optimal work state) from multiple states (in the above example, the work states). Here, the first learning unit 11d learns the first behavioral model π1 using reinforcement learning for each of the multiple states, and the selection unit 11e compares the learned first behavioral model π1 with the second behavioral model π2 for each of the multiple states and selects a specific state based on the results of the comparison. By providing the selected specific state to the first worker 30, the skills of the first worker 30 can be efficiently brought closer to those of the second worker 40. This allows the second worker 40 to transfer skills to the first worker 30 while saving time, making it possible to efficiently transfer skills at the work site.

Preferably, in the processing system 1, the selector 11e selects, from among multiple states (in the above example, work states), the state in which the first behavioral model π1 after learning has the highest similarity to the second behavioral model π2 as the specific state (in the above example, the optimal work state). This makes it possible to appropriately select the optimal state to promote improvement in the skills of the first worker 30. This therefore makes it possible to appropriately achieve efficient skills transfer at the workplace.

Preferably, in the processing system 1, the first learning unit 11d learns the first behavioral model π1 by imitation learning based on the behavior of the first worker 30 after the first worker 30 works in the specific state selected by the selection unit 11e (in the above example, the optimal working state). This allows the first behavioral model π1 to be updated so that it imitates the behavior of the first worker 30 after growth.

Preferably, in the processing system 1, the second behavioral model π2 is generated by integrating multiple imitation models that imitate the behavior of the second worker 40. This allows for the transfer of a wider range of skills.

Preferably, in the processing system 1, the second operator 40 is a worker with higher skills than the first operator 30. As a result, the above-described processing by the first learning unit 11d and the selection unit 11e appropriately achieves efficient improvement of the skills of the first operator 30 while saving the second operator 40's time. Therefore, efficient transfer of skills at the work site is appropriately achieved.

The above describes a preferred embodiment of the present invention with reference to the accompanying drawings. However, it goes without saying that the present invention is not limited to the above-described embodiment, and that various modifications and alterations within the scope of the claims also fall within the technical scope of the present invention.

For example, the processes described herein using flowcharts do not necessarily have to be performed in the order shown in the flowcharts. Some processing steps may be performed in parallel. Additionally, additional processing steps may be employed, and some processing steps may be omitted.

Furthermore, for example, the series of control processes performed by the processing system 1 described above may be realized using software, hardware, or a combination of software and hardware. The programs that make up the software are stored in advance, for example, on a storage medium provided inside or outside the information processing device.

1 Processing system 10 First robot 11 First processing device 11a First acquisition unit 11b First decision unit 11c First control unit 11d First learning unit 11e Selection unit 11f First storage unit 20 Second robot 21 Second processing device 21a Second acquisition unit 21b Second decision unit 21c Second control unit 21d Second learning unit 21e Second storage unit 30 First worker 40 Second worker π1 First behavioral model π2 Second behavioral model

Claims (6)

A processing system (1) for determining an action using a behavior model for determining an action according to a state,
The behavioral models include a first behavioral model (π1) that is an imitation model that imitates the behavior of a first worker (30) and a second behavioral model (π2) that is an imitation model that imitates the behavior of a second worker (40),
The processing system (1)
a first determination unit (11b) that determines an action using the first action model (π1);
a first learning unit (11d) that learns the first behavioral model (π1);
a selection unit (11e) for selecting a specific state from a plurality of states;
Equipped with
the first learning unit (11d) learns the first behavior model (π1) by reinforcement learning in each of the plurality of states;
the selection unit (11e) compares the first behavioral model (π1) and the second behavioral model (π2) after learning for each of the plurality of states, and selects the specific state based on a result of the comparison.
Processing system. the selection unit (11e) selects, from the plurality of states, a state in which the degree of similarity of the first behavioral model (π1) to the second behavioral model (π2) after learning is highest as the specific state;
The processing system of claim 1 . the first learning unit (11d) learns the first behavioral model (π1) by imitation learning based on the behavior of the first worker (30) after the first worker (30) works in the specific state selected by the selection unit (11e);
The processing system according to claim 1 or 2. The second behavioral model (π2) is generated by integrating a plurality of imitation models that imitate the behavior of the second worker (40).
The processing system according to any one of claims 1 to 3. The second worker (40) is a worker with higher skills than the first worker (30).
The processing system according to any one of claims 1 to 4. A processing method for determining an action using a behavior model for determining an action according to a state, comprising:
The behavioral models include a first behavioral model (π1) that is an imitation model that imitates the behavior of a first worker (30) and a second behavioral model (π2) that is an imitation model that imitates the behavior of a second worker (40),
a first step of determining an action using the first action model (π1);
a second step of learning the first behavioral model (π1);
a third step of selecting a particular state from the plurality of states;
Equipped with
In the second step, learning of the first behavioral model (π1) by reinforcement learning is performed in each of the plurality of states;
In the third step, a comparison is made between the first behavioral model (π1) and the second behavioral model (π2) after learning for each of the plurality of states, and the specific state is selected based on a result of the comparison.
Processing method.